[0.14.13] Multiplayer lag, no latency option

[Demosthenex]

I have successfully run Factorio servers on 0.12 and 0.13, and now have setup a new headless server on 0.14.13. In previous versions we used "--latency 200" on the server and had no problems. Our five players are split between EU and US so latency can be an issue. Setting our minimum latency server side allowed us to tremendously enjoy previous versions!

Now on 0.14 our busiest NA player is receiving frequent lag. It is likely that his connection is having a minor problem (either brief packet loss or lag spike) given his distance, however his ping spikes are never over 200 ms.

He is receiving frequent popups saying server not responding. Monitoring the latency it increases from 100 to 200, buffers changes from 0-5 to 50+, and jump from 0 to 100.

What can we do to improve his play experience? I have checked the headless server command line options, server-settings.json, and config.ini and there are no latency settings. It appears all minimum latency controls have been removed in 0.14.

[Rseding91]

Latency is automatically calculated and set dynamically as the game runs based off the players connection quality. Jumps of 50 means he's skipping 50 ticks which means his latency is around 833~ MS.

If his connection is that unreliable there's not much you can do. You could try getting a server closer to his location but even opposite sides of the world typically have less latency than that.

[Demosthenex]

Running a 12 hour ping test, we see an average latency of 155 ms, with his worst 315 and a stdev of 7 and a packet loss of 2%. That's pretty consistent.

I think the key here is we played at this distance very successfully in 0.12 and 0.13, and didn't expect it to break with 0.14.

Maybe we need a way to adjust the frequency of the popups? I think that may be the only real problem. When he's playing otherwise it's not a concern.

Can we set the delay before displaying server unresponsive? Is it currently at 200 ms meaning he's surfing the edge? Could we bump that to 500?

[sillyfly]

I think I may be facing a similar problem: viewtopic.php?f=49&t=34375

Maybe add an option to limit the minimum latency? Or lower it more gradually, dynamically calculating a "safe" lower bound?

[Demosthenex]

Is there any chance we can get a user specified minimum added to the calculated latency?

[Klonan]

Is there any chance we can get a user specified minimum added to the calculated latency?

The behaviour you describe doesn't sound like latency issues, more that his connection is dropping packets and missing some ticks

In this case increasing the latency wouldn't to much to help, he would still experience the popups, so long as he keeps dropping packets

[Demosthenex]

Is there any chance we can get a user specified minimum added to the calculated latency?

The behaviour you describe doesn't sound like latency issues, more that his connection is dropping packets and missing some ticks

In this case increasing the latency wouldn't to much to help, he would still experience the popups, so long as he keeps dropping packets

I can understand that, and yet I have to assert it worked fine on 0.12 and 0.13. We did set the server to a minimum 200 ms latency for all players which may have masked the occasional retransmit or packet issue. That masking is what I'm asking for.

Now when he plays it is more sensitive to a packet dropped because it's calculated it down to a very low value. Thus when the inevitable packet loss occurs, there is more impact.

That's why I'm asking for a minimum latency, so his client would never calculate below a specific user set threshold.

His actual internet connection is fine, the issue is playing from the US to EU. He can play intensive real time games like League of Legends with us with no issues. The behavior is confined to the new 0.14 Factorio. It's the nature of long distance connections.

[draktok]

Yeah, i'm having the same issue. We could play fine before, but now he gets micro stutters. It's very annoying. It didn't use to happen, when we could set the latency. He's in south FL, i'm in Minnesota. Same country, but obviously opposite sides = 100~150 ms when he drop a packet, and while i understand it may be HIS fault, its definitely not our fault that we lost our ability to set server ping that basically gave us the free option to account for it. It's more broken now, basically. Not sure why i can't just host P2P like i used to.

[daniel34]

That's why I'm asking for a minimum latency, so his client would never calculate below a specific user set threshold.

This was added in 0.14.18:

Minor features

• Added minimum_latency_in_ticks to server settings (for headless server) and into config (for starting game from gui).

To set this when hosting from within the game (from the GUI): go to %appdata%\Factorio\config and open the config.ini file with a text editor (Notepad will do). Set the minimum_latency_in_multiplayer (it's under [other]) to the desired value. You probably need to restart the game for this setting to take effect.

[Demosthenex]

I did see that. I'm looking at updating the server and testing, but I'll use Emacs. Thanks.

[Neutorio]

I can confirm we have the same issues

We had a save since before the patch and everything was fine. After the patch, the problems started appearing.

I'm hosting with my client and my mate connects to my IP.

When my mate connects, the map downloads perfectly fine, at around 5MB/s. But once it gets to the "Catching up" stage, it takes forever, as in it can take several minutes before he gets into the game. If I do many actions, sometimes the progress bar even moves backwards as it seems like his client can't catch up to the new events in the map, and I may have to pause the game to allow him to finally catch up.
Once he's in the game, the game stutters for him and he can't move properly.. Just a lot of rubberbanding. The game is unplayable for him. We tried to type in minimum_latency_in_multiplayer=100 in the config file with no luck (we used to play with the slider on 83 or 116ms or whatever it was)

We're in the same town, on the same ISP, both with symmetric fibre, and both connected with a network cable in each end. The latency is probably close to being on the same LAN, 3-4ms at most. We also tried to type in something like "9999KB/s" into the "maximum upload speed" box in case it wanted something there, but nope.

The factory isn't even "that" enormous (the save file is 32MB in size), though there's always some action with robots and such. If we try to start a fresh game, it is somewhat better but it's obvious it's not perfect.

I realise that the devs have upgraded the networking part lately by removing the need to freeze the game while transferring the map and moving on to automatic latency. so some issues are to be expected, but have they confirmed the bug and said they'll be working on it? It would be nice to hear from a dev. It's not an issue with our internet connections.

[boogy]

Same problem here earlier it worked fine, now only the map downloads on full speed, after that it drops to 1.2 kb/s limited up-/download, no matter if I or my friend host the game. We also tried different max. upload speeds and setting the minimum latency in the config file, also different beta branches.. Please fix this! ):

We also can't play online on other servers, we just have around 1.2 kb/s max download (after the map is downloaded on high speed) too.

[draktok]

We tried doing that setting to set the manual latency to 150, it still doesn't work. He's constantly dropping, where as he didn't before.

[boogy]

Yeah I think we can't do anything here, there are some bugs in the dynamic calculation. We tried everything, the game always sets our max. up-/download speed to 1.2 kb/s (watched outgoing / incoming packets) right after the map download is finished, even with more than 100 mbit/s stable connections. We tried v13.20 out, where this "feature" wasn't implemented yet: everything works fine with the good old ping-slider.

[daniel34]

When my mate connects, the map downloads perfectly fine, at around 5MB/s. But once it gets to the "Catching up" stage, it takes forever, as in it can take several minutes before he gets into the game. If I do many actions, sometimes the progress bar even moves backwards as it seems like his client can't catch up to the new events in the map, and I may have to pause the game to allow him to finally catch up.
Once he's in the game, the game stutters for him and he can't move properly.. Just a lot of rubberbanding. The game is unplayable for him. We tried to type in minimum_latency_in_multiplayer=100 in the config file with no luck (we used to play with the slider on 83 or 116ms or whatever it was)

Yeah I think we can't do anything here, there are some bugs in the dynamic calculation. We tried everything, the game always sets our max. up-/download speed to 1.2 kb/s (watched outgoing / incoming packets) right after the map download is finished, even with more than 100 mbit/s stable connections. We tried v13.20 out, where this "feature" wasn't implemented yet: everything works fine with the good old ping-slider.

During the catching-up phase it's not the network that is primarily used, after all the map has already been downloaded. Instead it's the CPU that is replaying the map from the point when the map was saved before you downloaded it to the point it is currently on the host. On bigger maps this can take some time, and may not progress at all or even go backwards if your CPU is not fast enough to handle the game at 60 UPS. During that time it's normal if the network transfer is only a few kb/s because it's only downloading map changes / player actions like it would do during actual gameplay.

[boogy]

During the catching-up phase it's not the network that is primarily used, after all the map has already been downloaded. Instead it's the CPU that is replaying the map from the point when the map was saved before you downloaded it to the point it is currently on the host. On bigger maps this can take some time, and may not progress at all or even go backwards if your CPU is not fast enough to handle the game at 60 UPS. During that time it's normal if the network transfer is only a few kb/s because it's only downloading map changes / player actions like it would do during actual gameplay.

Thanks for the detailed answer, I would love to believe that but we also have the problem with complete new maps and good rigs (i7-4790k, 16gb ram etc).. Also the catching up process pauses the game and opens a new loading panel, shouldnt it be done after this?

Furthermore: let us expect that the new client has to catch up: how should he do this, when the transfer speed is limited to 1.2kb/s (watched packets) but the hosts game runs with full speed? He would just pass away more and more. We've waited for about 30 minutes, nothing changed: the host can play, the client only gets a limited stream. He also would (theorically) have to catch up only a few seconds, because the map download finishes with new maps on 2-3 seconds, even when we pause the game there is no difference..

Also the addition that everything works fine with new or big maps on v13.20 makes this thing a bit odd

[daniel34]

Thanks for the detailed answer, I would love to believe that but we also have the problem with complete new maps and good rigs (i7-4790k, 16gb ram etc).. Also the catching up process pauses the game and opens a new loading panel, shouldnt it be done after this?

Furthermore: let us expect that the new client has to catch up: how should he do this, when the transfer speed is limited to 1.2kb/s (watched packets) but the hosts game runs with full speed? He would just pass away more and more. We've waited for about 30 minutes, nothing changed: the host can play, the client only gets a limited stream. He also would (theorically) have to catch up only a few seconds, because the map download finishes with new maps on 2-3 seconds, even when we pause the game there is no difference..

Also the addition that everything works fine with new or big maps on v13.20 makes this thing a bit odd

If you start a completely new map and have to wait longer than a few seconds for it to completely load and catch up then there's something wrong. Also, with 0.14 the server should never pause (except for saving the game before uploading the map).
Please start a new game with your friend and then post the log files of both host and client after you've waited a few minutes to load/catch-up the game (or after the client successfully joined and can play, if possible). Make sure you're both running 0.14.20. Also check on the catch-up bar and CPU/network usage during catching-up.

[boogy]

If you start a completely new map and have to wait longer than a few seconds for it to completely load and catch up then there's something wrong. Also, with 0.14 the server should never pause (except for saving the game before uploading the map).
Please start a new game with your friend and then post the log files of both host and client after you've waited a few minutes to load/catch-up the game (or after the client successfully joined and can play, if possible). Make sure you're both running 0.14.20. Also check on the catch-up bar and CPU/network usage during catching-up.

I've sent you the logs in a pm, also I can just repeat myself: there is no really catch-up-phase.. After the map is downloaded (with full speed) the game instantly limits up-/download speed to 1.2kb/s on a new map, where we waited 5-10 minutes. Our CPU usage is around 15% constantly, everything else is fine too. The game just limits the transfer speed. And as I said: version 13 works fine, so I guess it's not our rig, fault or whatever..

[draktok]

Just posting because, i set the latency thru the option to 150 - 500 and it hasn't resolved. He is constantly desyncing and teleporting. Yeah, we didn't have the issue in the previous. Why not just let me play p2p ...

[daniel34]

After the map is downloaded (with full speed) the game instantly limits up-/download speed to 1.2kb/s on a new map

The game doesn't limit the speed, it just doesn't require more. I tested this a few minutes ago on (1) the official team challenge server which was empty at the time and (2) a game running for a few hours already with an average base, 5 other players online. In both cases the download was 1.1-1.2 kb/s and upload 500-600 bytes/s. I could play without problems on both servers.

There is also nothing strange in the log files, client joins and downloads the map in 1.4 seconds, then takes another 0.2 seconds to catch up and then is in the game. Latency then stays around 20-25 most of the time which is ok. Only at the end (after 3.5 of the 4 minutes you played) the client skips over a few ticks, but that should just be a short lag spike and the game should run without any lag the rest of the time.

the host can play, the client only gets a limited stream

How does the limited stream affect the client? Does he have input lag, is he rubberbanding (teleporting around) or being (almost) dropped from the game?