Clients getting dropped from headless server

[Evio]

I started experiencing a bug in my private server, which has many mods, with one single symptom:

Right after connecting every single player appears to desync in about 3 seconds and is then dropped.

Server output:

Code: Select all

Info ServerMultiplayerManager.cpp:1061: Disconnect notification for peer (5)
Info ServerSynchronizer.cpp:627: nextHeartbeatSequenceNumber(28441) removing peer(5).

Client output:

Code: Select all

Warning ClientSynchronizer.cpp:104: nextHeartbeatSequenceNumber(1103056790) server wants a heartbeat(1103055579) that we don't have.
Error ClientMultiplayerManager.cpp:98: MultiplayerManager failed: "You have been dropped from the game."
Info ClientMultiplayerManager.cpp:608: UpdateTick(20128) changing state from(InGame) to(WaitingForUserToSaveOrQuitAfterServerLeft)
Info ClientMultiplayerManager.cpp:930: Disconnect notification for peer (5)

Normally, after connecting the server will periodically output these messages:

Code: Select all

89.942 Info ClientSynchronizer.cpp:335: Latency changed to (6)

But when the conditions for this bug are fulfilled, these messages are never shown.

I've been for the past two days methodically creating new multiplayer games with different combinations of all the mods that I'm using, so I managed to make a list of the minimum mods needed to reproduce this bug.

The strange thing of this bug is that it can be reproduced always and only with that exact set of mods, and only on a headless server.

I literally restarted the Factorio server and a 2 clients over a couple hundred times to try possible combinations of mods, server modes, and many iterations of the final mod-set minus a single mod. If you remove a single mod of that list the bug “gets broken”.

Requirements to reproduce the bug:

1. Use Factorio 1.1.59 (build 59719, linux64), fresh or non-fresh installation
2. Host a game in a headless server with the following list of mods enabled:
   • Bulk Rail Loader v1.1.4
   • Cargo Ships 0.1.18 (pulls Cargo Ships Graphics)
   • Fully Automated Rail Layer v4.1.2
   • Geothermal v1.105.1 (pulls Dragon Industries - Nauvis Division)
   • HelicopterRevival v0.2.1
   • Hovercrafts v1.2.1
   • Insectitronics v1.1.1
   • Inserter Cranes v0.0.6
   • Jetpack v0.3.2
   • Krastorio 2 v1.2.25 (pulls Krastorio 2 Assets and flib)
   • Larger Lamps v1.5.0
   • Laser Tanks Updated v0.19.0
   • LTN Manager v0.4.12 (pulls LTN - Logistic Train Network and flib)
   • Modular Turrets v7.2.1
   • Raven v0.1.12
   • Realistic Fusion Power v1.8.15
   • Recipe Book v3.4.5 (pulls flib)
   • Schall Pipe Scaling v1.1.1
   • Schall Tank Platoon v1.1.1
   • SE Space Trains v0.1.11
   • Shield Projector v0.1.5
   • Spidertron Extended v0.3.14
   • Text Plates v0.6.3
   • Turret Shields v1.1.0
   • UMR: Ultimate Modular Reactor v0.1.10 (pulls Squeak Through)

I don't really know how to debug anything after this. Can anybody else reproduce this?

(I wanted to post this under Bug Reports ‣ Desync with Mods but I'm unable to post there)

[Loewchen]

Post a link to the desync report.

[Evio]

I don't know what a desync report is.
What is it?

[Loewchen]

Do you actually know what a desync is?

[Evio]

Yes, it is when server and client don't agree in what happened in the game so they end up with a different state, generally triggered by network problems or mods not taking care in their state so server and client differ at some point.

So, what did you mean by “desync report”?

If you mean what Factorio outputs in the terminal I already provided the relevant output for both the headless server and the client.

Or you mean some other output that I can enable in the command line?

[Loewchen]

Did those who "desynced" see this window?

[Evio]

No, it's not a desync that triggers that message.

This desync is more like a it-never-synced to begin with. And this only happens if it's served by a headless server, it won't happen if you start the game as single player or host it in a graphical version of the game, it's as if some interaction between these mods is affected by the presence of a graphical host.

[Loewchen]

Ok there is no desync then, the players are just dropped from the game after connecting?
If yes, then this is likely 102595.

[lyvgbfh]

This isn't a desync in the sense of the term that is typically used on the forum. Please try changing flib's dictionary levels per batch setting and see if this helps.

[Evio]

Ok there is no desync then

More like “not the usual desync behavior”; we don't really know what's happening.

If server and client depart in their state, whether by race condition or inability to synchronize, it's still a desync.

the players are just dropped from the game after connecting?
If yes, then this is likely 102595.

All players are dropper after connecting, but it's different compared to that post.

1. It happens to every player, no exceptions.
2. FPS/UPS don't change at all, they steadily oscillate between 59.9 and 60.0
3. Unlike in that post, the message is never “your computer couldn't keep up” or “lost connection”, it's always “You have been dropped from the game”, no exceptions.
4. In that post, the client that connects has the “latency changed to...” messages in the output, as I stated in my post these messages are never shown in this case.
5. Unlike in that post, where OP said the bug won't happen in a new map, in my case in happens always in new or old map, and with a completely fresh Factorio directory.
6. Unlikely to be an inability to keep up like it was suggested in that post and in other post you linked there, in my test scenario I host in localhost. Whether it's localhost or a remote system network has nothing to do with it, change to mods only affect the ability to reproduce the bug.
7. None of the suggestions in those threads helped, I just tried them all.

[Evio]

This isn't a desync in the sense of the term that is typically used on the forum. Please try changing flib's dictionary levels per batch setting and see if this helps.

Where do I see this? I could not find any settings for this in the Mod Settings GUI.

Edit: nevermind, found it, testing now.

[Evio]

Please try changing flib's dictionary levels per batch setting and see if this helps.

This fixed the problem, thanks for your help.

I still wonder why this doesn't happen if the host is not a headless Factorio. There must be something that can be improved here.

[lyvgbfh]

Please try changing flib's dictionary levels per batch setting and see if this helps.

This fixed the problem, thanks for your help.

I still wonder why this doesn't happen if the host is not a headless Factorio. There must be something that can be improved here.

flib provides "translated" search, allowing users to search items/recipes/etc in their local language. This is done by requesting, one by one, the names of items from players in the game. Headless servers don't have a player, so they don't aid in this while a listen server which has a player, does.

[Evio]

Headless servers don't have a player, so they don't aid in this while a listen server which has a player, does.

I did not test to see this so I'll ask you since you know about flib.

The cached translations are supposed to be stored in the client side, right?

What puzzles me here is that I don't know who to blame, a client that gets stuck and won't synchronize with the server or a server that's somehow too busy to listen to clients.

I imagine that it might overlap, mods have to share a state so the server must still do something in coordination with the flib of connected clients.

Right before this post I did the following:

1. Started a single player map with all the conditions that would make this headless-server bug happen.
2. Let it run for 2 minutes and then save.
3. Moved the savegame to the headless server and hosted it.

After this the game now works, client from a fresh Factorio directory connects and plays just fine, and I didn't need to lower the batch number for flib.

This suggests me that the server needs to accomplish something regarding translations.

Edit: now that I think even a mod that translates texts meant for the GUI has to work in coordination from a headless server.

[lyvgbfh]

After this the game now works, client from a fresh Factorio directory connects and plays just fine, and I didn't need to lower the batch number for flib.

This suggests me that the server needs to accomplish something regarding translations.

Edit: now that I think even a mod that translates texts meant for the GUI has to work in coordination from a headless server.

Not really, as the mod only translates based on the languages of connected players, by asking each player "what is x prototype's name in your loaded language" since the game only loads one locale. I'm not sure if this is enough to answer your question?

[Evio]

Not really, as the mod only translates based on the languages of connected players, by asking each player "what is x prototype's name in your loaded language" since the game only loads one locale. I'm not sure if this is enough to answer your question?

Yes, it was a good answer.

It gave me the idea to test by first running a single player map with the conditions for the bug, let it run for a couple minutes and then move it to the headless server.

Now I connect from a client in English and the other one in any other language and as expected from your answer only the one in English isn't dropped, because I let the game run offline in that language before having it hosted in the headless server.

Now the only thing remaining is, there must be something that can be done about this without the user having to fine tune settings, how can a mod interfere with networking that much...

[Hornwitser]

I tried reproducing your issue and it doesn't happen on my system when I run a dedicated server locally with those mods. I also tried adding a bunch more mods but I'm still unable to reproduce the issue.

[Evio]

Apparently this has to do with the system that's serving the game too, I could reproduce it with another set to mods in one system but won't reproduce in another newer computer I got.

This isn't reproducible after all and as we saw the desync actually happens because some mods can effectively lock the networking. Lowering batches for flib helped in my case.

I tried to change the title of this topic but somebody already changed it to a less descriptive title and now I can't edit it any more.