Clients getting dropped from headless server

Anything that prevents you from playing the game properly. Do you have issues playing for the game, downloading it or successfully running it on your computer? Let us know here.
Post Reply
Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Clients getting dropped from headless server

Post by Evio »

I started experiencing a bug in my private server, which has many mods, with one single symptom:
Right after connecting every single player appears to desync in about 3 seconds and is then dropped.
Server output:

Code: Select all

Info ServerMultiplayerManager.cpp:1061: Disconnect notification for peer (5)
Info ServerSynchronizer.cpp:627: nextHeartbeatSequenceNumber(28441) removing peer(5).
Client output:

Code: Select all

Warning ClientSynchronizer.cpp:104: nextHeartbeatSequenceNumber(1103056790) server wants a heartbeat(1103055579) that we don't have.
Error ClientMultiplayerManager.cpp:98: MultiplayerManager failed: "You have been dropped from the game."
Info ClientMultiplayerManager.cpp:608: UpdateTick(20128) changing state from(InGame) to(WaitingForUserToSaveOrQuitAfterServerLeft)
Info ClientMultiplayerManager.cpp:930: Disconnect notification for peer (5)
Normally, after connecting the server will periodically output these messages:

Code: Select all

89.942 Info ClientSynchronizer.cpp:335: Latency changed to (6)
But when the conditions for this bug are fulfilled, these messages are never shown.

I've been for the past two days methodically creating new multiplayer games with different combinations of all the mods that I'm using, so I managed to make a list of the minimum mods needed to reproduce this bug.

The strange thing of this bug is that it can be reproduced always and only with that exact set of mods, and only on a headless server.

I literally restarted the Factorio server and a 2 clients over a couple hundred times to try possible combinations of mods, server modes, and many iterations of the final mod-set minus a single mod. If you remove a single mod of that list the bug “gets broken”.

Requirements to reproduce the bug:
  1. Use Factorio 1.1.59 (build 59719, linux64), fresh or non-fresh installation
  2. Host a game in a headless server with the following list of mods enabled:
I don't really know how to debug anything after this. Can anybody else reproduce this?

(I wanted to post this under Bug Reports ‣ Desync with Mods but I'm unable to post there)


Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

I don't know what a desync report is.
What is it?

Loewchen
Global Moderator
Global Moderator
Posts: 8284
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Loewchen »

Do you actually know what a desync is?

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

Yes, it is when server and client don't agree in what happened in the game so they end up with a different state, generally triggered by network problems or mods not taking care in their state so server and client differ at some point.

So, what did you mean by “desync report”?

If you mean what Factorio outputs in the terminal I already provided the relevant output for both the headless server and the client.

Or you mean some other output that I can enable in the command line?

Loewchen
Global Moderator
Global Moderator
Posts: 8284
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Loewchen »

Did those who "desynced" see this window?

Image

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

No, it's not a desync that triggers that message.

This desync is more like a it-never-synced to begin with. And this only happens if it's served by a headless server, it won't happen if you start the game as single player or host it in a graphical version of the game, it's as if some interaction between these mods is affected by the presence of a graphical host.

Loewchen
Global Moderator
Global Moderator
Posts: 8284
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Loewchen »

Ok there is no desync then, the players are just dropped from the game after connecting?
If yes, then this is likely 102595.

lyvgbfh
Fast Inserter
Fast Inserter
Posts: 165
Joined: Fri Jul 10, 2020 6:48 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by lyvgbfh »

This isn't a desync in the sense of the term that is typically used on the forum. Please try changing flib's dictionary levels per batch setting and see if this helps.

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

Loewchen wrote:
Wed Jun 22, 2022 2:45 am
Ok there is no desync then
More like “not the usual desync behavior”; we don't really know what's happening.

If server and client depart in their state, whether by race condition or inability to synchronize, it's still a desync.
Loewchen wrote:
Wed Jun 22, 2022 2:45 am
the players are just dropped from the game after connecting?
If yes, then this is likely 102595.
All players are dropper after connecting, but it's different compared to that post.
  1. It happens to every player, no exceptions.
  2. FPS/UPS don't change at all, they steadily oscillate between 59.9 and 60.0
  3. Unlike in that post, the message is never “your computer couldn't keep up” or “lost connection”, it's always “You have been dropped from the game”, no exceptions.
  4. In that post, the client that connects has the “latency changed to...” messages in the output, as I stated in my post these messages are never shown in this case.
  5. Unlike in that post, where OP said the bug won't happen in a new map, in my case in happens always in new or old map, and with a completely fresh Factorio directory.
  6. Unlikely to be an inability to keep up like it was suggested in that post and in other post you linked there, in my test scenario I host in localhost. Whether it's localhost or a remote system network has nothing to do with it, change to mods only affect the ability to reproduce the bug.
  7. None of the suggestions in those threads helped, I just tried them all.
Last edited by Evio on Wed Jun 22, 2022 3:21 am, edited 1 time in total.

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

lyvgbfh wrote:
Wed Jun 22, 2022 2:48 am
This isn't a desync in the sense of the term that is typically used on the forum. Please try changing flib's dictionary levels per batch setting and see if this helps.
Where do I see this? I could not find any settings for this in the Mod Settings GUI.

Edit: nevermind, found it, testing now.

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

lyvgbfh wrote:
Wed Jun 22, 2022 2:48 am
Please try changing flib's dictionary levels per batch setting and see if this helps.
This fixed the problem, thanks for your help.

I still wonder why this doesn't happen if the host is not a headless Factorio. There must be something that can be improved here.

lyvgbfh
Fast Inserter
Fast Inserter
Posts: 165
Joined: Fri Jul 10, 2020 6:48 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by lyvgbfh »

Evio wrote:
Wed Jun 22, 2022 3:43 am
lyvgbfh wrote:
Wed Jun 22, 2022 2:48 am
Please try changing flib's dictionary levels per batch setting and see if this helps.
This fixed the problem, thanks for your help.

I still wonder why this doesn't happen if the host is not a headless Factorio. There must be something that can be improved here.
flib provides "translated" search, allowing users to search items/recipes/etc in their local language. This is done by requesting, one by one, the names of items from players in the game. Headless servers don't have a player, so they don't aid in this while a listen server which has a player, does.

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

lyvgbfh wrote:
Wed Jun 22, 2022 3:52 am
Headless servers don't have a player, so they don't aid in this while a listen server which has a player, does.
I did not test to see this so I'll ask you since you know about flib.

The cached translations are supposed to be stored in the client side, right?

What puzzles me here is that I don't know who to blame, a client that gets stuck and won't synchronize with the server or a server that's somehow too busy to listen to clients.

I imagine that it might overlap, mods have to share a state so the server must still do something in coordination with the flib of connected clients.

Right before this post I did the following:
  1. Started a single player map with all the conditions that would make this headless-server bug happen.
  2. Let it run for 2 minutes and then save.
  3. Moved the savegame to the headless server and hosted it.
After this the game now works, client from a fresh Factorio directory connects and plays just fine, and I didn't need to lower the batch number for flib.

This suggests me that the server needs to accomplish something regarding translations.

Edit: now that I think even a mod that translates texts meant for the GUI has to work in coordination from a headless server.

lyvgbfh
Fast Inserter
Fast Inserter
Posts: 165
Joined: Fri Jul 10, 2020 6:48 pm
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by lyvgbfh »

Evio wrote:
Wed Jun 22, 2022 4:45 am
After this the game now works, client from a fresh Factorio directory connects and plays just fine, and I didn't need to lower the batch number for flib.

This suggests me that the server needs to accomplish something regarding translations.

Edit: now that I think even a mod that translates texts meant for the GUI has to work in coordination from a headless server.
Not really, as the mod only translates based on the languages of connected players, by asking each player "what is x prototype's name in your loaded language" since the game only loads one locale. I'm not sure if this is enough to answer your question?

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

lyvgbfh wrote:
Wed Jun 22, 2022 5:16 am
Not really, as the mod only translates based on the languages of connected players, by asking each player "what is x prototype's name in your loaded language" since the game only loads one locale. I'm not sure if this is enough to answer your question?
Yes, it was a good answer.

It gave me the idea to test by first running a single player map with the conditions for the bug, let it run for a couple minutes and then move it to the headless server.

Now I connect from a client in English and the other one in any other language and as expected from your answer only the one in English isn't dropped, because I let the game run offline in that language before having it hosted in the headless server.

Now the only thing remaining is, there must be something that can be done about this without the user having to fine tune settings, how can a mod interfere with networking that much...

Hornwitser
Fast Inserter
Fast Inserter
Posts: 204
Joined: Fri Oct 05, 2018 4:34 pm
Contact:

Re: Clients getting dropped from headless server

Post by Hornwitser »

I tried reproducing your issue and it doesn't happen on my system when I run a dedicated server locally with those mods. I also tried adding a bunch more mods but I'm still unable to reproduce the issue.

Evio
Burner Inserter
Burner Inserter
Posts: 11
Joined: Wed Jun 22, 2022 12:48 am
Contact:

Re: Always reproducible desync with only a set of mods and only if headless

Post by Evio »

Apparently this has to do with the system that's serving the game too, I could reproduce it with another set to mods in one system but won't reproduce in another newer computer I got.

This isn't reproducible after all and as we saw the desync actually happens because some mods can effectively lock the networking. Lowering batches for flib helped in my case.

I tried to change the title of this topic but somebody already changed it to a less descriptive title and now I can't edit it any more.

Post Reply

Return to “Technical Help”