[0.14.2][posila] Desync

[credomane]

Not entirely sure what else to put in the title.

Played for hours last night on a public server with 10+ people on at any given time. Think we almost got to 30 people for a short while. Was having a blast. I haven't been able to play multiplayer prior to this without causing massive lag to all players. This update is awesome!
Anyhow back on subject; Multiple people said they would report it but I've gone to bed and since woken up but don't see it here. There isn't a special desync only sub-forum is there?

I was busy attempting to finish automating building of bots and was running belts around to accomplish this. Other players were doing just about everything. We had tanks clearing trees. Turret creeping biters with both gun turrets and laser turrets. 40MW power deficiency. People becoming pancakes. You name it; it was happening.

This is the first desync report that hit all players at the same time. my all players at once desync report
During the following map redownload and catch up phase some players experienced their own "local" desync. I was one such player. This is my local desync report for that.

[TheVeteraNoob]

26 people were online at the time if I remember correctly aswell there was some kind of desync echo afterwards. And here is another desync report for redundancy.

[posila]

Hi, thanks for the report.

Desync happened because player "tehepicdude" had selected "diesel-locomotive" at position {-14.1875, -260.921875} on server, and "tree-04" at position {-13.3125, -258.3125} on clients. Desync reports for the "global" desync provided by credomane and TheVeteraNoob are identical. So, there is probably something wrong with SelectedEntityChanged input action. We are looking into that.

[Loewchen]

Hi, thanks for the report.

Desync happened because player "tehepicdude" had selected "diesel-locomotive" at position {-14.1875, -260.921875} on server, and "tree-04" at position {-13.3125, -258.3125} on clients. Desync reports for the "global" desync provided by credomane and TheVeteraNoob are identical. So, there is probably something wrong with SelectedEntityChanged input action. We are looking into that.

Did you guys add some additional diagnostic info to the report? The last time I saw desync analysis results it was about comparing screenshots
Btw the Y-coordinate of the "diesel-locomotive" seems wrong.

[posila]

So, it should be fixed for 0.14.3
It was really tough to find

@Loewchen: Desync analysis was never primary about comparing screenshots. First we resave the maps with human readable tags and compare them (see https://www.factorio.com/blog/post/fff-55). Sometimes it is easier to make sense difference in binary data when I can also see it, so I take screenshots. And then we think really hard how the desync we see could happen. Sometimes changes in maps are so big, that it is impossible to find "seed" of the desync.

[credomane]

So, it should be fixed for 0.14.3
It was really tough to find

Might I ask what it was exactly?

[posila]

Selection box of locomotives and trains depended partialy on locomotive sprite. Since headless sever doesn't load sprites, selection box of locomotive was different on headless server and on clients ... which caused "tehepicdude" to mine a tree on clients and a locomotive on server.

[credomane]

Dang. That is a fun one. How in the world do you fix that one so that any mods that change train sprite sizes or the size of any sprite) don't reintroduce this bug?

Make the headless load the sprites just long enough to determine the size then unload them?

Sorry if I'm nosey. Bugs and their fixes fascinate me.

[posila]

That's ok
We just added the value we needed to entity prototype, which makes it more pain to change graphics, but we don't want the game state to be dependent on graphics at all.

[credomane]

How do you guys go about comparing the desyncs to determine the cause or rather the difference like you did here? Is it possible for the community to have/use some of these desync tools?

I got two more desyncs from the same server that I haven't uploaded because they are really big, for my Internet, and I don't want to waste dev time looking at more of the same exact desync issues that are already reported and resolved.

[posila]

Hi, I forgot to respond .
We investigate deync by creating special save with tags, so when we use some binary comparator, we can find out fairly easily what is meaning of values that are different, or at least what entities are different. We had special build for this, but recently we added this feature to normal release, so when desync report is created it already contains tags (and also some additional state values that are not saved in regular save). This makes desync reports bigger, but hopefully it will be worth it.

We are also able to load a save in "heavy mod", which makes the simulation run in two instances and we compare full states every tick. This doesn't help too much to investigate desync reports, as it usualy takes seconds to run single tick on saves people send us, and it might take lot of time to desync again, or it might not desync unless somebody is actually playing the game. But when we find factory setup that caused the problem, we can recreate it on small (lets say 50x50 or 20x20) map and run that map in heavy mode very fast. (fff about heavy mode: https://www.factorio.com/blog/post/fff-63)

In the latest version we added command for headless server to start running in heavy mode. This will help us greatly with fixing desync loops on player connect.

[credomane]

I thought the recent mentions and changes to desync reports were the answer to this question of mine.

You guys are amazing!