[0.11.10] [cube] Multiplayer Rolling Desync

[Fatmice]

Issue continued from https://forums.factorio.com/forum/vie ... =11&t=7804

These desync is getting to be very annoying.
Savegame
http://www.mediafire.com/download/z36di ... desync.zip

Mods
http://www.mediafire.com/download/dkvco ... c_mods.zip

[roy7]

I think when a desync happens it might create a folder with debugging data in %appdata%/Factorio/? If so, including that and the factorio-current.log files might help the developer locate/fix it, if he can't reproduce the desyncs himself.

[kovarex]

I would like to just give you notice about the progress of this bug.

Cuba was working on investigating this bug for some time and found out, that the problem is related to the treefarm mod.
The problem is, that lua doesn't have deterministic hash tables when they are used in a certain way, so cube is now working on the modification of the lua implementation to work properly.
It should not only fix this one particular bug, but all the other probable problems with mods iterating over hash table contents.

If he can make it work, we might have it fixed for the next release.

[Fatmice]

Welp, this is very good to hear. I had also suspected the trees but had no way of reliably testing that theory. I suppose this problem surfaced in my game since I farm so much trees. Can't wait for the next release as that would allow me to return to my multiplayer game. I had some plans to build a complex for producing some pretty hefty amount of chips and plastic bars to support a crazy build with rocket defense.

[drs9999]

The problem is, that lua doesn't have deterministic hash tables when they are used in a certain way, so cube is now working on the modification of the lua implementation to work properly.

Could you please be more precisly about "used in a certain way"? Is there a (good) solution to avoid that?

[kovarex]

The problem is, that lua doesn't have deterministic hash tables when they are used in a certain way, so cube is now working on the modification of the lua implementation to work properly.

Could you please be more precisly about "used in a certain way"? Is there a (good) solution to avoid that?

The problem is, when the hash table is iterated. The order of the iteration is not well defined. We are avoiding this problem in our scripts by having list + hash table when we need to use it.

[Fatmice]

Hello, I would like to touch base on this bug. Have you guys found a way of fixing this yet? I see no mention of this bug on the recent release note of 0.11.18 nor is this bug marked resolved.

If this bug can not be easily resolved due to the random iteration behavior of using lua pairs, then would it be better to dictate the use of ipairs for iteration? I would like some clear direction on what is going on... It has been a while and I would like to get back to my multiplayer save. It has languished for some time now.

[kovarex]

The pair iterating has been fixed, but there was some other problem if I remember it right.

[Fatmice]

I see, so would it be much of a stretch to say that this problem will continue into 0.12 or will the coming stable release for 0.11 squish this bug? Could you elaborate on the additional problem(s) that was found? Did they exacerbate or attenuate this bug?

[Rseding91]

I see, so would it be much of a stretch to say that this problem will continue into 0.12 or will the coming stable release for 0.11 squish this bug? Could you elaborate on the additional problem(s) that was found? Did they exacerbate or attenuate this bug?

https://forums.factorio.com/forum/vie ... f=7&t=8919

[Fatmice]

So I play tested the multiplayer save with the same mods on 0.11.18. The desyncs are still there, but can be remedied by changing just pairs to ipairs on line 312 in control.lua of the Treefarm mod. This has a strange effect of stopping all tree growth. The trees do not actually grow but are .destroy() and recreated in a different state at the same location by game.createentity{} during certain game tick. I've encountered in my own lua scripting that .destroy() and .die() can also cause multiplayer desyncs, depending on how it is used.

I find it very strange that changing the pair to ipair iterator would change the mod behavior, but that is another issue.

[Fatmice]

The pair iterating has been fixed, but there was some other problem if I remember it right.

I see, so would it be much of a stretch to say that this problem will continue into 0.12 or will the coming stable release for 0.11 squish this bug? Could you elaborate on the additional problem(s) that was found? Did they exacerbate or attenuate this bug?

https://forums.factorio.com/forum/vie ... f=7&t=8919

So now that 0.11.20 is out and the long iteration crash is fixed, any chance this bug will pop next?

[Rseding91]

The pair iterating has been fixed, but there was some other problem if I remember it right.

I see, so would it be much of a stretch to say that this problem will continue into 0.12 or will the coming stable release for 0.11 squish this bug? Could you elaborate on the additional problem(s) that was found? Did they exacerbate or attenuate this bug?

https://forums.factorio.com/forum/vie ... f=7&t=8919

So now that 0.11.20 is out and the long iteration crash is fixed, any chance this bug will pop next?

I believe the "other problems" was a lockup bug related to iterating large tables (which was fixed in 0.11.19 or 0.11.20). So unless you're still getting a desync error this should be fixed. There's also the chance the mod is written incorrectly and that could cause a desync but the mod author would have to fix it in that instance.

[Fatmice]

Alright, this is the result of my tests with the 0.11.20 build using the same set of mods as in my first post in this thread.

Continuing from a 0.11.13 save - MP-279 - Surprisingly no desyncs even after an 1h of play. So I made a save called MP-280.
Continuing from a 0.11.20 save made from MP-279 - MP-280 - Rolling desyncs within 30 seconds.

Continuing from a 0.11.10 save - MP-270 - Immediate rolling desyncs. This is the save that I had provided in the first post. I made a save from this call MP-282.
Continuing from a 0.11.20 save made from MP-270 - MP-282 - Rolling desyncs within 30 seconds.

Conclusion? Desync errors are still there, only this time they are inconsistent. So the fixes introduced to correct game state divergence due to iteration may or may not have worked or simply exposed some other hidden state(s) that's not yet accounted for. Please note the game time of all the save games.

Here are the mods for your convenience.

I believe the "other problems" was a lockup bug related to iterating large tables (which was fixed in 0.11.19 or 0.11.20). So unless you're still getting a desync error this should be fixed. There's also the chance the mod is written incorrectly and that could cause a desync but the mod author would have to fix it in that instance.

I read the tree-farm mod lua code very thoroughly, at least for the version used in these save games, and there isn't anything "incorrectly" written other than extensive use of pair iterations. IMO, a mod should not be able to break a game and in the event that it does then that simply means the game code should be fixed to prevent such thing, not the other way around. Indeed, I believe the devs and I hold the same opinion.

[SvenDee]

We got the same Error in Multiplayer. Creating a new game works really good now ( had there some problems in older version ) But loading a save is impossible. It doesnt end into a crash, but continuing desyncs right after client joined the game.
It could be one of the mods. We are disabling and trying one by one. this took some time, playing, saving, testing. So i cant give a good feedback right now.

What we have done: ( same mods )
11.20. starting new map, played 9 hours. saved. want to continue, desyncs.
11.20. loading older map from 11.19. desyncs
11.20. loading map older then 11.18. no desync. saving this game and loading = 2x desyn early, then it was stable for the next hours.

Desync report:
https://www.dropbox.com/s/hy40irkxn5u8q ... 39.7z?dl=0

Mod folder:
https://www.dropbox.com/s/voseqhf546he3yi/mods.7z?dl=0
( we testet bob, yuoki and some smaller mods disabeld, still loading save desyncs)

Edit: happens also without treefarm activated. no desync after client joind, but desync after ca. 30 seconds. ( all on 11.20. generated or saved maps )
Edit: loading an older map ( our 100h game ) with 11.20. guess when we save this game, desync appears again. Edit: desyncs happens, but not that often as in the test before.
Edit: in our testing session with the old map we only had 2 desyncs early when we tried to open an assembly machine or changed something (stealchest to logistic chest). after that we had no desync problem for the next hours.

We have our next testing session on sunday with the new version.

Edit: short testing session with the old map which runs good on 11.20, 11.21 ends in rolling desyncs every 30 seconds. logs coming sunday

[SvenDee]

Testing session today:

tested base mod only = no desync in 10min, so it must be mod related.
testing in progress which mod it is. adding one by one

@ Fatmice please update your mods to newest version. Treefarm to AC and Lite to 0.1.4 Yuoki Industries to 0.2.27 i do not have the other mods. Some mods especially older version dont work correctly in multiplayer with the new version ( we had this problem on older mods like the tankwertz mod too, new version works fine now)

In our case i found out that it was the smart splitters 0.0.4 that isnt compatible to multiplayer and causes desync, but its really strange.
we still got desyncs.

[Fatmice]

@ Fatmice please update your mods to newest version. Treefarm to AC and Lite to 0.1.4 Yuoki Industries to 0.2.27 i do not have the other mods. Some mods especially older version dont work correctly in multiplayer with the new version ( we had this problem on older mods like the tankwertz mod too, new version works fine now)

Absolutely not. Updating the mod or mods to the newest version is like changing out the foundation of your house while you are trying to determine if there are any cracks (Yes, I'm fully aware that there are newer versions of Treefarm; Yuoki is not to blame since his core mod has no scripts). It is illogical. I have kept the game and mod as is since January 19th, the last day I played my multiplayer map, in order to keep all variables and hidden state as static as possible so that they can be ferreted out. The fact that a mod or some mod causes desyncs are issues inherent in the game code that must be resolved by the devs.

If you are suffering these rolling desyncs, which are annoying and aggravating, I would suggest the best thing to do is to keep everything as is and find out which mod is causing the problem through binary elimination. However, even if you figure out which mod it is, and can replicate the sequence of event leading to desyncs, it is still the devs who must fix the game code. The mod simply exposed a leak that must be patched.

To give you are historical perspective, I first reported this "bug" back in 0.11.8. It was purported to be fixed, then crept back in 0.11.10, and has been unresolved up till now, 0.11.21. Through the fourteen releases, the mods are kept the same as they were when the bug was first reported in 0.11.8. So what you do think have changed? Countless game codes have been added and removed during the four months that this bug has been around...So it is futile for you and any of us to try to figure out the root cause. Leave it to the devs as they alone have the necessary tools to step through the game state incrementally, which is the only way to see what caused the bifurcations in their deterministic code.

You should also observe that this bug is among a list of outstanding bugs, the oldest is from Jan 10, 2014, which is not to say that it is intractable. This bug is not easy to resolve and may continue to be around for some time. I duly hope that it is fixed before the stable candidate for 0.11.x is released. I'm sure the devs would like that to be so, but reality may not be so nice.

[DaveMcW]

However, even if you figure out which mod it is, and can replicate the sequence of event leading to desyncs, it is still the devs who must fix the game code. The mod simply exposed a leak that must be patched.

Sometimes it is the modder who must fix the mod code.

[Fatmice]

Sometimes it is the modder who must fix the mod code.

If that be so, then let the devs make such announcement. I've yet to see such a pronouncement, in this case.

[SvenDee]

we had again some testing yesterday and it was really strange. i tried some elimination by activating/deactivating mods, but, sometimes it works fine, sometimes desyncs directly without changing someting to previous test. so i cant say exactly which mod / maybe modcombo or the combo of factorio and mod causes the desync.

sadly i cant say more after the session yesterday.

also its sad that i dont have a save 11.18 or below so i cant test something. 11.18 was very stable for us and we played it many hours. at 11.19 we got desyncs and also cant create a new game.

Continuing from a 0.11.13 save - MP-279 - Surprisingly no desyncs even after an 1h of play. So I made a save called MP-280.
Continuing from a 0.11.20 save made from MP-279 - MP-280 - Rolling desyncs within 30 seconds.

can you make a test loading the 11.13. save into 11.18. saving the map, loading again? i guess you will have no rolling desyncs. So if there is a bug with the update that makes save corruption it must be changes in the 11.19 update that isnt fixed in the 11.20. also you can try the 11.21 version which has a bugfix:

Fixed potential save corruption if the game was saved while mousing over a mining drill.

maybe something like this happend to our saves in 11.19 / 11.20.

im testing a 11.18 map of a 3 player session in 11.21 right now. playing a bit. saving to a new state loading with 11.21.
11.18 save loading in 11.21 no desync = saved
saved game continuing > no desync

i cant reconstruate the error for now by just changing the version or changing the mods. maybe it only happens on some actions during a save like the one that was fixed.