[0.13.17][posila]Desync on headless server

This subforum contains all the issues which we already resolved.
Post Reply
Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

[0.13.17][posila]Desync on headless server

Post by Michi302 »

I'm having trouble with desyncs when playing on a headless server. It started two days ago on 0.13.15 and still happens.
~30 seconds after joining a desync occurs and it happens to everyone on the server all the time.
We tried removing any mods and tested it on vanilla, but the issue kept occurring.
I tried loading it locally and letting it run for a while before saving it and trying it again on the server, but that didn't help too.
Right now I'm out of ideas :/

Desync Report:
https://dl.dropboxusercontent.com/u/658 ... -47-02.zip

Please let me know if I can do anything to help!

Edit: The desync also happens on 0.13.17.
Last edited by Michi302 on Sun Aug 21, 2016 3:03 pm, edited 1 time in total.

Zuzek06
Manual Inserter
Manual Inserter
Posts: 1
Joined: Sat Aug 20, 2016 7:00 pm
Contact:

Re: [0.13.16] Desync on headless server

Post by Zuzek06 »

We have the same problem, but we never had any mods installed and don't have a headless server. We tested it in the lates Version (0.13.17) the following way: I hosted the map in LAN mode, my buddy connected via a virtual Tunngle network. Then we just stood still and waited for the desync to happen and sure enought, after a short waiting period (under 1 minute) it happened. See the desync report and the host's log file:

https://dl.dropboxusercontent.com/u/204 ... report.zip
https://dl.dropboxusercontent.com/u/204 ... o-host.log

Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Michi302 »

We managed to fix it for now by removing the last factory we built locally. Something in it must have caused it.

rfvgyhn
Burner Inserter
Burner Inserter
Posts: 18
Joined: Fri Apr 22, 2016 5:14 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by rfvgyhn »

I'm also experience a lot of desyncs on a vanilla 0.13.17 headless server. Map was generated on 0.13.16. It happens both when I'm the only one on the server and when there are two people on the server (haven't tried with more than two).

Sometimes it desyncs every now and then. Sometimes I get stuck in a desync loop and have to alt+f4 to quit.

I haven't noticed a pattern to the desyncs. It's seemingly random.

1. Non-desync-loop report
https://app.box.com/s/ntcqbmjr6ppkcmv69321h70bf3irhmnx

2. Last two desync-loop reports
https://app.box.com/s/sv2mabpfdu7wvz2fhf7gsjcpo8aahavf
https://app.box.com/s/09iq1dwowtp1g0yqrucrn7adwod654dj

Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Michi302 »

Michi302 wrote:We managed to fix it for now by removing the last factory we built locally. Something in it must have caused it.
Turned out that it isn't fixed. Desync loop again :( . Same as before - Doesn't matter if modded or not.

Deync Report:
https://dl.dropboxusercontent.com/u/658 ... -28-16.zip

It there something like a tool that can show us what exactly went out of sync so that we can remove it locally?

Loewchen
Global Moderator
Global Moderator
Posts: 8283
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Loewchen »

Michi302 wrote:
Michi302 wrote:We managed to fix it for now by removing the last factory we built locally. Something in it must have caused it.
Turned out that it isn't fixed. Desync loop again :( . Same as before - Doesn't matter if modded or not.

Deync Report:
https://dl.dropboxusercontent.com/u/658 ... -28-16.zip

It there something like a tool that can show us what exactly went out of sync so that we can remove it locally?
No, not even the game itself knows what went out of sync, it just hashes parts of the map and compares the hashes...
Is it always the last player that connected that desyncs?
Or is it mbe always the same player (and therefore pc)?

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.13.17] Desync on headless server

Post by Rseding91 »

Does it desync without the mods?
If you want to get ahold of me I'm almost always on Discord.

Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Michi302 »

Rseding91 wrote:Does it desync without the mods?
Yes, it also happens without the mods.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.13.17] Desync on headless server

Post by Rseding91 »

Michi302 wrote:
Rseding91 wrote:Does it desync without the mods?
Yes, it also happens without the mods.
I'm not able to reproduce the issue on my computer. Are there any actions/steps I need to do to make it happen?
If you want to get ahold of me I'm almost always on Discord.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by posila »

After inspecting all the provided desync reports, I can say that circuit networks is acting up.

In Michi302's reports the desync is caused by stack inserters with logistic condition in iron-gear factory station Kupferentladung.

In rfvgyhn's reports an inserter at Iron Out Two station desynced.

In Zuzek06's report, no entities are desynced, but electric network statistics are, so the desync was caught too late ...

I haven't managed to reproduce the desyncs myself, I'll let the games run overnight. In meantime I am trying to find the exact cause of desync by other means.

Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Michi302 »

Rseding91 wrote:
Michi302 wrote:
Rseding91 wrote:Does it desync without the mods?
Yes, it also happens without the mods.
I'm not able to reproduce the issue on my computer. Are there any actions/steps I need to do to make it happen?
I only have to join a game with this map and it will desync after around ~90 seconds and get into a desync loop then.
posila wrote:After inspecting all the provided desync reports, I can say that circuit networks is acting up.

In Michi302's reports the desync is caused by stack inserters with logistic condition in iron-gear factory station Kupferentladung.

In rfvgyhn's reports an inserter at Iron Out Two station desynced.

In Zuzek06's report, no entities are desynced, but electric network statistics are, so the desync was caught too late ...

I haven't managed to reproduce the desyncs myself, I'll let the games run overnight. In meantime I am trying to find the exact cause of desync by other means.
I have removed all the stack inserters of the iron-grear factory and I have had only two desyncs during a 10 minutes test but no desync loop. So this already helped me :) Thanks alot. I have attached the two desync reports, no idea whether they are caused by the same issue.

https://dl.dropboxusercontent.com/u/658 ... -35-52.zip
https://dl.dropboxusercontent.com/u/658 ... -43-03.zip

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.13.17] Desync on headless server

Post by Rseding91 »

Are you still able to reproduce the desync loop after restarting the server?

If so, can you please upload the server save file (not in a desync report)?
If you want to get ahold of me I'm almost always on Discord.

Michi302
Burner Inserter
Burner Inserter
Posts: 7
Joined: Tue Aug 16, 2016 11:50 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by Michi302 »

Rseding91 wrote:Are you still able to reproduce the desync loop after restarting the server?

If so, can you please upload the server save file (not in a desync report)?
Okay - For the latest desync-report-2016-08-24-16-43-03:
https://dl.dropboxusercontent.com/u/658 ... save22.zip

After ~3 minutes a desync occurs. This is reproducible. If I start the server with this save and join and then just stand there and do nothing then after some minutes the desync happens.

Do you want the older saves with the desync of the stack inserters at the iron-gear factory too? (I don't know if the latest desync is caused by the same problem as the older ones)

EDIT:
Oh and right now with our newest save I haven't had any desycs more since the last two happened.

rfvgyhn
Burner Inserter
Burner Inserter
Posts: 18
Joined: Fri Apr 22, 2016 5:14 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by rfvgyhn »

I have restarted the server many times and I still get desyncs.

Here is my normal save file. This one is newer than the last desync report I uploaded. I still have iron out 2 and still regularly get desyncs. When I get off work, I'll dismantle iron out 2 and see if that helps at all. Actually, I'll probably end up dismantling all of my train loaders as they all use the same circuit network to balance the loading.

https://app.box.com/s/3j59ajiqiuufz2f80wgsm8lqjgl2q7z0

EDIT: Well, the inserters connected to the circuit network definitely seems to be the issue. I set all of my inserters to "none" and haven't had a desync for the hour that I've been playing. The desyncs would occur every couple minutes before the change.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.13.17] Desync on headless server

Post by posila »

Thank you! I managed to reproduce the desync on my computer with the latest save you provided.

rfvgyhn
Burner Inserter
Burner Inserter
Posts: 18
Joined: Fri Apr 22, 2016 5:14 pm
Contact:

Re: [0.13.17][posila]Desync on headless server

Post by rfvgyhn »

I see this bug fix has been included in the 0.13.19 release. Will it also be included in the next 0.14 experimental release?

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.13.17][posila]Desync on headless server

Post by posila »

I believe your desync was solved in 0.13.19, but Michi302's was not.
All fixes that go to 0.13 will eventually end up in 0.14 ... this one should be in 0.14.2

rfvgyhn
Burner Inserter
Burner Inserter
Posts: 18
Joined: Fri Apr 22, 2016 5:14 pm
Contact:

Re: [0.13.17][posila]Desync on headless server

Post by rfvgyhn »

Awesome. Thanks. :)

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.13.17][posila]Desync on headless server

Post by posila »

After some experimentation with Zuzek06's map I got some more desyncs, which led me to setup displayed on attached image.
Basically two stack inserters taking from same belt tile while fast-inserter puts items onto the same tile. That exactly same setup I saw to cause desync in Michi's save, but there the stack-inserters had logistic conditions and I assumed that was what caused the desyncs. Now it is clear that it is caused by inserter themselves. We found the issue and are working on a fix.

In addition to that I found out that Michi's map uses same station design as rfvgyhn's map, so that explains why it kept desyncing when Michi removed the bad behaving inserters.

posila
Factorio Staff
Factorio Staff
Posts: 5201
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.13.17][posila]Desync on headless server

Post by posila »

The issue should be fixed in 0.14.2.
Thank you all for your help with investigation of this desync.

Post Reply

Return to “Resolved Problems and Bugs”