Make desync reports helpful

[Optera]

When developing mods for MP hunting down desyncs is a near impossible task.

While normal errors have to the point error messages like line 123 expected table got nil.
Desyncs just tell that some crc doesnt match another crc and attach both server and client side save of up to several hundred MB.

Why not tell us which global table and subtable(s) caused the crc error and at what line in which mod the game encountered the desync.

[DaveMcW]

That's what /toggle-heavy-mode is for.

[bobingabout]

Error on line 123, expected table got nil.

Line 123 is an if statement, followed by a 5 part block of comparisons, many of which are function calls... Which one of these erred, and what is the variable name?

Yeah, it's not even desync reports, but error reports in general that could use more information.

That's what /toggle-heavy-mode is for.

I don't know what this is.

[eradicator]

Added /toggle-heavy-mode command. It can be used to generate files that help us to investigate server in a state where all new players get a desync loop.

Though reddit says --heavy and --complete are startup parameters. ¯\_(ツ)_/¯

Error on line 123, expected table got nil.

Line 123 is an if statement, followed by a 5 part block of comparisons, many of which are function calls... Which one of these erred, and what is the variable name?

That should still be line accurate to the specific if clause that failed. At the very least it's not in one of the functions called, or the error would be thrown in that function. Could still be caused by the return value of the function ofc P;. If i know the approximate location i just use debug.traceback to get a more complete view of the functions involved.

[Optera]

I critisized how desync reports don't even contain the line number.
As you said if I know the line and table throwing an error or desync i can use serpent.dump or serpent.block to find out what's wrong myself.
Would be great if the game itself did those in the error though.

[bobingabout]

whatever this heavy mode is... should probably be the default. cut down error messages are pointless, they might as well just say "An error happened."
Most bugs I end up fixing are just from a screenshot someone sent me of the error message, trying to get more than that out of people can be like trying to draw blood from a stone, so if there's not much information in their screenshot, I don't get the information I need, and therefore can't fix their problem.

of course my mods are so popular that sometimes I have a whole army of players who are also modders point me not only to the exact line of code, but why it's broken, but that's not the point I'm trying to make.

[eradicator]

Btw i totally agree that the desync notification is almost completely useless for bugfixing right now. I had a desync bug once that i could only fix because i implemented it while playing an mp game, so i knew exactly what recent change must have introduced it. And i have no clue how to generate a "desync report" (which ofc presumes you know at least how to reproduce the desync).

[betrok]

As far as I understand "some crc doesnt match another crc" is exactly how desyncs look from the engine's point of view.
It is not possible to point to some line of mod code and the actual source of trouble can be in a million ticks in the past.

On the other hand, an easy way to get a brief description of the diffirence between the two saves will be handy.
Something like factorio diff save1 save2 which will result in the report

Code: Select all

tables global.foo.bar of some_mod differ
...
areas {{x1, y1}, {x2, y2}} differ

When the difference in the tables is rather insignificant, it will be nice to have a standard output of unix diff for indent stringified data.

Probably, devs have some tool for this, but I do not know if it is publicly available.

[eradicator]

As far as I understand "some crc doesnt match another crc" is exactly how desyncs look from the engine's point of view.
It is not possible to point to some line of mod code and the actual source of trouble can be in a million ticks in the past.

Hrng. I don't like when people point out totally obvious facts that destroy my dreamscapes :P. Though i think the crc is calculated at least once per tick, so the direct cause should always be in the last tick.
Diffs are a nice idea, but would require that the clients send each other the global tables after a desync, which can be anything between a few kilobytes and tens of megabytes. And the not-desynced instance would have to keep a copy of the table in memory until the transfer is completed. And it's not even guaranteed that the global table is the actual problem.

Would be nice if a dev could decend onto this thread and tell us if heavy mode has any relevance to this, or if there's some other modder-accessible way to do this.

[orzelek]

Did you take a look at full desync report?

If you spend few minutes you can find mod data tables in it and other things are also present. I haven't seen heavy mode desync I think so can't tell much about those.

[eradicator]

Did you take a look at full desync report?

If you spend few minutes you can find mod data tables in it and other things are also present. I haven't seen heavy mode desync I think so can't tell much about those.

Like stated above i don't even know how to get one (Luckily i haven't had too many desync issues yet...). Is it just in the log? Can you attach/pm me an example?

[orzelek]

Did you take a look at full desync report?

If you spend few minutes you can find mod data tables in it and other things are also present. I haven't seen heavy mode desync I think so can't tell much about those.

Like stated above i don't even know how to get one (Luckily i haven't had too many desync issues yet...). Is it just in the log? Can you attach/pm me an example?

You can find them in reports - I did look through few at some point for RSO desyncs.
Attaching them is tricky - they tend to be well into 100's of MB's in size.

Actual desync report contains save of both situations. If you extract the saves then using binary diff tools is helpful - you can find stuff like differences in mod global tables etc.

This thread contains a link to actual full desync report:
viewtopic.php?f=209&t=60764

Usually most interesting parts are in level with tags files in reference level and desynced level. Need some decent diff tool to compare those since they can be pretty big. In that report level with tags has 600MB+. There is also a script.dat file that contains lua tables of mods - in this case there is a difference here that I can't fully understand - it looks like binary value differes. Someone who knows more about how lua stores tables would need to look at that.

PS.
I belive that that report is what started this thread in first place

[betrok]

Though i think the crc is calculated at least once per tick, so the direct cause should always be in the last tick.

But the lua context itself is not included in the crc(at least I think so). Thus, you can spoil some variables(e.g. missinitialize them on join), and it will go unnoticed for a while.

[Rseding91]

Why not tell us which global table and subtable(s) caused the crc error and at what line in which mod the game encountered the desync.

Because as others have said: the game has no idea what caused the desync. It could be a mod, it could be a core bit of logic broken or it could be someone using cheat engine. There's no way to tell what triggered the desync just that it happened.

How you figure out what caused a desync is by doing what we do: https://www.reddit.com/r/factorio/comme ... ncs_looks/ You load up the desync report and pick through it until you find what went wrong then try to reverse-guess how it got into that state.

[Optera]

Even without knowing what exactly caused a crc mismatch, for useful desync reports the game should serpent dump/block all tables per mod from memory into the reports.
It would be extra nice if it'd also included a diff file between those dumps.

[Rseding91]

Even without knowing what exactly caused a crc mismatch, for useful desync reports the game should serpent dump/block all tables per mod from memory into the reports.
It would be extra nice if it'd also included a diff file between those dumps.

It's already there. Inside the zip file it contains script.dat of both mods which contains the mod data in exactly that format.

[Optera]

Thanks, that's good to know.