multiplayer desync debugging

[db48x]

I've discovered a mod which is causing desyncs in multiplayer games. I found the archived desync report and diffed the two sets of files, but there were dozens of differences. Some of them seemed to be inconsequential (members of a table in a different order), others seemed more important. Has anyone else figured out how to use this information to track down a desync? I was thinking of writing a lua script to read the dat files and pretty printing them so they'd be more readable; has anyone else done that already?

Also, are the per-tick hashes created by serializing the data to a string and hashing the string, or by hashing the data structure more directly?

[DaveMcW]

Desync reports are for the developers, modders don't have enough information to use them.

The per-tick hash is calculated by hashing a bunch of c++ stuff you don't have access to.

[db48x]

Desync reports are for the developers, modders don't have enough information to use them.

Certainly we do. The game log is certainly pretty useless, but when a client detects a that it was the one that desynced and reconnects, it saves both copies of the game state to an archive directory. These files contain the map data as well as the mod data, all serialized in a way that's only moderately difficult to read. Diffing these two states should give us all we need in order to figure out what went wrong.

The .dat files are technically binary files, but the format is quite simple. First, four bytes which are probably a magic number, or perhaps are the overall length of the file (I didn't check). This is followed by a bunch of strings. Each string is preceded by a 4-byte integer to give the length of the string (most likely in bytes). In the script data the strings are in pairs, first the mod name then the serialized data for that mod. To save a few bytes this data is not pretty-printed, with each value on a new line properly indented from the last, so it's not very easy to read. A simple tool which extracts the values and pretty-prints them would make diffing much, much easier.

The per-tick hash is calculated by hashing a bunch of c++ stuff you don't have access to.

This wouldn't matter even if I were trying to recalculate the hash, since all of the information the hash could be based on is presumably contained in the game state uploaded to a connecting client. Still, I only care about what information is relevant to the hash. For example, does {x:42,y:42} have the same hash value as {y:42,x:42}? These lua objects are equivalent, but the order in which the properties have been serialized doesn't match.

[orzelek]

I'd recommend checking if your mod can cause table order to change - this is actually quite important difference.
I don't think you need a hash really - in theory only stuff you need is to make a diff between your mod data (since you proven that it causes the problem) and see how processing of this data is written to find out why it differs.

[db48x]

Indeed. Actually, I should be clearer: I _think_ I know which mod caused it, but diffing the files showed a dozen differences spread out across a dozen mods. I was hoping to eliminate some of those as irrelevant; it seems unlikely for a dozen things to have gone wrong all in the same tick.

What I need to do next is run two clients locally (or a server and client) and wait for or cause the desync, wile using just the most-suspect mod. That'll make diffing easier, if nothing else. Then I'll write a script to pretty-print the content of the data files.