Friday Facts #340 - Deep desyncs

[5thHorseman]

More NEW official content please, not just bugfixes for mods...

Yeah where's that new character gui you've been promising or how about some new animations or something geez.

This is sarcasm right?

It was. And it apparently fell right in the sar-chasm.

[theolderbeholder]

Stop that or be pun ished. SCNR.

[Hiladdar]

Mods are the new content right now, at least until release 1.0. I think Webe is doing the right thing, but freezing incorporation of new content at this time, and focusing on cleaning up what is currently in the base game. I think putting some more polish on existing graphics and sounds makes the most sense at this time. Likewise, removing what does not work well, at the moment makes a lot of sense.

The optimum situation, in late September 2020, there should be a release 1.0, that does not introduce any module breaking functionality. What would be nice, is if a module worked Factorio 0.18.xx, it would automatically work in Factorio verions 1.0.0, without even having to modify the .json file. After everyone has sobered up, a few weeks after release of 1.0, lets's start the discussion on DLCs, version 1.1 or 2.0 of and of course new functionality and features.

Hiladdar

[EnerJi]

What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.

[Jap2.0]

What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.

"Desynchronization" - how multiplayer works is that each player's computer gets a copy of all the other players' actions, and then simulates the game on its own. If not everyone simulates the game exactly identically, you get a desync. To prevent further side effects (and so that people know something went wrong, be it the developers or mod authors), whoever desynced gets kicked off the server and has to re-join.

[raidho36]

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code. You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did. Why wouldn't you check what the library is doing before using it? And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? It took me a couple of hours to write a human-readable debug memory dumper which had the same circular reference problem to deal with and it worked the first time, because I implemented it properly to begin with. Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.

EDIT: I actually checked my debug memory dumper, that's not quite what it's doing - the list is for objects that it's going to dump, it puts stuff it's gonna dump into the list and ignores everything else, but eventually it works through everything. I can actually post it even though nobody cares, let's be real.

[ptx0]

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code. You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did. Why wouldn't you check what the library is doing before using it? And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? It took me a couple of hours to write a human-readable debug memory dumper which had the same circular reference problem to deal with and it worked the first time, because I implemented it properly to begin with. Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.

yeah, they had a quick release recently because looking at the world in map view caused it to crash almost immediately. another one killed event handlers in mods. it's almost like they don't even TEST the builds. some of the old questions they asked on Stack Overflow indicate their 'rookie' status.

[AndrewIRL]

What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.

https://wiki.factorio.com/Desynchronization

[AndrewIRL]

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing.

They sold 1.5M copies of this game. If the bug managed to survive undetected through 10s or 100s of millions of hours of game time it is not serious. Some of us played hundreds or thousands of hours without a single crash. I never personally encountered a crash until I started using mods.

If the Factorio team make rookie mistakes then the people that build most of the software products I've ever encountered are sub-rookie. I think the ranking must go:

Raidho36 - best programmer in the world
Google, Facebook, Microsoft - Passable
Factorio Team - Rookies
The remaining 99.9% of programmers - Sub Rookie

I guess that puts me in the Sub-Rookie class with everyone else.

[raidho36]

If the bug managed to survive undetected through 10s or 100s of millions of hours of game time it is not serious.
Some of us played hundreds or thousands of hours without a single crash. I never personally encountered a crash until I started using mods.

"Desync is not a serious bug." -AndrewIRL 2020.
"You're playing it wrong." -AndrewIRL 2020.

On gamedev forums I never stop telling rookies that the game code doesn't need to be good, it just needs to function. I do it as to encourage people to actually make the game instead of re-writing the same code over and over as they learn the ropes. For professional programmers though, the bar is set very high - needless to say you lose your rookie pass.

[Bilka]

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.

Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.

[raidho36]

Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.

Here's a file. I appreciate your sarcasm, it's not really funny but it's a much needed to everyone sobering reminder that they're not some programming Jesus.

EDIT: Wow I haven't used that thing in a while. The loader function loads debug data, not restores memory from debug dump. It was for debug dump analysis tool and naturally it's not supposed to return the program into crashing state, but it could've been named more appropriately. Oh well.

EDIT2: Well it kinda loads circular objects anyway, I mean that's the point. It's just it loads them into debug data format, not directly into memory as actual raw data.

[Bilka]

Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.

Here's a file. I appreciate your sarcasm, it's not really funny but it's a much needed to everyone sobering reminder that they're not some programming Jesus.

EDIT: Wow I haven't used that thing in a while. The loader function loads debug data, not restores memory from debug dump. It was for debug dump analysis tool and naturally it's not supposed to return the program into crashing state, but it could've been named more appropriately. Oh well.

I'm a wiki admin, I have nothing to do with the internal Lua workings, just to alleviate any concern about my lacking experience. Your file doesn't work with Factorio because it uses io. Could you provide examples of what a circular dump looks like? I'm seriously trying to learn here.

[raidho36]

I'm a wiki admin, I have nothing to do with the internal Lua workings, just to alleviate any concern about my lacking experience. Your file doesn't work with Factorio because it uses io. Could you provide examples of what a circular dump looks like? I'm seriously trying to learn here.

You could simply monkey-patch it by pasting this code at the top:

Code: Select all

io = {
    open = function ( ) return
        {
        write = function ( self, txt ) print ( txt ) end,
        close = function ( ) end,
        lines = function ( )
            return function ( ) end
        end
        }
    end
}

Hopefully the "lines" iterator works but if not, just comment out that loop. Actually, I have to hope that any of this works because I didn't test it and never wrote it before to know for a fact.

In my dumping script it marks the tables it had already dumped and then simply ignores them when they come up on the dumping list again due to circular reference. Note that instead of following the references immediately, it puts these tables into a list for further dumping. In a pseudocode it's something like this:

Code: Select all

function dumpobjects ( list )
    local ignore =  { }
    while #list > 0 do
        local object = list:pop ( )
        if not ignore[ object ] then
            ignore[ object ] = true
            for k, v in pairs ( object ) do
                if type ( v ) == 'table' then
                    list:push ( )
                    dumpreference ( k, v )
                else
                    dumpvalue ( k, v )
                end
            end
        end
    end
end

Then again, my script has to explore the entire memory from a singular starting point (the stack), if Factorio has a global list of all entities which can be used with a numeric for-loop instead, then no insertion, deletion and ignoring is needed because you don't actually have to follow anywhere, it should work just fine by default, circular references and all. Which now that I think of it, it makes me somewhat more baffled that this problem even existed at all.

[posila]

You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did.

Then again, my script has to explore the entire memory from a singular starting point (the stack), if Factorio has a global list of all entities which can be used with a numeric for-loop instead, then no insertion, deletion and ignoring is needed because you don't actually have to follow anywhere, it should work just fine by default, circular references and all. Which now that I think of it, it makes me somewhat more baffled that this problem even existed at all.

Could you please describe in your own words, what was the problem and how it was fixed?

You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code.

On gamedev forums I never stop telling rookies that the game code doesn't need to be good, it just needs to function. I do it as to encourage people to actually make the game instead of re-writing the same code over and over as they learn the ropes. For professional programmers though, the bar is set very high - needless to say you lose your rookie pass.

Would you be also so kind to elaborate on that? Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?

[raidho36]

The problem was that you used a faulty serialization library and the solution was that you zip tied the parts that didn't quite come together. I would've used a better serialization library. I mean, I just looked at its source code and immediately noped out. That thing is horrendous, and I can't imagine it runs particularly fast. And the deserialization code, it straight up uses loadstring. Wow. You can't make this up.

Do you need your networking data to be a pretty printed text? Maybe consider making a gzipped binary string instead? I assume it's gzipped before it's sent through the network, but do store the stuff as 32 bit (or less) floats and integers, not as text.

Rookie mistakes my ass, you guys need to go back to the college.

Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?

Rookies get a rookie pass. Professionals - don't.

[Oktokolo]

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing.

yeah, they had a quick release recently because looking at the world in map view caused it to crash almost immediately.

Wube really should have outsourced making the engine to you two. Obviously, having it done by developers who never let any testable bug slip though is better, than letting actual humans write the code. Humans err, but you two obviously don't.

[posila]

The problem was that you used a faulty serialization library and the solution was that you zip tied the parts that didn't quite come together. I would've used a better serialization library. I mean, I just looked at its source code and immediately noped out. That thing is horrendous, and I can't imagine it runs particularly fast. And the deserialization code, it straight up uses loadstring. Wow. You can't make this up.

Do you need your networking data to be a pretty printed text? Maybe consider making a gzipped binary string instead? I assume it's gzipped before it's sent through the network, but do store the stuff as 32 bit (or less) floats and integers, not as text.

Well, you are just making broad assumptions about how things work in Factorio and then just call your imagined implementation as "shortsighted mistake". The lua serialization is used only for saving state for mods and scenarios with custom scripts, because the scripts are written in Lua. If mod has a state it needs to preserve across save/load, it adds it to global table named "global" and that gets serialized to the save file. Serialization works fine, the problem is that while everything is a table in lua, it differentiates between tables and arrays, which behave slightly differently under some circumstances. This is problem for us, because of lock-step multiplayer - if client loaded map with a mod that has in its state table that should have been array, mod script might end up doing something different on client than on server resulting in desync. Except for being part of the save, lua serialization is not utilized in networking in any way.

If you have for example:

Code: Select all

local A = { "one", "two", "three", "four" }
A[1] = A

and you serailized A and then want to deserialize it, it might seem like perfectly valid solution for deserializer to do

Code: Select all

local Result = { nil, "two", "three", "four" }
-- fixups
Result[1] = Result

but this may cause Lua interpreter to decide Result should be a table not an array. To fix this, Bokid changed the deserialization to do

Code: Select all

local Result = { 0, "two", "three", "four" }
-- fixups
Result[1] = Result

which will give the first element a value, keeping consecutive indices starting from 1, making Lua interpreter recongnize it as an array.

I must admit, had I worked on Factorio back when this was added, and I was tasked to evaluate Lua interpreters and Lua serialization libs, I wouldn't have spotted this has poteintal to cause desyncs when multiplayer is added two years later. Should have finished my college degree :/

Rookie mistakes my ass, you guys need to go back to the college.

Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?

Rookies get a rookie pass. Professionals - don't.

I did understand the line about professionals having high bar. But you didn't call it professional mistake, or just mistake ... you called it rookie mistake, so it's a mistake that rookies commonly do. But that's how you advise them to do it, so ... how come it's a mistake for rookies to do it that way?

[Rseding91]

I don't see how feeding the trolls is helpful here. If someone wants to actually be helpful they would be helpful. The internet is full of "internet programmers" who like to act like they are smarter than everyone else and yet they aren't the ones who are making Factorio: we (Wube) are.

[posila]

I don't see how feeding the trolls is helpful here. If someone wants to actually be helpful they would be helpful. The internet is full of "internet programmers" who like to act like they are smarter than everyone else and yet they aren't the ones who are making Factorio: we (Wube) are.

Sorry I'll do my social distancing better.