Friday Facts #340 - Deep desyncs

Regular reports on Factorio development.
User avatar
5thHorseman
Filter Inserter
Filter Inserter
Posts: 954
Joined: Fri Jun 10, 2016 11:21 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by 5thHorseman »

MakeItGraphic wrote:
Fri Mar 27, 2020 10:46 pm
5thHorseman wrote:
Fri Mar 27, 2020 1:54 pm
Drison wrote:
Fri Mar 27, 2020 1:40 pm
More NEW official content please, not just bugfixes for mods...
Yeah where's that new character gui you've been promising or how about some new animations or something geez.
This is sarcasm right?
It was. And it apparently fell right in the sar-chasm.
Factorio For Beginners Come along with me as I detail (hopefully) every single thing you can do in Factorio with a focus on explaining to beginners, in a lighthearted, slow paced Let's Play style.

theolderbeholder
Fast Inserter
Fast Inserter
Posts: 125
Joined: Wed Sep 20, 2017 5:45 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by theolderbeholder »

Stop that or be pun ished. SCNR. :lol:

Hiladdar
Fast Inserter
Fast Inserter
Posts: 189
Joined: Mon May 14, 2018 6:47 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Hiladdar »

Mods are the new content right now, at least until release 1.0. I think Webe is doing the right thing, but freezing incorporation of new content at this time, and focusing on cleaning up what is currently in the base game. I think putting some more polish on existing graphics and sounds makes the most sense at this time. Likewise, removing what does not work well, at the moment makes a lot of sense.

The optimum situation, in late September 2020, there should be a release 1.0, that does not introduce any module breaking functionality. What would be nice, is if a module worked Factorio 0.18.xx, it would automatically work in Factorio verions 1.0.0, without even having to modify the .json file. After everyone has sobered up, a few weeks after release of 1.0, lets's start the discussion on DLCs, version 1.1 or 2.0 of and of course new functionality and features.

Hiladdar

EnerJi
Inserter
Inserter
Posts: 33
Joined: Thu Aug 16, 2018 2:32 am
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by EnerJi »

What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.

Jap2.0
Smart Inserter
Smart Inserter
Posts: 2204
Joined: Tue Jun 20, 2017 12:02 am
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Jap2.0 »

EnerJi wrote:
Sun Mar 29, 2020 2:05 am
What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.
"Desynchronization" - how multiplayer works is that each player's computer gets a copy of all the other players' actions, and then simulates the game on its own. If not everyone simulates the game exactly identically, you get a desync. To prevent further side effects (and so that people know something went wrong, be it the developers or mod authors), whoever desynced gets kicked off the server and has to re-join.
There are 10 types of people: those who get this joke and those who don't.

raidho36
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Wed Jun 01, 2016 2:08 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by raidho36 »

Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code. You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did. Why wouldn't you check what the library is doing before using it? And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? It took me a couple of hours to write a human-readable debug memory dumper which had the same circular reference problem to deal with and it worked the first time, because I implemented it properly to begin with. Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.

EDIT: I actually checked my debug memory dumper, that's not quite what it's doing - the list is for objects that it's going to dump, it puts stuff it's gonna dump into the list and ignores everything else, but eventually it works through everything. I can actually post it even though nobody cares, let's be real.
Last edited by raidho36 on Sun Mar 29, 2020 6:34 am, edited 1 time in total.

ptx0
Filter Inserter
Filter Inserter
Posts: 296
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by ptx0 »

raidho36 wrote:
Sun Mar 29, 2020 6:03 am
Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code. You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did. Why wouldn't you check what the library is doing before using it? And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? It took me a couple of hours to write a human-readable debug memory dumper which had the same circular reference problem to deal with and it worked the first time, because I implemented it properly to begin with. Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.
yeah, they had a quick release recently because looking at the world in map view caused it to crash almost immediately. another one killed event handlers in mods. it's almost like they don't even TEST the builds. some of the old questions they asked on Stack Overflow indicate their 'rookie' status.
Last edited by ptx0 on Sun Mar 29, 2020 6:35 am, edited 1 time in total.

AndrewIRL
Fast Inserter
Fast Inserter
Posts: 237
Joined: Fri Mar 24, 2017 2:17 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by AndrewIRL »

EnerJi wrote:
Sun Mar 29, 2020 2:05 am
What's a desync? This FFF talks about them as if everyone should know what they are, but I've got no clue.
https://wiki.factorio.com/Desynchronization

AndrewIRL
Fast Inserter
Fast Inserter
Posts: 237
Joined: Fri Mar 24, 2017 2:17 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by AndrewIRL »

raidho36 wrote:
Sun Mar 29, 2020 6:03 am
Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing.
They sold 1.5M copies of this game. If the bug managed to survive undetected through 10s or 100s of millions of hours of game time it is not serious. Some of us played hundreds or thousands of hours without a single crash. I never personally encountered a crash until I started using mods.

If the Factorio team make rookie mistakes then the people that build most of the software products I've ever encountered are sub-rookie. I think the ranking must go:

Raidho36 - best programmer in the world
Google, Facebook, Microsoft - Passable
Factorio Team - Rookies
The remaining 99.9% of programmers - Sub Rookie

I guess that puts me in the Sub-Rookie class with everyone else.

raidho36
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Wed Jun 01, 2016 2:08 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by raidho36 »

AndrewIRL wrote:
Sun Mar 29, 2020 6:48 am
If the bug managed to survive undetected through 10s or 100s of millions of hours of game time it is not serious.
Some of us played hundreds or thousands of hours without a single crash. I never personally encountered a crash until I started using mods.
"Desync is not a serious bug." -AndrewIRL 2020.
"You're playing it wrong." -AndrewIRL 2020.

On gamedev forums I never stop telling rookies that the game code doesn't need to be good, it just needs to function. I do it as to encourage people to actually make the game instead of re-writing the same code over and over as they learn the ropes. For professional programmers though, the bar is set very high - needless to say you lose your rookie pass.

Bilka
Factorio Staff
Factorio Staff
Posts: 2481
Joined: Sat Aug 13, 2016 9:20 am
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Bilka »

raidho36 wrote:
Sun Mar 29, 2020 6:03 am
Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. And after that, why wouldn't you modify it to write down values immediately and mark them as "visited" instead of doing this multi-pass roundabout whatever-the-heck? Sure, I'm the one to talk, I only ever finished a handful my projects, but that's because I'm a lazy bum with no motivation, not because I suck at programming - not with 15 years of experience I'm not - so you don't need to take this with a grain of salt, face value is good.
Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.

raidho36
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Wed Jun 01, 2016 2:08 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by raidho36 »

Bilka wrote:
Sun Mar 29, 2020 8:37 am
Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.
Here's a file. I appreciate your sarcasm, it's not really funny but it's a much needed to everyone sobering reminder that they're not some programming Jesus.

EDIT: Wow I haven't used that thing in a while. The loader function loads debug data, not restores memory from debug dump. It was for debug dump analysis tool and naturally it's not supposed to return the program into crashing state, but it could've been named more appropriately. Oh well.

EDIT2: Well it kinda loads circular objects anyway, I mean that's the point. It's just it loads them into debug data format, not directly into memory as actual raw data.
Attachments
memorydump.lua
(11.04 KiB) Downloaded 23 times

Bilka
Factorio Staff
Factorio Staff
Posts: 2481
Joined: Sat Aug 13, 2016 9:20 am
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Bilka »

raidho36 wrote:
Sun Mar 29, 2020 8:52 am
Bilka wrote:
Sun Mar 29, 2020 8:37 am
Super-uber-giga-noob here, only 3 years of experience. Can you explain me how you write down circular references immediately? I could imagine how to do it while simply dumping, but not how to make that also loadable.
Here's a file. I appreciate your sarcasm, it's not really funny but it's a much needed to everyone sobering reminder that they're not some programming Jesus.

EDIT: Wow I haven't used that thing in a while. The loader function loads debug data, not restores memory from debug dump. It was for debug dump analysis tool and naturally it's not supposed to return the program into crashing state, but it could've been named more appropriately. Oh well.
I'm a wiki admin, I have nothing to do with the internal Lua workings, just to alleviate any concern about my lacking experience. Your file doesn't work with Factorio because it uses io. Could you provide examples of what a circular dump looks like? I'm seriously trying to learn here.
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.

raidho36
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Wed Jun 01, 2016 2:08 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by raidho36 »

Bilka wrote:
Sun Mar 29, 2020 9:04 am
I'm a wiki admin, I have nothing to do with the internal Lua workings, just to alleviate any concern about my lacking experience. Your file doesn't work with Factorio because it uses io. Could you provide examples of what a circular dump looks like? I'm seriously trying to learn here.
You could simply monkey-patch it by pasting this code at the top:

Code: Select all

io = {
    open = function ( ) return
        {
        write = function ( self, txt ) print ( txt ) end,
        close = function ( ) end,
        lines = function ( )
            return function ( ) end
        end
        }
    end
}
Hopefully the "lines" iterator works but if not, just comment out that loop. Actually, I have to hope that any of this works because I didn't test it and never wrote it before to know for a fact.

In my dumping script it marks the tables it had already dumped and then simply ignores them when they come up on the dumping list again due to circular reference. Note that instead of following the references immediately, it puts these tables into a list for further dumping. In a pseudocode it's something like this:

Code: Select all

function dumpobjects ( list )
    local ignore =  { }
    while #list > 0 do
        local object = list:pop ( )
        if not ignore[ object ] then
            ignore[ object ] = true
            for k, v in pairs ( object ) do
                if type ( v ) == 'table' then
                    list:push ( )
                    dumpreference ( k, v )
                else
                    dumpvalue ( k, v )
                end
            end
        end
    end
end
Then again, my script has to explore the entire memory from a singular starting point (the stack), if Factorio has a global list of all entities which can be used with a numeric for-loop instead, then no insertion, deletion and ignoring is needed because you don't actually have to follow anywhere, it should work just fine by default, circular references and all. Which now that I think of it, it makes me somewhat more baffled that this problem even existed at all.

posila
Factorio Staff
Factorio Staff
Posts: 4509
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by posila »

raidho36 wrote:
Sun Mar 29, 2020 6:03 am
You know it's not an instance of having good hindsight when you see a problem description in the blog post and you immediately can tell what's wrong and how to fix it without even scrolling down to see what you actually did.
raidho36 wrote:
Sun Mar 29, 2020 9:35 am
Then again, my script has to explore the entire memory from a singular starting point (the stack), if Factorio has a global list of all entities which can be used with a numeric for-loop instead, then no insertion, deletion and ignoring is needed because you don't actually have to follow anywhere, it should work just fine by default, circular references and all. Which now that I think of it, it makes me somewhat more baffled that this problem even existed at all.
Could you please describe in your own words, what was the problem and how it was fixed?
raidho36 wrote:
Sun Mar 29, 2020 6:03 am
You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing. It's far from the first time you do something this incredibly shortsighted and then it comes back to bite you in the rear, and that's only going by your own blogs without ever seeing a line of your code.
raidho36 wrote:
Sun Mar 29, 2020 6:59 am
On gamedev forums I never stop telling rookies that the game code doesn't need to be good, it just needs to function. I do it as to encourage people to actually make the game instead of re-writing the same code over and over as they learn the ropes. For professional programmers though, the bar is set very high - needless to say you lose your rookie pass.
Would you be also so kind to elaborate on that? Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?

raidho36
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Wed Jun 01, 2016 2:08 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by raidho36 »

The problem was that you used a faulty serialization library and the solution was that you zip tied the parts that didn't quite come together. I would've used a better serialization library. I mean, I just looked at its source code and immediately noped out. That thing is horrendous, and I can't imagine it runs particularly fast. And the deserialization code, it straight up uses loadstring. Wow. You can't make this up.

Do you need your networking data to be a pretty printed text? Maybe consider making a gzipped binary string instead? I assume it's gzipped before it's sent through the network, but do store the stuff as 32 bit (or less) floats and integers, not as text.

Rookie mistakes my ass, you guys need to go back to the college.
posila wrote:
Sun Mar 29, 2020 12:36 pm
Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?
Rookies get a rookie pass. Professionals - don't.
Last edited by raidho36 on Sun Mar 29, 2020 1:40 pm, edited 1 time in total.

User avatar
Oktokolo
Filter Inserter
Filter Inserter
Posts: 856
Joined: Wed Jul 12, 2017 5:45 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Oktokolo »

raidho36 wrote:
Sun Mar 29, 2020 6:03 am
Wow. You guys just keep making rookie's mistakes when it comes to programming, it's embarrassing.
ptx0 wrote:
Sun Mar 29, 2020 6:33 am
yeah, they had a quick release recently because looking at the world in map view caused it to crash almost immediately.
Wube really should have outsourced making the engine to you two. Obviously, having it done by developers who never let any testable bug slip though is better, than letting actual humans write the code. Humans err, but you two obviously don't.

posila
Factorio Staff
Factorio Staff
Posts: 4509
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by posila »

raidho36 wrote:
Sun Mar 29, 2020 1:31 pm
The problem was that you used a faulty serialization library and the solution was that you zip tied the parts that didn't quite come together. I would've used a better serialization library. I mean, I just looked at its source code and immediately noped out. That thing is horrendous, and I can't imagine it runs particularly fast. And the deserialization code, it straight up uses loadstring. Wow. You can't make this up.

Do you need your networking data to be a pretty printed text? Maybe consider making a gzipped binary string instead? I assume it's gzipped before it's sent through the network, but do store the stuff as 32 bit (or less) floats and integers, not as text.
Well, you are just making broad assumptions about how things work in Factorio and then just call your imagined implementation as "shortsighted mistake". The lua serialization is used only for saving state for mods and scenarios with custom scripts, because the scripts are written in Lua. If mod has a state it needs to preserve across save/load, it adds it to global table named "global" and that gets serialized to the save file. Serialization works fine, the problem is that while everything is a table in lua, it differentiates between tables and arrays, which behave slightly differently under some circumstances. This is problem for us, because of lock-step multiplayer - if client loaded map with a mod that has in its state table that should have been array, mod script might end up doing something different on client than on server resulting in desync. Except for being part of the save, lua serialization is not utilized in networking in any way.

If you have for example:

Code: Select all

local A = { "one", "two", "three", "four" }
A[1] = A
and you serailized A and then want to deserialize it, it might seem like perfectly valid solution for deserializer to do

Code: Select all

local Result = { nil, "two", "three", "four" }
-- fixups
Result[1] = Result
but this may cause Lua interpreter to decide Result should be a table not an array. To fix this, Bokid changed the deserialization to do

Code: Select all

local Result = { 0, "two", "three", "four" }
-- fixups
Result[1] = Result
which will give the first element a value, keeping consecutive indices starting from 1, making Lua interpreter recongnize it as an array.

I must admit, had I worked on Factorio back when this was added, and I was tasked to evaluate Lua interpreters and Lua serialization libs, I wouldn't have spotted this has poteintal to cause desyncs when multiplayer is added two years later. Should have finished my college degree :/
raidho36 wrote:
Sun Mar 29, 2020 1:31 pm
Rookie mistakes my ass, you guys need to go back to the college.
posila wrote:
Sun Mar 29, 2020 12:36 pm
Do I understand it correctly "rookies" are supposed to make things just work, but also not knowing all implementation details of every library one uses, so that they can predict potential issues it might cause months or years down the road in the future is a shortsighted rookie mistake?
Rookies get a rookie pass. Professionals - don't.
I did understand the line about professionals having high bar. But you didn't call it professional mistake, or just mistake ... you called it rookie mistake, so it's a mistake that rookies commonly do. But that's how you advise them to do it, so ... how come it's a mistake for rookies to do it that way?

Rseding91
Factorio Staff
Factorio Staff
Posts: 10781
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by Rseding91 »

I don't see how feeding the trolls is helpful here. If someone wants to actually be helpful they would be helpful. The internet is full of "internet programmers" who like to act like they are smarter than everyone else and yet they aren't the ones who are making Factorio: we (Wube) are.
If you want to get ahold of me I'm almost always on Discord.

posila
Factorio Staff
Factorio Staff
Posts: 4509
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Friday Facts #340 - Deep desyncs

Post by posila »

Rseding91 wrote:
Sun Mar 29, 2020 2:44 pm
I don't see how feeding the trolls is helpful here. If someone wants to actually be helpful they would be helpful. The internet is full of "internet programmers" who like to act like they are smarter than everyone else and yet they aren't the ones who are making Factorio: we (Wube) are.
Sorry :( I'll do my social distancing better.

Post Reply

Return to “News”

Who is online

Users browsing this forum: Bored_Mike, James K