[0.16.51] [Solved] Desync with random generator
[0.16.51] [Solved] Desync with random generator
Mod behaviour: Every time, when new chunk is generated - there is a chance what mod place tiles and/or entities on it.
To make development of new variations faster and easy i'm using blueprint strings with packed area in it and then placing it on the surface.
All is fine in SP, but desynced in MP.
I finded, what any of mine manipulations with [ItemStack]Blueprint lead to desync.
Usualy i'm create a chest with blueprint in it on separate surface, save reference to it in globals and then use it. Tried the same, but w/o saving reference in globals. All the same - desync when i importing stack in it.
Was tested each time on fresh map with all settings by default, except amount of water, trees, cliffs and aliens.
To reproduce - just run at any direction out of starting area until script not tried to generate new area upon fresh chunk. Or wait some time while the game not to start generate new chunks by itself.
I will be appreciated for any help or hints how to solve this.
To make development of new variations faster and easy i'm using blueprint strings with packed area in it and then placing it on the surface.
All is fine in SP, but desynced in MP.
I finded, what any of mine manipulations with [ItemStack]Blueprint lead to desync.
Usualy i'm create a chest with blueprint in it on separate surface, save reference to it in globals and then use it. Tried the same, but w/o saving reference in globals. All the same - desync when i importing stack in it.
Was tested each time on fresh map with all settings by default, except amount of water, trees, cliffs and aliens.
To reproduce - just run at any direction out of starting area until script not tried to generate new area upon fresh chunk. Or wait some time while the game not to start generate new chunks by itself.
I will be appreciated for any help or hints how to solve this.
- Attachments
-
- mod-list.json
- (338 Bytes) Downloaded 47 times
-
- ZAdventure_1.17.0.zip
- Unpublished version, what i'm working with.
- (6.95 MiB) Downloaded 54 times
-
- desync-report-2018-08-17_06-22-11.zip
- (4.46 MiB) Downloaded 41 times
Last edited by ZlovreD on Fri Aug 17, 2018 8:20 pm, edited 2 times in total.
Re: [0.16.51] Desync with entity property operations.
The "desyncs with mods" subforum is not for getting help to fix your mod. Moved to modding help.
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.
- eradicator
- Smart Inserter
- Posts: 5206
- Joined: Tue Jul 12, 2016 9:03 am
- Contact:
Re: [0.16.51] Desync with entity property operations.
You solve it by dissecting your mod into tiny bits, until you can reliably on the press of a button reproduce the desync, or can at least supply a minimal example mod that shows the problem. Your codebase is far too large for anyone to casually browse through it and find desyncs.ZlovreD wrote:I will be appreciated for any help or hints how to solve this.
A desync happens whenever you store and later use any value to a variable that is not in the table literally named "global". Just storing things in an abitrary global-scoped variable you created yourself doesn't work. If you use global variables you might have data "leaking" from one function into another when it should not, you therefore should use global variables only for functions or tables that do not change during runtime, everything else should be done with function parameters.
(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Re: [0.16.51] Desync with entity property operations.
Here is a hint:ZlovreD wrote:I will be appreciated for any help or hints how to solve this.
None of that is valid, don't access global outside of event handlers.control.lua wrote:Code: Select all
-- -- global variables -- global.ZADV = global.ZADV or {} global.ZADV.errors = global.ZADV.errors or {} global.ZADV.InProcess = false global.ZADV.ForceUnlock = false global.ZADV.NextForceUnlock = 0 global.ZADV.EventLock = 0 global.ZADV.debug = 0 global.ZADV.Color = { ... }
0.16 already limits it to 200.eradicator wrote:(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.
- eradicator
- Smart Inserter
- Posts: 5206
- Joined: Tue Jul 12, 2016 9:03 am
- Contact:
Re: [0.16.51] Desync with entity property operations.
I was about to agree but read over data-lifecycle again,and:Bilka wrote: None of that is valid, don't access global outside of event handlers.
So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!Data Lifecycle wrote:Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.
Interesting.Bilka wrote:0.16 already limits it to 200.eradicator wrote:(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Re: [0.16.51] Desync with entity property operations.
Interesting, that means it won't even get reset by loading the file (which is what I thought would happen), so that should be completely fine. The more you know :)eradicator wrote:So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!Data Lifecycle wrote:Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.
Re: [0.16.51] Desync with entity property operations.
Ok, no problem. But this is not provoking desync in my case. But i'll move it to on_init.Bilka wrote: Here is a hint:None of that is valid, don't access global outside of event handlers.control.lua wrote:Code: Select all
-- -- global variables -- global.ZADV = global.ZADV or {} global.ZADV.errors = global.ZADV.errors or {} global.ZADV.InProcess = false global.ZADV.ForceUnlock = false global.ZADV.NextForceUnlock = 0 global.ZADV.EventLock = 0 global.ZADV.debug = 0 global.ZADV.Color = { ... }
Desync happens in 100% tests after this code:
Code: Select all
local Blueprint = PrepareBlueprint()
if Blueprint and Blueprint.valid then
Blueprint.import_stack(type(newarea.bp) == 'table' and newarea.bp[Rnd(1,#newarea.bp)] or newarea.bp)
Variant #1
Code: Select all
local function PrepareBlueprint()
if not global.ZADV.blueprint then
if not game.surfaces['ZADV_SURFACE'] then
game.create_surface("ZADV_SURFACE",{width=3,height=3,peaceful_mode=true})
debug("Creating operable surface")
end
local entity = game.surfaces['ZADV_SURFACE'].create_entity{name="wooden-chest", position={0,0}, force=game.forces["neutral"]}
entity.insert{name="blueprint", count=1}
debug("Creating operable entity")
global.ZADV.blueprint = entity.get_inventory(defines.inventory.chest).find_item_stack("blueprint")
debug("Creating operable blueprint")
end
end
Code: Select all
local function PrepareBlueprint()
if not game.surfaces['ZADV_SURFACE'] then
game.create_surface("ZADV_SURFACE",{width=3,height=3,peaceful_mode=true})
debug("Creating operable surface")
end
local entity = game.surfaces['ZADV_SURFACE'].create_entity{name="wooden-chest", position={0,0}, force=game.forces["neutral"]}
entity.insert{name="blueprint", count=1}
debug("Creating operable entity")
local blueprint = entity.get_inventory(defines.inventory.chest).find_item_stack("blueprint")
debug("Creating operable blueprint")
return blueprint
end
Heh.. =)Bilka wrote:0.16 already limits it to 200.eradicator wrote:(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)
Code: Select all
local dump = serpent.dump(ZADV.Data)
local chunks = math.floor(#dump / 199)
data:extend({
{
type = "flying-text",
name = "ZADV_DATA_C",
time_to_live = 0,
speed = 1,
order = "".. chunks+1
}
})
for i=0, chunks do
local name = "ZADV_DATA_C_"..i
data:extend({
{
type = "flying-text",
name = name,
time_to_live = 0,
speed = 1,
order = "".. dump:sub(i*199, (i+1)*199-1)
}})
end
Code: Select all
Data size: 691.738 Kb
Data chunks: 3706
- eradicator
- Smart Inserter
- Posts: 5206
- Joined: Tue Jul 12, 2016 9:03 am
- Contact:
Re: [0.16.51] Desync with entity property operations.
What event is that?ZlovreD wrote: Desync happens in 100% tests after this code:Code: Select all
local Blueprint = PrepareBlueprint() if Blueprint and Blueprint.valid then Blueprint.import_stack(type(newarea.bp) == 'table' and newarea.bp[Rnd(1,#newarea.bp)] or newarea.bp)
What is newarea?
Is Rnd() a desync-free random generator?
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.
Re: [0.16.51] Desync with entity property operations.
on_chunk_generatederadicator wrote:What event is that?
array with bpstring, functions and propertieseradicator wrote:What is newarea?
eradicator wrote:Is Rnd() a desync-free random generator?
Code: Select all
local function Rnd(min,max,adseed)
min = min or 1
max = max or 1
adseed = adseed or game.tick
global.ZADV.adseed = global.ZADV.adseed or 42
global.ZADV.adseed = global.ZADV.adseed + adseed
global.ZADV.adseed = global.ZADV.adseed < 0 and 0 - global.ZADV.adseed or global.ZADV.adseed
global.ZADV.adseed = global.ZADV.adseed >= 2^32-1 and 1 or global.ZADV.adseed
local seed = game.tick + floor(tonumber(tostring({}):sub(8,-4))) + adseed
seed = seed < 0 and 0 - seed or seed
seed = seed >= 2^32-1 and game.tick or seed
if not global.ZADV.generator then
global.ZADV.generator = game.create_random_generator(seed)
else
global.ZADV.generator.re_seed(seed)
end
return math.min(max, math.max(min, floor(global.ZADV.generator(min, math.max(min, base(max + global.ZADV.adseed))) % max) ) )
end
I'm just trying to do 3 task at the same time: working, rewriting and testing mod and checking a forum. =)
Re: [0.16.51] Desync with entity property operations.
Two small things:
1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.
2. Post the link to desync report somewhere and I can diff it to see if something pops out.
1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.
2. Post the link to desync report somewhere and I can diff it to see if something pops out.
Re: [0.16.51] Desync with entity property operations.
I had come to this when starts receiving similar results inside the same game tick.orzelek wrote:1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.
Gimme a few mins, seems i found one of possible reason of desync...orzelek wrote:2. Post the link to desync report somewhere and I can diff it to see if something pops out.
Re: [0.16.51] Desync with entity property operations.
If you seed with tick you will get exactly same results on same tick Thats the determinism of rng.ZlovreD wrote:I had come to this when starts receiving similar results inside the same game tick.orzelek wrote:1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.Gimme a few mins, seems i found one of possible reason of desync...orzelek wrote:2. Post the link to desync report somewhere and I can diff it to see if something pops out.
Re: [0.16.51] Desync with entity property operations.
Yeah, yeah... Nwm, it's leveled by empty table and in last line...orzelek wrote:If you seed with tick you will get exactly same results on same tick Thats the determinism of rng.
Fresh report:
- Attachments
-
- desync-report-2018-08-17_20-03-53.zip
- (3.15 MiB) Downloaded 46 times
Re: [0.16.51] Desync with entity property operations.
This might be of use:
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.
A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.
Re: [0.16.51] Desync with entity property operations.
Weird.. Random generators must be non-random for all instances of the mod?orzelek wrote:A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.
Maybe is better to make what only one instance can handle events and operate with data?
Also, what is a point of "generator.re_seed(seed)" then?
Also my investigation points on what if event handler code execution time exceeds some limits (game tick, cpu step or something else) - desync occurs.
Digging further...
Re: [0.16.51] Desync with entity property operations.
It's a bit strange - but I think you need to check how do seeds that created.ZlovreD wrote:Weird.. Random generators must be non-random for all instances of the mod?orzelek wrote:A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.
Maybe is better to make what only one instance can handle events and operate with data?
Also, what is a point of "generator.re_seed(seed)" then?
Also my investigation points on what if event handler code execution time exceeds some limits (game tick, cpu step or something else) - desync occurs.
Digging further...
If you use game tick and some part executes on tick earlier for some reason it would create different seeds. Rng's are deterministic and I'd assume two of them started from same seed would net same sequence. Unless your code manages to go through two similar branches and one of them is using more random numbers then the other.
Re: [0.16.51] Desync with entity property operations.
Ok. One small improvement cost me few days of headache.
No needs to be smarter than needs.
As result - Any kind of references to C may leads to desync ( in my case "tonumber(tostring({}))" as seed for randomizer)
Thanks to all for the help. Now i need to put all the pieces back together.
No needs to be smarter than needs.
As result - Any kind of references to C may leads to desync ( in my case "tonumber(tostring({}))" as seed for randomizer)
Thanks to all for the help. Now i need to put all the pieces back together.
Re: [0.16.51] Desync with entity property operations.
Hmm I'm not sure but what does this code really do?ZlovreD wrote:Ok. One small improvement cost me few days of headache.
No needs to be smarter than needs.
As result - Any kind of references to C may leads to desync ( in my case "tonumber(tostring({}))" as seed for randomizer)
Thanks to all for the help. Now i need to put all the pieces back together.
Doesn't it end up with using address of the table as seed? - that would cause instant desync.
Re: [0.16.51] Desync with entity property operations.
Yep, it is convertion of reference address into a number.orzelek wrote:Hmm I'm not sure but what does this code really do?
And is works perfectly in SP.
Re: [0.16.51] Desync with entity property operations.
Bilka wrote:Interesting, that means it won't even get reset by loading the file (which is what I thought would happen), so that should be completely fine. The more you knoweradicator wrote:So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!Data Lifecycle wrote:Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.
The problem with doing it outside of an event is this
edit:
global.a = 1
save and edit:
global.a = 2
load:
global.a == 1
This is because during init "global" is saved to the map and during load the data from "global" that is stored in the savefile overwrites whatever you have for global.
on_init and and on_configuration_changed are the best places to populate and edit your "global"