[0.16.51] [Solved] Desync with random generator

[ZlovreD]

Mod behaviour: Every time, when new chunk is generated - there is a chance what mod place tiles and/or entities on it.
To make development of new variations faster and easy i'm using blueprint strings with packed area in it and then placing it on the surface.

All is fine in SP, but desynced in MP.

I finded, what any of mine manipulations with [ItemStack]Blueprint lead to desync.
Usualy i'm create a chest with blueprint in it on separate surface, save reference to it in globals and then use it. Tried the same, but w/o saving reference in globals. All the same - desync when i importing stack in it.

Was tested each time on fresh map with all settings by default, except amount of water, trees, cliffs and aliens.

To reproduce - just run at any direction out of starting area until script not tried to generate new area upon fresh chunk. Or wait some time while the game not to start generate new chunks by itself.

I will be appreciated for any help or hints how to solve this.

[Bilka]

The "desyncs with mods" subforum is not for getting help to fix your mod. Moved to modding help.

[eradicator]

I will be appreciated for any help or hints how to solve this.

You solve it by dissecting your mod into tiny bits, until you can reliably on the press of a button reproduce the desync, or can at least supply a minimal example mod that shows the problem. Your codebase is far too large for anyone to casually browse through it and find desyncs.

A desync happens whenever you store and later use any value to a variable that is not in the table literally named "global". Just storing things in an abitrary global-scoped variable you created yourself doesn't work. If you use global variables you might have data "leaking" from one function into another when it should not, you therefore should use global variables only for functions or tables that do not change during runtime, everything else should be done with function parameters.

(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)

[Bilka]

I will be appreciated for any help or hints how to solve this.

Here is a hint:

Code: Select all

--
-- global variables
--

global.ZADV = global.ZADV or {}
global.ZADV.errors = global.ZADV.errors or {}
global.ZADV.InProcess = false
global.ZADV.ForceUnlock = false
global.ZADV.NextForceUnlock = 0 
global.ZADV.EventLock = 0
global.ZADV.debug = 0

global.ZADV.Color = { ... }

None of that is valid, don't access global outside of event handlers.

(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)

0.16 already limits it to 200.

[eradicator]

None of that is valid, don't access global outside of event handlers.

I was about to agree but read over data-lifecycle again,and:

Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.

So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!

(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)

0.16 already limits it to 200.

Interesting.

[Bilka]

Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.

So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!

Interesting, that means it won't even get reset by loading the file (which is what I thought would happen), so that should be completely fine. The more you know :)

[ZlovreD]

Here is a hint:

Code: Select all

--
-- global variables
--

global.ZADV = global.ZADV or {}
global.ZADV.errors = global.ZADV.errors or {}
global.ZADV.InProcess = false
global.ZADV.ForceUnlock = false
global.ZADV.NextForceUnlock = 0 
global.ZADV.EventLock = 0
global.ZADV.debug = 0

global.ZADV.Color = { ... }

None of that is valid, don't access global outside of event handlers.

Ok, no problem. But this is not provoking desync in my case. But i'll move it to on_init.
Desync happens in 100% tests after this code:

Code: Select all

local Blueprint = PrepareBlueprint()
if Blueprint and Blueprint.valid then
	
	Blueprint.import_stack(type(newarea.bp) == 'table' and newarea.bp[Rnd(1,#newarea.bp)] or newarea.bp)

About blueprint. I've tried preparing it with direct reference stored in globals and creating it each time.
Variant #1

Code: Select all

local function PrepareBlueprint()
	if not global.ZADV.blueprint then
		if not game.surfaces['ZADV_SURFACE'] then
			game.create_surface("ZADV_SURFACE",{width=3,height=3,peaceful_mode=true})
			debug("Creating operable surface")
		end
		local entity = game.surfaces['ZADV_SURFACE'].create_entity{name="wooden-chest", position={0,0}, force=game.forces["neutral"]}
		entity.insert{name="blueprint", count=1}
		debug("Creating operable entity")
		global.ZADV.blueprint = entity.get_inventory(defines.inventory.chest).find_item_stack("blueprint")
		debug("Creating operable blueprint")
	end
end

Variant #2

Code: Select all

local function PrepareBlueprint()
	if not game.surfaces['ZADV_SURFACE'] then
		game.create_surface("ZADV_SURFACE",{width=3,height=3,peaceful_mode=true})
		debug("Creating operable surface")
	end
	local entity = game.surfaces['ZADV_SURFACE'].create_entity{name="wooden-chest", position={0,0}, force=game.forces["neutral"]}
	entity.insert{name="blueprint", count=1}
	debug("Creating operable entity")
	local blueprint = entity.get_inventory(defines.inventory.chest).find_item_stack("blueprint")
	debug("Creating operable blueprint")
	return blueprint
end

(Btw, unrelated: i just heared that 0.17 might limit string length of "order" strings. Have fun preparing for that.)

0.16 already limits it to 200.

Heh.. =)

Code: Select all

local dump = serpent.dump(ZADV.Data)
local chunks = math.floor(#dump / 199)
data:extend({
	{
		type = "flying-text",
		name = "ZADV_DATA_C",
		time_to_live = 0,
		speed = 1,
		order = "".. chunks+1
	}
})
for i=0, chunks do
	local name = "ZADV_DATA_C_"..i
	data:extend({
	{
		type = "flying-text",
		name = name,
		time_to_live = 0,
		speed = 1,
		order = "".. dump:sub(i*199, (i+1)*199-1)
	}})
end

And as result (when all pices enabled):

Code: Select all

Data size: 691.738 Kb
Data chunks: 3706

Btw... Is there any easy way to parse desync reports (dat files)?

[eradicator]

Desync happens in 100% tests after this code:

Code: Select all

local Blueprint = PrepareBlueprint()
if Blueprint and Blueprint.valid then
	
	Blueprint.import_stack(type(newarea.bp) == 'table' and newarea.bp[Rnd(1,#newarea.bp)] or newarea.bp)

What event is that?
What is newarea?
Is Rnd() a desync-free random generator?

[ZlovreD]

What event is that?

on_chunk_generated

What is newarea?

array with bpstring, functions and properties

Is Rnd() a desync-free random generator?

Code: Select all

local function Rnd(min,max,adseed)

	min = min or 1
	max = max or 1
	adseed = adseed or game.tick
	
	global.ZADV.adseed = global.ZADV.adseed or 42
	global.ZADV.adseed = global.ZADV.adseed + adseed
	global.ZADV.adseed = global.ZADV.adseed < 0 and 0 - global.ZADV.adseed or global.ZADV.adseed
	global.ZADV.adseed = global.ZADV.adseed >= 2^32-1 and 1 or global.ZADV.adseed
	
	local seed = game.tick + floor(tonumber(tostring({}):sub(8,-4))) + adseed
	seed = seed < 0 and 0 - seed or seed
	seed = seed >= 2^32-1 and game.tick or seed
	
	if not global.ZADV.generator then 
		global.ZADV.generator = game.create_random_generator(seed)
	else
		global.ZADV.generator.re_seed(seed)
	end
	
	return math.min(max, math.max(min, floor(global.ZADV.generator(min, math.max(min, base(max + global.ZADV.adseed))) % max) ) )
end

There is no conspiracy by any way...
I'm just trying to do 3 task at the same time: working, rewriting and testing mod and checking a forum. =)

[orzelek]

Two small things:
1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.
2. Post the link to desync report somewhere and I can diff it to see if something pops out.

[ZlovreD]

1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.

I had come to this when starts receiving similar results inside the same game tick.

2. Post the link to desync report somewhere and I can diff it to see if something pops out.

Gimme a few mins, seems i found one of possible reason of desync...

[orzelek]

1. Why would you reseed the gen every time? It's not really necessary most likely and costs quite a lot of math time.

I had come to this when starts receiving similar results inside the same game tick.

2. Post the link to desync report somewhere and I can diff it to see if something pops out.

Gimme a few mins, seems i found one of possible reason of desync...

If you seed with tick you will get exactly same results on same tick Thats the determinism of rng.

[ZlovreD]

If you seed with tick you will get exactly same results on same tick Thats the determinism of rng.

Yeah, yeah... Nwm, it's leveled by empty table and in last line...

Fresh report:

[orzelek]

This might be of use:

DataDif.png (75.48 KiB) Viewed 3653 times

A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.

[ZlovreD]

A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.

Weird.. Random generators must be non-random for all instances of the mod?
Maybe is better to make what only one instance can handle events and operate with data?

Also, what is a point of "generator.re_seed(seed)" then?

Also my investigation points on what if event handler code execution time exceeds some limits (game tick, cpu step or something else) - desync occurs.
Digging further...

[orzelek]

A bit lazy so a screenshot - it's a diff of script.dat files showing differences between global table data for your mod most likely.
It would seem that different seeds are generated on both machines = massive desync due to different decisions. There are a lot of differences in level file that might be caused by this.

Weird.. Random generators must be non-random for all instances of the mod?
Maybe is better to make what only one instance can handle events and operate with data?

Also, what is a point of "generator.re_seed(seed)" then?

Also my investigation points on what if event handler code execution time exceeds some limits (game tick, cpu step or something else) - desync occurs.
Digging further...

It's a bit strange - but I think you need to check how do seeds that created.
If you use game tick and some part executes on tick earlier for some reason it would create different seeds. Rng's are deterministic and I'd assume two of them started from same seed would net same sequence. Unless your code manages to go through two similar branches and one of them is using more random numbers then the other.

[ZlovreD]

Ok. One small improvement cost me few days of headache.
No needs to be smarter than needs.

As result - Any kind of references to C may leads to desync ( in my case "tonumber(tostring({}))" as seed for randomizer)

Thanks to all for the help. Now i need to put all the pieces back together.

[orzelek]

Ok. One small improvement cost me few days of headache.
No needs to be smarter than needs.

As result - Any kind of references to C may leads to desync ( in my case "tonumber(tostring({}))" as seed for randomizer)

Thanks to all for the help. Now i need to put all the pieces back together.

Hmm I'm not sure but what does this code really do?
Doesn't it end up with using address of the table as seed? - that would cause instant desync.

[ZlovreD]

Hmm I'm not sure but what does this code really do?

Yep, it is convertion of reference address into a number.
And is works perfectly in SP.

[Nexela]

Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.

So...that bit should behave pretty much the same as setting default values via "global.key = global.key or value" in on_init. And as such it might not actually be invalid. If it was written like that intentionally (ok, not very likely :p) i might actually consider it clever api usage..... hm... think think thing.... but... it'll break when a new version introduces new values. *scratches head*. But then why are the values even persisted...?!

Interesting, that means it won't even get reset by loading the file (which is what I thought would happen), so that should be completely fine. The more you know

The problem with doing it outside of an event is this

edit:
global.a = 1

save and edit:
global.a = 2

load:
global.a == 1

This is because during init "global" is saved to the map and during load the data from "global" that is stored in the savefile overwrites whatever you have for global.
on_init and and on_configuration_changed are the best places to populate and edit your "global"