Inconsistent entitytarget= values in desync reports?

[Uthrom]

While comparing script.dat files between server and client on a number of desync reports, it seems that the value of entitytarget= is inconsistent between server and client?

Should the value be predictably the same on server and client, or does it point to some memory location that mostly will differ between server and client?

Example seen in the attached screenshot.

The actual desync report can be seen here: https://www.dropbox.com/s/q49nmtiqv9dqt ... 4.zip?dl=0

[Klonan]

This difference doesnt matter, it will generally be different for each client

[Uthrom]

Thanks!

[Rseding91]

This difference doesnt matter, it will generally be different for each client

Generally be different for most desyncs. In a non-desynced game if the client and server make a save at the exact same tick they should be identical. However, those values are never the thing that *triggers* the desync report to be created - there's always something else that started it.

[rocifier]

I'm seeing the same differences consistently across a range of desync reports for comfy's Mountain mod. If I can rule out entitytarget values then the only differences are <force-data> contents having some binary data off by 1, and most of the time (but not always) <entity-count> and <active-entities> being up by 1 value on the server but not desynced client. There must be an easier way to narrow down this situation as we haven't been able to determine despite hundreds of desyncs, what the trigger is. Also, multiple code reviews and debug data and process of elimination have been unhelpful so far. Any suggestions for this specific situation?

[Klonan]

Any suggestions for this specific situation?

Have you been testing with heavy-mode ?

[posila]

When I can't find origin of desync in tagged desync report, I use mod to take screenshot of every chunk directly after load (for both desynced and reference saves), and then use file compare to filter out chunks that are the same and finally just go through screenshots in alternating order (reference, desync, referene, desync, ...) and search for significant differences with my eyes (there are visual only variations that are not saved, so not all differences in screenshots are desync, restarting the game fresh and with CIL parameter --global-seed=12345 helps to weed out lot of the random visual variations; CIL parameter --disable-prototype-history might help speed up game restart if your modpack is large). That sometimes helps me to find an entity that's possible seed of the desync and then I can reason about what could have caused the entity to behave differently on client and server. If you already narrowed down caused of desync to let's say difference in active entity count, it might be also helpful to enable "show-active-state" in debug settigns (F4) before taking the screenshots.

For inspiration, my debugging mod is attached, but it's not pretty (because it has some unrelated stuff in it).

michaltest_1.0.0.zip

(4.08 KiB) Downloaded 109 times

[rocifier]

Wow that is an interesting idea to compare screenshots. We have had some success since posting by "guessing" what the off-by-one might be and testing our hypothesis. We found that we need to properly destroy tags before resetting the map (creating a new surface). This has been a major breakthrough in reducing the number of desyncs following soft resets (our hypothesis). For some of the other less frequent desyncs I think the screenshot comparison may be more useful to help with those.

Personally I haven't used heavy mode... I wouldn't really know how to use it. Until we were able to reproduce the desyncs we had to play the mod for hours at a time until they "appeared". Reading a bit more about it, I'm not sure if heavy mode would have been useful for that situation either?

[Rseding91]

Chart tags?

[rocifier]

@posila I'm trying out your screenshot mod. In my instance when I load the map, it spits out 619 bitmaps in 31921_des folder. The problem is, all of the bmp files are at night, there are only a few variations and repeat screenshots of seemingly the same areas. I also tried setting cfg_screenshot_chunks to false and cfg_take_screenshot to true and it produced a set of dark pngs. cfg_screenshot_at_day is set to true. Has this happened to you before?

EDIT: cfg_screenshot_players set to true output non-black images for all players. This seems like a good enough place to start.
EDIT2: I found the cause of it not setting up daytime. It's hard coded to game.surfaces[1] but it seems that after the soft reset we create a new surface so it needed to be surfaces[2] to change the daylight (control.lua:55).

[rocifier]

Cool, after running the images through WinMerge and carefully examining the visual diff, I noticed what I initially thought were lighting changes to actually be different ore dropped on the ground. This happens when a player mines a rock, it spills ore in our map. Even though it's based on the random seed and I've set both seeds to be the same, there must be something else going on here. We also noticed the desync data was different inside <inventory> now. Hmmmm. The screenshot mod is helping to find the issue!

[Klonan]

Cool, after running the images through WinMerge and carefully examining the visual diff, I noticed what I initially thought were lighting changes to actually be different ore dropped on the ground. This happens when a player mines a rock, it spills ore in our map. Even though it's based on the random seed and I've set both seeds to be the same, there must be something else going on here. We also noticed the desync data was different inside <inventory> now. Hmmmm. The screenshot mod is helping to find the issue!

Can you share the code you are using around the random generator?

[rocifier]

Here is an example of the reference and desynced screenshots:

https://imgur.com/a/a7GIfbd

And here is the suspected code because of the screenshot:

https://gist.github.com/rocifier/9caee0 ... 614431163b

Code: Select all

--destroying and mining rocks yields ore -- load as last module
local math_random = math.random
local max_spill = 48

local rock_yield = {
	["rock-big"] = 1,
	["rock-huge"] = 2,
	["sand-rock-big"] = 1	
}

local rock_mining_chance_weights = {
	{"iron-ore", 25},
	{"copper-ore",17},
	{"coal",13},
	{"stone",9},
	{"uranium-ore",2}
}

local texts = {
	["iron-ore"] = {"Iron ore", {r = 200, g = 200, b = 180}},
	["copper-ore"] = {"Copper ore", {r = 221, g = 133, b = 6}},
	["uranium-ore"] = {"Uranium ore", {r= 50, g= 250, b= 50}},
	["coal"] = {"Coal", {r = 0, g = 0, b = 0}},
	["stone"] = {"Stone", {r = 200, g = 160, b = 30}},
}

local particles = {
	["iron-ore"] = "iron-ore-particle",
	["copper-ore"] = "copper-ore-particle",
	["uranium-ore"] = "coal-particle",
	["coal"] = "coal-particle",
	["stone"] = "stone-particle"
}
		
local ore_raffle = {}				
for _, t in pairs (rock_mining_chance_weights) do
	for x = 1, t[2], 1 do
		table.insert(ore_raffle, t[1])
	end			
end

local function create_particles(surface, name, position, amount, cause_position)	
	local direction_mod = (-100 + math_random(0,200)) * 0.0004
	local direction_mod_2 = (-100 + math_random(0,200)) * 0.0004
	
	if cause_position then
		direction_mod = (cause_position.x - position.x) * 0.025
		direction_mod_2 = (cause_position.y - position.y) * 0.025
	end
	
	for i = 1, amount, 1 do 
		local m = math_random(4, 10)
		local m2 = m * 0.005
		
		surface.create_entity({
			name = name,
			position = position,
			frame_speed = 1,
			vertical_speed = 0.130,
			height = 0,
			movement = {
				(m2 - (math_random(0, m) * 0.01)) + direction_mod,
				(m2 - (math_random(0, m) * 0.01)) + direction_mod_2
			}
		})
	end	
end

local function get_amount(entity)
	local distance_to_center = math.floor(math.sqrt(entity.position.x ^ 2 + entity.position.y ^ 2))
	
	local distance_modifier = 0.25
	local base_amount = 35
	local maximum_amount = 100
	if global.rocks_yield_ore_distance_modifier then distance_modifier = global.rocks_yield_ore_distance_modifier end
	if global.rocks_yield_ore_base_amount then base_amount = global.rocks_yield_ore_base_amount end
	if global.rocks_yield_ore_maximum_amount then maximum_amount = global.rocks_yield_ore_maximum_amount end
	
	local amount = base_amount + (distance_to_center * distance_modifier)
	if amount > maximum_amount then amount = maximum_amount end
	
	local m = (70 + math_random(0, 60)) * 0.01
	
	amount = math.floor(amount * rock_yield[entity.name] * m)
	if amount < 1 then amount = 1 end
		
	return amount
end

local function on_player_mined_entity(event)
	local entity = event.entity
	if not entity.valid then return end
	if not rock_yield[entity.name] then return end
	
	event.buffer.clear()
	
	local ore = ore_raffle[math_random(1, #ore_raffle)]	
	local player = game.players[event.player_index]
	
	local inventory = player.get_inventory(defines.inventory.character_main)
	if not inventory.can_insert({name = ore, count = 1}) then
		local e = entity.surface.create_entity({name = entity.name, position = entity.position})
		e.health = entity.health
		player.print("Inventory full.", {200, 200, 200})
		return
	end
			
	local count = get_amount(entity)
	local position = {x = entity.position.x, y = entity.position.y}
	
	player.surface.create_entity({name = "flying-text", position = position, text = "+" .. count .. " [img=item/" .. ore .. "]", color = {r = 200, g = 160, b = 30}})
	create_particles(player.surface, particles[ore], position, 64, {x = player.position.x, y = player.position.y})
	
	entity.destroy()
	
	if count > max_spill then
		player.surface.spill_item_stack(position,{name = ore, count = max_spill}, true)
		count = count - max_spill
		local inserted_count = player.insert({name = ore, count = count})
		count = count - inserted_count
		if count > 0 then
			player.surface.spill_item_stack(position,{name = ore, count = count}, true)
		end
	else			
		player.surface.spill_item_stack(position,{name = ore, count = count}, true)
	end	
end

local function on_entity_died(event)	
	local entity = event.entity
	if not entity.valid then return end	
	if not rock_yield[entity.name] then return end
	
	local surface = entity.surface
	local ore = ore_raffle[math_random(1, #ore_raffle)]
	local pos = {entity.position.x, entity.position.y}		
	create_particles(surface, particles[ore], pos, 16, false)
	
	if event.cause then
		if event.cause.valid then
			if event.cause.force.index == 2 or event.cause.force.index == 3 then
				entity.destroy()
				return
			end
		end
	end		
		
	local amount = get_amount(entity)
	amount = math.ceil(amount * 0.1)
	if amount > 16 then amount = 16 end	
	entity.destroy()		
	surface.spill_item_stack(pos,{name = ore, count = amount}, true)
end

local event = require 'utils.event'
event.add(defines.events.on_entity_died, on_entity_died)	
event.add(defines.events.on_player_mined_entity, on_player_mined_entity)

[rocifier]

Chart tags?

yes

[posila]

Looking at the screenshot, even the terrain tile under the ore items is different, which might point to actual desync happening much sooner. Do you do custom map-gen?

[rocifier]

Looking at the screenshot, even the terrain tile under the ore items is different, which might point to actual desync happening much sooner. Do you do custom map-gen?

Yes. And things start going wrong from when we soft-reset the map. I captured a brief heavy-mode recording which started from the first tick after a soft reset:

https://drive.google.com/open?id=1oVlHM ... HKoSixXVeY

Here's our terrain generation code:

Code: Select all

local math_random = math.random
local simplex_noise = require "utils.simplex_noise".d2
local rock_raffle = {"sand-rock-big","sand-rock-big","rock-big","rock-big","rock-big","rock-big","rock-big","rock-big","rock-huge"}
local spawner_raffle = {"biter-spawner", "biter-spawner", "biter-spawner", "spitter-spawner"}
local noises = {
	["no_rocks"] = {{modifier = 0.0033, weight = 1}, {modifier = 0.01, weight = 0.22}, {modifier = 0.05, weight = 0.05}, {modifier = 0.1, weight = 0.04}},
	["no_rocks_2"] = {{modifier = 0.013, weight = 1}, {modifier = 0.1, weight = 0.1}},
	["large_caves"] = {{modifier = 0.0033, weight = 1}, {modifier = 0.01, weight = 0.22}, {modifier = 0.05, weight = 0.05}, {modifier = 0.1, weight = 0.04}},	
	["small_caves"] = {{modifier = 0.008, weight = 1}, {modifier = 0.03, weight = 0.15}, {modifier = 0.25, weight = 0.05}},	
	["cave_ponds"] = {{modifier = 0.01, weight = 1}, {modifier = 0.1, weight = 0.06}},
	["cave_rivers"] = {{modifier = 0.005, weight = 1}, {modifier = 0.01, weight = 0.25}, {modifier = 0.05, weight = 0.01}},
}
local caves_start = -360

local function get_noise(name, pos, seed)
	local noise = 0
	local d = 0
	for _, n in pairs(noises[name]) do
		noise = noise + simplex_noise(pos.x * n.modifier, pos.y * n.modifier, seed) * n.weight
		d = d + n.weight
		seed = seed + 10000
	end
	noise = noise / d
	return noise
end

function get_cave_density_modifer(y)
	if y < caves_start then y = y - 2048 end
	local m = 1 + ((y) * 0.000175)
	if m < 0.10 then m = 0.10 end
	return m
end

local function get_replacement_tile(surface, position)
	for i = 1, 128, 1 do
		local vectors = {{0, i}, {0, i * -1}, {i, 0}, {i * -1, 0}}
		table.shuffle_table(vectors)
		for k, v in pairs(vectors) do
			local tile = surface.get_tile(position.x + v[1], position.y + v[2])
			if not tile.collides_with("resource-layer") then return tile.name end
		end
	end
	return "grass-1"
end

local function process_rock_chunk_position(p, seed, tiles, entities, markets, treasure)
	local m = get_cave_density_modifer(p.y)
		
	local small_caves = get_noise("small_caves", p, seed)	
	local noise_large_caves = get_noise("large_caves", p, seed)
	
	if noise_large_caves > m * -1 and noise_large_caves < m then	
		
		local noise_cave_ponds = get_noise("cave_ponds", p, seed)
		--Green Water Ponds
		if noise_cave_ponds > 0.80 then
			tiles[#tiles + 1] = {name = "deepwater-green", position = p}
			if math_random(1,16) == 1 then entities[#entities + 1] = {name="fish", position=p} end
			return
		else
			if noise_cave_ponds > 0.785 then
				tiles[#tiles + 1] = {name = "dirt-7", position = p}
				return 
			end
		end
		
		--Chasms
		if noise_cave_ponds < 0.12 and noise_cave_ponds > -0.12 then
			if small_caves > 0.55 then
				tiles[#tiles + 1] = {name = "out-of-map", position = p}
				return
			end
			if small_caves < -0.55 then
				tiles[#tiles + 1] = {name = "out-of-map", position = p}
				return
			end
		end	
		
		--Rivers
		local cave_rivers = get_noise("cave_rivers", p, seed + 100000)
		if cave_rivers < 0.024 and cave_rivers > -0.024 then
			if noise_cave_ponds > 0 then
				tiles[#tiles + 1] = {name = "water-shallow", position = p}
				if math_random(1,64) == 1 then entities[#entities + 1] = {name="fish", position=p} end
				return				
			end
		end
		
		--Market Spots 
		if noise_cave_ponds < -0.80 then
			tiles[#tiles + 1] = {name = "grass-" .. math.floor(noise_cave_ponds * 32) % 3 + 1, position = p}
			if math_random(1,32) == 1 then markets[#markets + 1] = p end
			if math_random(1,16) == 1 then entities[#entities + 1] = {name = "tree-0" .. math_random(1, 9), position=p} end
			return
		end
		
		local no_rocks = get_noise("no_rocks", p, seed + 25000)
		--Worm oil Zones
		if p.y < -64 + noise_cave_ponds * 10 then
			if no_rocks < 0.08 and no_rocks > -0.08 then
				if small_caves > 0.35 then
					tiles[#tiles + 1] = {name = "dirt-" .. math.floor(noise_cave_ponds * 32) % 7 + 1, position = p}
					if math_random(1,500) == 1 then entities[#entities + 1] = {name = "crude-oil", position = p, amount = math.abs(p.y) * 500} end
					if math_random(1,96) == 1 then
						wave_defense_set_worm_raffle(math.abs(p.y) * 0.5)
						entities[#entities + 1] = {name = wave_defense_roll_worm_name(), position = p, force = "enemy"} 
					end
					if math_random(1,1024) == 1 then treasure[#treasure + 1] = p end
					return
				end
			end
		end
		
		--Main Rock Terrain			
		
		local no_rocks_2 = get_noise("no_rocks_2", p, seed + 75000)
		if no_rocks_2 > 0.80 or no_rocks_2 < -0.80 then
			tiles[#tiles + 1] = {name = "dirt-" .. math.floor(no_rocks_2 * 8) % 2 + 5, position = p}
			if math_random(1,512) == 1 then treasure[#treasure + 1] = p end
			return 
		end
		
		if math_random(1,2048) == 1 then treasure[#treasure + 1] = p end
		tiles[#tiles + 1] = {name = "dirt-7", position = p}
		if math_random(1,4) > 1 then entities[#entities + 1] = {name = rock_raffle[math_random(1, #rock_raffle)], position = p} end		
		return
	end
	
	if math.abs(noise_large_caves) > m * 7 then
		tiles[#tiles + 1] = {name = "water", position = p}
		if math_random(1,16) == 1 then entities[#entities + 1] = {name="fish", position=p} end
		return
	end	
	if math.abs(noise_large_caves) > m * 6.5 then
		if math_random(1,16) == 1 then entities[#entities + 1] = {name="tree-02", position=p} end
		if math_random(1,64) == 1 then markets[#markets + 1] = p end
	end	
	if math.abs(noise_large_caves) > m * 5 then
		tiles[#tiles + 1] = {name = "grass-2", position = p}
		if math_random(1,512) == 1 then markets[#markets + 1] = p end
		if math_random(1,384) == 1 then
			wave_defense_set_worm_raffle(math.abs(p.y) * 0.5)
			entities[#entities + 1] = {name = wave_defense_roll_worm_name(), position = p, force = "enemy"} 
		end
		return
	end
	if math.abs(noise_large_caves) > m * 4.75 then
		tiles[#tiles + 1] = {name = "dirt-7", position = p}
		if math_random(1,3) > 1 then entities[#entities + 1] = {name = rock_raffle[math_random(1, #rock_raffle)], position = p} end
		if math_random(1,2048) == 1 then treasure[#treasure + 1] = p end
		return
	end
	
	if small_caves > (m + 0.05) * -1 and small_caves < m - 0.05 then
		tiles[#tiles + 1] = {name = "dirt-7", position = p}
		if math_random(1,5) > 1 then entities[#entities + 1] = {name = rock_raffle[math_random(1, #rock_raffle)], position = p} end
		if math_random(1, 512) == 1 then treasure[#treasure + 1] = p end
		return
	end			
		
	tiles[#tiles + 1] = {name = "out-of-map", position = p}
end

local function rock_chunk(surface, left_top)
	local tiles = {}
	local entities = {}
	local markets = {}
	local treasure = {}
	local seed = surface.map_gen_settings.seed
	for y = 0, 31, 1 do
		for x = 0, 31, 1 do
			local p = {x = left_top.x + x, y = left_top.y + y}
			process_rock_chunk_position(p, seed, tiles, entities, markets, treasure)
		end
	end
	surface.set_tiles(tiles, true)

	if #markets > 0 then
		local position = markets[math_random(1, #markets)]
		if surface.count_entities_filtered{area = {{position.x - 96, position.y - 96}, {position.x + 96, position.y + 96}}, name = "market", limit = 1} == 0 then
			local market = mountain_market(surface, position, math.abs(position.y) * 0.004)
			market.destructible = false
		end
	end
	
	for _, p in pairs(treasure) do	treasure_chest(surface, p) end
	
	for _, e in pairs(entities) do
		if game.entity_prototypes[e.name].type == "simple-entity" or game.entity_prototypes[e.name].type == "turret" then
			surface.create_entity(e)
		else
			if surface.can_place_entity(e) then
				surface.create_entity(e)
			end
		end
	end
end

local function border_chunk(surface, left_top)
	local trees = {"dead-grey-trunk", "dead-grey-trunk", "dry-tree"}
	for x = 0, 31, 1 do
		for y = 5, 31, 1 do
			local pos = {x = left_top.x + x, y = left_top.y + y}
			if math_random(1, math.ceil(pos.y + pos.y) + 64) == 1 then
				surface.create_entity({name = trees[math_random(1, #trees)], position = pos})			
			end
		end
	end		
	
	for x = 0, 31, 1 do
		for y = 0, 31, 1 do
			local pos = {x = left_top.x + x, y = left_top.y + y}
			if math_random(1, pos.y + 2) == 1 then
				surface.create_decoratives{
				check_collision=false,
				decoratives={
						{name = "rock-medium", position = pos, amount = math_random(1, 1 + math.ceil(20 - y / 2))}
					}
				}
			end
			if math_random(1, pos.y + 2) == 1 then
				surface.create_decoratives{
				check_collision=false,
				decoratives={
						{name = "rock-small", position = pos, amount = math_random(1, 1 + math.ceil(20 - y / 2))}
					}
				}
			end
			if math_random(1, pos.y + 2) == 1 then
				surface.create_decoratives{
				check_collision=false,
				decoratives={
						{name = "rock-tiny", position = pos, amount = math_random(1, 1 + math.ceil(20 - y / 2))}
					}
				}
			end									
			if math_random(1, math.ceil(pos.y + pos.y) + 2) == 1 then
				surface.create_entity({name = rock_raffle[math_random(1, #rock_raffle)], position = pos})			
			end
		end
	end
	
	for _, e in pairs(surface.find_entities_filtered({area = {{left_top.x, left_top.y},{left_top.x + 32, left_top.y + 32}}, type = "cliff"})) do	e.destroy() end
end

local function biter_chunk(surface, left_top)
	local tile_positions = {}
	for x = 0, 31, 1 do
		for y = 0, 31, 1 do
			local p = {x = left_top.x + x, y = left_top.y + y}
			tile_positions[#tile_positions + 1] = p
		end
	end
	for i = 1, 2, 1 do
		local position = surface.find_non_colliding_position("biter-spawner", tile_positions[math_random(1, #tile_positions)], 16, 2)
		if position then
			local e = surface.create_entity({name = spawner_raffle[math_random(1, #spawner_raffle)], position = position})
			e.destructible = false
			e.active = false
		end		
	end
	for _, e in pairs(surface.find_entities_filtered({area = {{left_top.x, left_top.y},{left_top.x + 32, left_top.y + 32}}, type = "cliff"})) do	e.destroy() end
end

local function replace_water(surface, left_top)
	for x = 0, 31, 1 do
		for y = 0, 31, 1 do
			local p = {x = left_top.x + x, y = left_top.y + y}
			if surface.get_tile(p).collides_with("resource-layer") then
				surface.set_tiles({{name = get_replacement_tile(surface, p), position = p}}, true)
			end		
		end
	end	
end

local function out_of_map(surface, left_top)
	for x = 0, 31, 1 do
		for y = 0, 31, 1 do
			surface.set_tiles({{name = "out-of-map", position = {x = left_top.x + x, y = left_top.y + y}}})				
		end
	end
end
--[[
local function wall(surface, left_top, seed)
	local entities = {}
	for x = 0, 31, 1 do
		for y = 0, 31, 1 do
			local p = {x = left_top.x + x, y = left_top.y + y}
			local small_caves = get_noise("small_caves", p, seed)	
			local cave_ponds = get_noise("cave_rivers", p, seed + 100000)
			if y > 9 + cave_ponds * 6 and y < 23 + small_caves * 6 then
				if small_caves > 0.35 or cave_ponds > 0.35 then
					surface.set_tiles({{name = "water-shallow", position = p}})
				else
					surface.set_tiles({{name = "dirt-7", position = p}})
					if math_random(1, 5) ~= 1 then
						surface.create_entity({name = rock_raffle[math_random(1, #rock_raffle)], position = p})
					end
				end
			else
				surface.set_tiles({{name = "dirt-7", position = p}})
				
				if surface.can_place_entity({name = "stone-wall", position = p, force = "enemy"}) then
					if math_random(1,512) == 1 and y > 3 and y < 28 then
						treasure_chest(surface, p)
					else
						if y < 7 or y > 23 then
							if y <= 15 then
								if math_random(1, y + 1) == 1 then
									surface.create_entity({name = "stone-wall", position = p, force = "enemy"})
								end
							else
								if math_random(1, 32 - y)  == 1 then
									surface.create_entity({name = "stone-wall", position = p, force = "enemy"})
								end
							end
						end
					end				
				end		
				
				if math_random(1, 32) == 1 then
					if surface.can_place_entity({name = "small-worm-turret", position = p, force = "enemy"}) then
						wave_defense_set_worm_raffle(math.abs(p.y) * 0.5)
						surface.create_entity({name = wave_defense_roll_worm_name(), position = p, force = "enemy"})
					end
				end			
			end
		end
	end
end
]]
local function process_chunk(surface, left_top)
	if not surface then return end
	if not surface.valid then return end
	if left_top.x >= 768 then return end
	if left_top.x < -768 then return end
	
	--if left_top.y % 1024 == 0 then wall(surface, left_top, surface.map_gen_settings.seed) return end
	
	if left_top.y >= 0 then replace_water(surface, left_top) end
	if left_top.y > 32 then game.forces.player.chart(surface, {{left_top.x, left_top.y},{left_top.x + 31, left_top.y + 31}}) end	
	if left_top.y == -128 and left_top.x == -128 then
		local p = global.locomotive.position
		for _, entity in pairs(surface.find_entities_filtered({area = {{p.x - 3, p.y - 4},{p.x + 3, p.y + 10}}, type = "simple-entity"})) do	entity.destroy() end
	end
	if left_top.y < 0 then rock_chunk(surface, left_top) return end
	if left_top.y > 96 then out_of_map(surface, left_top) return end
	if left_top.y > 64 then biter_chunk(surface, left_top) return end
	if left_top.y >= 0 then border_chunk(surface, left_top) return end
end

--[[
local function process_chunk_queue()
	for k, chunk in pairs(global.chunk_queue) do		
		process_chunk(game.surfaces[chunk.surface_index], chunk.left_top)		
		global.chunk_queue[k] = nil
		return
	end
end


local function process_chunk_queue()
	local chunk = global.chunk_queue[#global.chunk_queue]
	if not chunk then return end
	process_chunk(game.surfaces[chunk.surface_index], chunk.left_top)		
	global.chunk_queue[#global.chunk_queue] = nil
end
]]

local function on_chunk_generated(event)
	if event.surface.index == 1 then return end
	process_chunk(event.surface, event.area.left_top)
	--global.chunk_queue[#global.chunk_queue + 1] = {left_top = {x = event.area.left_top.x, y = event.area.left_top.y}, surface_index = event.surface.index}
end

local event = require 'utils.event'
event.on_nth_tick(4, process_chunk_queue)
event.add(defines.events.on_chunk_generated, on_chunk_generated)

And the soft reset logic

Code: Select all

function reset_map()
	global.chunk_queue = {}
	
	local map_gen_settings = {
		["seed"] = math.random(1, 1000000),
		--["height"] = 256,
		["width"] = 1536,
		["water"] = 0.001,
		["starting_area"] = 1,
		["cliff_settings"] = {cliff_elevation_interval = 0, cliff_elevation_0 = 0},
		["default_enable_all_autoplace_controls"] = true,
		["autoplace_settings"] = {
			["entity"] = {treat_missing_as_default = false},
			["tile"] = {treat_missing_as_default = true},
			["decorative"] = {treat_missing_as_default = true},
		},
	}
	
	if not global.active_surface_index then
		global.active_surface_index = game.create_surface("mountain_fortress", map_gen_settings).index
	else
		game.forces.player.set_spawn_position({-2, 16}, game.surfaces[global.active_surface_index])	
		global.active_surface_index = soft_reset_map(game.surfaces[global.active_surface_index], map_gen_settings, starting_items).index
	end
	
	local surface = game.surfaces[global.active_surface_index]

	--surface.freeze_daytime = true
	--surface.daytime = 0.5
	surface.request_to_generate_chunks({0,0}, 2)
	surface.force_generate_chunk_requests()
	
	for x = -768 + 32, 768 - 32, 32 do
		surface.request_to_generate_chunks({x, 96}, 1)
		surface.force_generate_chunk_requests()
	end
	
	game.map_settings.enemy_evolution.destroy_factor = 0
	game.map_settings.enemy_evolution.pollution_factor = 0	
	game.map_settings.enemy_evolution.time_factor = 0
	game.map_settings.enemy_expansion.enabled = true
	game.map_settings.enemy_expansion.max_expansion_cooldown = 3600
	game.map_settings.enemy_expansion.min_expansion_cooldown = 3600
	game.map_settings.enemy_expansion.settler_group_max_size = 8
	game.map_settings.enemy_expansion.settler_group_min_size = 16
	game.map_settings.pollution.enabled = false
	
	game.forces.player.technologies["landfill"].enabled = false
	game.forces.player.technologies["railway"].researched = true
	game.forces.player.set_spawn_position({-2, 16}, surface)
	
	locomotive_spawn(surface, {x = 0, y = 16})
	
	reset_wave_defense()
	global.wave_defense.surface_index = global.active_surface_index
	global.wave_defense.target = global.locomotive_cargo
	global.wave_defense.side_target_search_radius = 768
	
	global.wave_defense.game_lost = false
	
	--for _, p in pairs(game.connected_players) do
	--	if p.character then p.character.disable_flashlight() end
	--end
end

local function reset_forces(new_surface, old_surface)
	for _, f in pairs(game.forces) do
		local spawn = {x = game.forces.player.get_spawn_position(old_surface).x, y = game.forces.player.get_spawn_position(old_surface).y}
		f.reset()
		f.reset_evolution()
		f.set_spawn_position(spawn, new_surface)
	end
end

local function teleport_players(surface)	
	for _, player in pairs(game.connected_players) do
		local spawn = player.force.get_spawn_position(surface)
		local chunk = {math.floor(spawn.x / 32), math.floor(spawn.y / 32)}
		if not surface.is_chunk_generated(chunk) then
			surface.request_to_generate_chunks(spawn, 1)
			surface.force_generate_chunk_requests()
		end		
		local pos = surface.find_non_colliding_position("character", spawn, 3, 0.5)
		player.teleport(pos, surface)
	end
end

local function equip_players(player_starting_items)
	for k, player in pairs(game.connected_players) do
		if player.character then	player.character.destroy()	end
		player.character = nil
		player.set_controller({type=defines.controllers.god})
		player.create_character()
		for item, amount in pairs(player_starting_items) do
			player.insert({name = item, count = amount})
		end
	end
end

function soft_reset_map(old_surface, map_gen_settings, player_starting_items)
	if not global.soft_reset_counter then global.soft_reset_counter = 0 end
	if not global.original_surface_name then global.original_surface_name = old_surface.name end
	global.soft_reset_counter = global.soft_reset_counter + 1

	local new_surface = game.create_surface(global.original_surface_name .. "_" .. tostring(global.soft_reset_counter), map_gen_settings)
	new_surface.request_to_generate_chunks({0,0}, 1)
	new_surface.force_generate_chunk_requests()
	
	reset_forces(new_surface, old_surface)
	teleport_players(new_surface)
	equip_players(player_starting_items)
	
	game.delete_surface(old_surface)
	
	local message = table.concat({">> Welcome to ", global.original_surface_name, "!"})
	if global.soft_reset_counter > 1 then
		message = table.concat({">> The world has been reshaped, welcome to ", global.original_surface_name, " number ", tostring(global.soft_reset_counter), "!"})
	end
	game.print(message, {r=0.98, g=0.66, b=0.22})
	server_commands.to_discord_embed(message)
	
	return new_surface
end

[Klonan]

Code: Select all

local simplex_noise = require "utils.simplex_noise".d2

I guess knowing whats going on in here too would be useful

[rocifier]

So far we narrowed one of the desyncs down to this by process of elimination: if we add a robot locomotive driver and then call reset_map to force a soft reset, the driver desyncs, I believe he is the off-by-one active entity and <force> difference.

Code: Select all

local function accelerate()
	if not global.locomotive then return end
	if not global.locomotive.valid then return end
	if global.locomotive.get_driver() then return end	
	global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"})
	global.locomotive_driver.driving = true
	global.locomotive_driver.riding_state = {acceleration = defines.riding.acceleration.accelerating, direction = defines.riding.direction.straight}
end

since from our understanding destroying a surface should destroy all its entities deterministically, could this potentially be a core game bug? The driver may even still be "being created" in some weird way when the reset is occuring. If we destroy this entity before we reset the map, we don't get any logs from heavy-mode, and we get much fewer desyncs.

To answer your question Klonan, the `simplex_noise` function is from here: https://github.com/thenumbernine/lua-si ... ter/2d.lua

Would it help if I put together a super simple map that illustrates the locomotive driver bug, or have we done something obviously wrong here? Full locomotive module:

Code: Select all

function locomotive_spawn(surface, position)
	for y = -6, 6, 2 do
		surface.create_entity({name = "straight-rail", position = {position.x, position.y + y}, force = "player", direction = 0})
	end
	global.locomotive = surface.create_entity({name = "locomotive", position = {position.x, position.y + -3}, force = "player"})
	global.locomotive.get_inventory(defines.inventory.fuel).insert({name = "wood", count = 100})
	
	global.locomotive_cargo = surface.create_entity({name = "cargo-wagon", position = {position.x, position.y + 3}, force = "player"})
	global.locomotive_cargo.get_inventory(defines.inventory.cargo_wagon).insert({name = "raw-fish", count = 8})
	
	rendering.draw_light({
		sprite = "utility/light_medium", scale = 5.5, intensity = 1, minimum_darkness = 0,
		oriented = true, color = {255,255,255}, target = global.locomotive,
		surface = surface, visible = true, only_in_alt_mode = false,
	})
	
	global.locomotive.color = {0, 255, 0}
	global.locomotive.minable = false
	global.locomotive_cargo.minable = false
	global.locomotive_cargo.operable = false
end

local function fish_tag()
	if not global.locomotive_cargo then return end
	if not global.locomotive_cargo.valid then return end
	if not global.locomotive_cargo.surface then return end
	if not global.locomotive_cargo.surface.valid then return end
	if global.locomotive_tag then
		if global.locomotive_tag.valid then
			if global.locomotive_tag.position.x == global.locomotive_cargo.position.x and global.locomotive_tag.position.y == global.locomotive_cargo.position.y then return end
			global.locomotive_tag.destroy() 
		end
	end
	global.locomotive_tag = global.locomotive_cargo.force.add_chart_tag(
		global.locomotive_cargo.surface,
		{icon = {type = 'item', name = 'raw-fish'},
		position = global.locomotive_cargo.position,
		text = " "
	})
end

local function accelerate()
	if not global.locomotive then return end
	if not global.locomotive.valid then return end
	if global.locomotive.get_driver() then return end	
	global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"})
	global.locomotive_driver.driving = true
	global.locomotive_driver.riding_state = {acceleration = defines.riding.acceleration.accelerating, direction = defines.riding.direction.straight}
end

local function remove_acceleration()
	if not global.locomotive then return end
	if not global.locomotive.valid then return end
	if global.locomotive_driver then global.locomotive_driver.destroy() end
	global.locomotive_driver = nil
end

local function set_player_spawn_and_refill_fish()
	if not global.locomotive_cargo then return end
	if not global.locomotive_cargo.valid then return end
	global.locomotive_cargo.get_inventory(defines.inventory.cargo_wagon).insert({name = "raw-fish", count = 4})
	local position = global.locomotive_cargo.surface.find_non_colliding_position("stone-furnace", global.locomotive_cargo.position, 16, 2)
	if not position then return end
	game.forces.player.set_spawn_position({x = position.x, y = position.y}, global.locomotive_cargo.surface)
end

local function set_daytime()
	if not global.locomotive_cargo then return end
	if not global.locomotive_cargo.valid then return end
	local p = global.locomotive_cargo.position.y
	local t = math.abs(global.locomotive_cargo.position.y) * 0.02
	if t > 0.5 then t = 0.5 end	
	global.locomotive_cargo.surface.daytime = t
	game.print(t)
end

local function tick()
	if game.tick % 30 == 0 then	
		if global.game_reset_tick then
			if global.game_reset_tick < game.tick then
				global.game_reset_tick = nil
				reset_map()
			end
		end
		fish_tag()
		accelerate()
		if game.tick % 1800 == 0 then
			set_player_spawn_and_refill_fish()
		end

	else
		remove_acceleration()
	end
end

local event = require 'utils.event'
event.on_nth_tick(5, tick)

[Bilka]

I cannot create a desync by simply doing global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"}) global.locomotive_driver.driving = true and then deleting that surface. Neither with /c global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"}) global.locomotive_driver.driving = true; global.locomotive_driver.riding_state = {acceleration = defines.riding.acceleration.accelerating, direction = defines.riding.direction.straight}; game.delete_surface(global.locomotive.surface);.

[rocifier]

I cannot create a desync by simply doing global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"}) global.locomotive_driver.driving = true and then deleting that surface. Neither with /c global.locomotive_driver = global.locomotive.surface.create_entity({name = "character", position = global.locomotive.position, force = "player"}) global.locomotive_driver.driving = true; global.locomotive_driver.riding_state = {acceleration = defines.riding.acceleration.accelerating, direction = defines.riding.direction.straight}; game.delete_surface(global.locomotive.surface);.

If I enter the /c command with our scenario running, I get heuristic fails in heavy mode.

I have spent a long while eliminating code right down to these lines:

Code: Select all

    surface.request_to_generate_chunks({0,0}, 2)
    surface.force_generate_chunk_requests()
    
    for x = -768 + 32, 768 - 32, 32 do
        surface.request_to_generate_chunks({x, 96}, 1)
        surface.force_generate_chunk_requests()
    end
    

if these lines are executed in on_init(), it is MUCH more predictable to be able to cause a desync. In fact I can do it every time if I then do the train command at around tick 20. I think at that moment the chunk is half way generating or something. They don't need to be forced either, that just seems to make it full deterministic.