Page 1 of 1

[0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 7:33 am
by Eximius
The Minor update of 0.17.53 -> 0.17.54 seems to have caused chunk-scanning to slow down 100 times.

This is seen in a save with Alien Loot Economy ("alien-module" mod). Before the update, it was only taking < 0.25 ms per tick. Now it is taking 25ms.

I am attaching two different saves. An older (8) that had a smaller active map, and now has 4ms / tick of time spent in alien-module mod, and the latest game (11), that now has 25ms / tick in alien-module mod.

The reason I am attributing this to filtered chunk-scanning, is this part of the "alien-module" mod:

Code: Select all

	for index, surface_iterator in pairs(global.surface_iterators) do
		for i=1, batch_size do
			--***Want to get all the chunk logic in here so we can scan a surface more quickly per cycle.

			local chunk = surface_iterator()
			if chunk == nil then
				-- Disable the printing if not in debug mode
				if debug_mode then
					game.print('Rescanning chunks on surface # ' .. tostring(index))
				end
				global.surface_iterators[index] = game.surfaces[index].get_chunks()
			else
				--game.print("x: " .. tostring(chunk.x).. "y: " .. tostring(chunk.y))
				--include logic here to scan each surface @ chunk.
				local surface = game.surfaces[index]
				local chunk_position = { x = chunk.x * 32, y = chunk.y*32 }
				update_modules_on_surface(surface,chunk_position)
				--game.print(serpent.block(chunk_position))

				local assemblers = surface.find_entities_filtered { type = "assembling-machine", position = chunk_position, radius = 16}
				local miners = surface.find_entities_filtered { type = "mining-drill" , position = chunk_position, radius = 16}
				local labs = surface.find_entities_filtered { type = "lab" , position = chunk_position, radius = 16}
				local furnaces = surface.find_entities_filtered { type = "furnace" , position = chunk_position, radius = 16}
				local rocketSilos = surface.find_entities_filtered { name = "rocket-silo" , position = chunk_position, radius = 16}
				local chests = surface.find_entities_filtered { type = "container" , position = chunk_position, radius = 16}
				local logisticChests = surface.find_entities_filtered { type = "logistic-container" , position = chunk_position, radius = 16}
				local beacons = surface.find_entities_filtered { type = "beacon" , position = chunk_position, radius = 16}

				if debug_mode then
					rendering.draw_circle{color = {r = 1, g = 0, b = 0, a = 0.5}, radius = 16, target = chunk_position, filled = true, surface = game.surfaces[index], time_to_live = 30}
				end

				update_modules(assemblers, "machine")
				update_modules(miners, "machine")
				update_modules(labs, "machine")
				update_modules(furnaces, "machine")
				update_modules(rocketSilos, "machine")
				update_modules(chests, "chest")
				update_modules(logisticChests, "chest")
				update_modules(beacons, "machine")

				update_recipes(assemblers)

			end
		end
Which is run every 2 secs.

I am not the author of the mod, just peered into it.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 10:12 am
by Rseding91
Thanks for the report however nothing about that logic changed in 0.17.54.

Based off the code you've provided that mod is just doing an incredibly slow operation: scanning entire chunks using entity search multiple times is an incredibly slow operation.

Unless you can give me a save file which takes 0.25 MS/tick in 0.17.53 and 25 MS/tick in 0.17.54 I'm just going to say: don't use that mod because it's written terribly.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 10:24 am
by darkfrei
Is the using of chnk middle position better than the corner one?

Code: Select all

local chunk_top_left_position = { x = chunk.x*32, y = chunk.y*32 }
local entities = surface.find_entities_filtered { type = types, 
  position = chunk_top_left_position, 
  radius = 16}
vs

Code: Select all

local chunk_middle_position = { x = chunk.x*32+16, y = chunk.y*32+16}
local entities = surface.find_entities_filtered { type = types, 
  position = chunk_middle_position, 
  radius = 16}
A lot of lua rendering needs too much UPS time too.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 4:48 pm
by Eximius
I agree that the mod is written terribly, however, please find that both of the save files are 0.17.53.

For further proof, please find the attached screenshot of savefile (8) on my newly downloaded version of 0.17.53:
Screenshot_20190705_194544.png
Screenshot_20190705_194544.png (1.01 MiB) Viewed 2171 times
With alien-modules taking <0.01 ms

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 5:26 pm
by Rseding91
I loaded your (11) save file and it's not taking any measurable amount of time.

You must have changed something on your end when you updated the game - added/removed mods, changed mod settings, or been running the computer in low power mode?
show

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 6:00 pm
by Eximius
Loading up (11) with synchronization still leaves my 0.17.54 game having alien-modules take 29ms / tick. Also, seems the game is unable to differentiate mod versions under sync, because now I see that the game actually loaded a debug version I added (which I had set a different mod version number for) :| . I removed this folder from mods.

As a note, low-power laptop mode would not disproportionately increase 100-fold the time usage of a single mod, I do not think :)

I tried turning off all mods and then synchronizing, loading (11) :
Screenshot_20190705_205811.png
Screenshot_20190705_205811.png (267.13 KiB) Viewed 2150 times
I definitely updated a few mods that had new versions, none of them were alien-module. Also, none of them had any major changes (I actually check the changelogs out of morbid curiosity).

My go to guess would be paging, considering how big the world (with space exploration), but I am running steadily on 50% usage of 16GB of ram, with no swap enabled.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 6:07 pm
by Rseding91
Since I can't reproduce any slowdown and I know for a fact we haven't changed anything that would cause the mod to take more time than before this version i'm going to move this to technical help.

If you do manage to track down what mod is causing the issue and it ends up being a base game thing let me know.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 6:09 pm
by Eximius
Seems the only mods that have multiple mod zips under mods/ are:
laser_fix_0.17.9
laser_fix_0.17.10

space-exploration_0.1.96.zip
space-exploration_0.1.98.zip
space-exploration_0.1.103.zip
space-exploration_0.1.106.zip
space-exploration_0.1.111.zip
space-exploration_0.1.112.zip
space-exploration_0.1.113.zip
space-exploration_0.1.114.zip

stdlib_1.1.0.zip
stdlib_1.1.1.zip
stdlib_1.2.0.zip

Does mod sinchronization failure count as base-game? :) Otherwise, I understand.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 6:18 pm
by Bilka
My guess would be that the "reset technology effects" migration that we shipped with 0.17.54 triggered the chunk scan code of the mod which upgrades the modules. That might not be intended to happen from the mods side, so it would be best if you could report that as a bug to the mod author. However, I also cannot reproduce the slowdown.

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 6:32 pm
by Eximius
I tried synchronizing from empty mods to the load (11), but still no dice. 30ms / tick.

Reset technology effects? I did not see that in 0.17.54 changelog.

The mod does not really have a trigger, it is coded that it just runs a batch of chunks to scan for modules... Which is the so-called poorman's code part :P

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 7:03 pm
by Eximius
Um... Having _only_ the alien-module mod enabled, I still get 20ms / tick:

Re: [0.17.54] Filtered Chunk Scanning slowed down by a factor of 100

Posted: Fri Jul 05, 2019 8:59 pm
by Eximius
Yep, this was a false alarm. Sorry. Apparently, I was using version 0.6.5 of the mod, but that got updated to 0.6.6 which was the bad one, and now the mod is updated to 0.6.6, 0.6.7, 0.6.8 literally today.