Page 1 of 1

Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 1:07 am
by husnikadam
Hi, I am exploring the limits of Factorio engine and I am doing an experiment of generating milions of chunks via pollution. I know that pollution has a performance impact, especially on this extreme scale, but I am wondering, how do I read that from the debug menu? It says it takes about 75ms on Entity Update each frame, but entity time usage shows basically zero time as only entities on the map are rocks (and removing the rocks doesn't help). Am I reading the stats wrong, or is there a missing debug property, something like a pollution calculation time usage?
factorio_QuefHIfkCW.png
factorio_QuefHIfkCW.png (4.44 MiB) Viewed 2614 times
factorio_syNhzu5T7n.png
factorio_syNhzu5T7n.png (383.31 KiB) Viewed 2614 times

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 9:09 am
by disentius
Assuming biters are off: fish could be responsible for the entity count.
interesting:)

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 11:46 am
by husnikadam
Map generation is such that there is no water, no trees, no cliffs, no ores, no biters, basically nothing excepts rocks. The poor performance might also have something to do with requested chunks count, not sure what that means though

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 1:24 pm
by FuryoftheStars
Using /editor, there is a button under the map manipulation tools that can remove all entities. It can also be used to force generating lab tiles and thus guarantee no more entities will be created.

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 1:48 pm
by Bilka
If you're looking for more granular performance profiling than the debug options provide then you can use a profiler like Very Sleepy. Some instructions for how to run it can be found here: viewtopic.php?p=501968#p501968

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 1:55 pm
by husnikadam
I have tried that and it had no effect on the performance. I have used a command to force generate all requested chunks and after the unfroze after 20 minutes, the performance jumped from 16 ups to about 70 ups (requested game speed is 100x). The requested chunks had dramatic impact on performance, just don't know why. After removing the pollustion, the game runs on about 5500 UPS, takes 24GB of RAM and has about 3.7M chunks.

My remaining question is, why was the slowdown reported on entity-update usage even though there were no entities and entity-time-usage info show no use either? (check previous images)

Command used to force generate chunks: /c game.player.surface.force_generate_chunk_requests()

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 2:00 pm
by FuryoftheStars
force_generate_chunk_requests() does exactly as the name implies... it forces the generation of chunks that have been requested to be generated. It will use all CPU cores to their max to accomplish this. This is the only thing I have ever had peg my CPU at 100% and cause thermal throttling. :lol:

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 2:59 pm
by husnikadam
Well it used only about 1 of 8 of my CPU cores and it didn't actually generate new chunks, didn't help with my confusion, what the requested chunks mean

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 3:22 pm
by FuryoftheStars
On mine all cores get used. But also, generating a chunk =/= making a chunk visible. You need to use one of the chart commands for that.

https://lua-api.factorio.com/latest/cla ... ate_chunks
https://lua-api.factorio.com/latest/cla ... k_requests
https://lua-api.factorio.com/latest/cla ... html#chart
https://lua-api.factorio.com/latest/cla ... #chart_all
husnikadam wrote: Sun Feb 04, 2024 2:59 pm [...]what the requested chunks mean
As some things near the edge of what's already been generated, such as pollution, the game will request more chunks to be generated so there's someplace for it to go rather than a hard stop to it. However, this is queued up in a way to minimize impact on the running game, so in your case where you have a super large cloud, it's possible for it to be requesting chunks faster than it can actually generate them.

Re: Debugging poor performance on gigantic map

Posted: Sun Feb 04, 2024 3:57 pm
by Rseding91
If you can post the save file I can tell you where the time is spent.

Re: Debugging poor performance on gigantic map

Posted: Mon Feb 05, 2024 8:57 am
by husnikadam

Re: Debugging poor performance on gigantic map

Posted: Mon Feb 05, 2024 7:39 pm
by Rseding91
The time is spent trying to spread pollution. Specifically it's requesting more chunks be generated as pollution spreads but due to the speed map generation happens at, the list of chunks to generate is growing faster than they generate. Pollution keeps checking to add chunks to the list and that check. There are 2.5 million chunks requested to be generated so every time pollution spreads to another chunk it has to do Log(2'500'000) checks (roughly 7). There are roughly 40'000 chunks spreading pollution per tick and each one spreads theoretically in 1 direction meaning: 40'000 * 7 checks to add a chunk to the to-generate list each tick.

Re: Debugging poor performance on gigantic map

Posted: Tue Feb 06, 2024 1:09 pm
by husnikadam
Very interesting insight into the map generation. Thank you very much!

Re: Debugging poor performance on gigantic map

Posted: Tue Feb 06, 2024 1:18 pm
by vjbone
TLDR pollution bad

Re: Debugging poor performance on gigantic map

Posted: Tue Feb 06, 2024 2:34 pm
by Rseding91
vjbone wrote: Tue Feb 06, 2024 1:18 pm TLDR pollution bad
No. The levels of pollution here are I.N.S.A.N.E. to the point where they have overwhelmed the map generator. The actual pollution spread logic is measuring almost nothing compared to overwhelming the map generator.