Performance optimization - post your saves

[SoShootMe]

I'm not sure if memory latency would show itself in %CPU ussage (on one hand, if CPU instruction execution is stalled by need to wair on memory, it should bee seen as %CPU usage, on the other hand, modern CPUs are so complex I can't be sure here.

CPU usage is calculated by the OS, eg for a given core it is the proportion of time over some period (eg the last second) that the OS was able to schedule a thread on that core. Waiting on RAM (or some other things like execution resources, especially with SMT) is handled by the CPU itself, so a thread that is "stalled" because of this is still "running" from the OS point of view, and the time still counts as CPU usage.

On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.

[jodokus31]

I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date

i'm on DDR4-2666 Kingston ECC memory + Ryzen 3700x and it's 15 UPS because i'm using mimalloc on Linux as my system-wide memory allocator. if I start it up with glibc malloc there's 11 UPS. i'm sure with 4400MHz memory CL19 and mimalloc you might even get 25UPS! but not 30 UPS. probably never 30 with this map.

I just tested mimalloc:
- Without I get 17-18 UPS, if I stand zoomed-in somewhere
- With I get 19-20 UPS

So, there is a bit of potential, I guess.

BTW:

This is my CPU graph

Screenshot from 2021-05-07 10-06-31.png (100.35 KiB) Viewed 9903 times

It alternates between the cores and one core is quite high

[ptx0]

On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.

except on linux where iowait is included in system load calculations

It alternates between the cores and one core is quite high

too bad we can't pin threads to cores

[SoShootMe]

On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.

except on linux where iowait is included in system load calculations

Fair point, but I don't consider iowait "CPU usage" (which I'd say is user, system, and perhaps nice time depending on context). As I understand it, it is basically a type (special case) of idle time. On the other hand, threads in iowait state contribute to the load average calculation (if that's what you meant), just like runnable threads, but then load average has a different meaning to CPU usage.

It alternates between the cores and one core is quite high

too bad we can't pin threads to cores

AFAIK you can, and I've often thought it may offer some (small) performance benefit due to caches. But you have to make sure other things are excluded from running on those cores too, so in most cases it seems like micro-management overkill. Generally I think it is better to just ignore per-core usage, which is not very meaningful with "busy" threads flitting around, and look at the total .

[AdamK]

Just to finish my thread-within-a-thread, today I updated my box from I7-8086K with DDR-2166 to I7-11700K with DDR-3200, both have/had decent mobos, memory also with as low CL as I could easily get, and I'm up from 9-10 FPS to 12-13FPS. I'd say that 20% increase here is significant (I haven't tried oveclocking it yet)

I forgot one thing: my base uses robots for building and some little logistic. 9-10 FPS on old config was with pretty idle bots, new one is with bots having a bit of work, so I'd say, I may expect +1FPS at lease

[Abarel]

Just to finish my thread-within-a-thread, [...] so I'd say, I may expect +1FPS at lease

Just in case someone gets here and have similar doubts, Ram bandwidth (not just frequency, but latency) means A LOT difference.
As fun sidenote, save your map before using console commands if you are afraid of achievements, then run this little script (it will take several seconds, be patient):

Code: Select all

/c for key, ent in pairs(game.player.surface.find_entities_filtered({force=game.player.force})) do
	if string.find(ent.name,"locomotive") then
		ent.destroy()
	end
end

My PC went from 13 UPS to 60 UPS on your map

[jodokus31]

Just to finish my thread-within-a-thread, [...] so I'd say, I may expect +1FPS at lease

Just in case someone gets here and have similar doubts, Ram bandwidth (not just frequency, but latency) means A LOT difference.
As fun sidenote, save your map before using console commands if you are afraid of achievements, then run this little script (it will take several seconds, be patient):

Code: Select all

/c for key, ent in pairs(game.player.surface.find_entities_filtered({force=game.player.force})) do
	if string.find(ent.name,"locomotive") then
		ent.destroy()
	end
end

My PC went from 13 UPS to 60 UPS on your map

Destroying all the locomotives is surely not a good workaround
And destroying the whole production is not a good way to save UPS

[azesmbog]

Just to finish my thread-within-a-thread, today I updated my box from I7-8086K with DDR-2166 to I7-11700K with , both have/had decent mobos, memory also with as low CL as I could easily get, and I'm up from 9-10 FPS to 12-13FPS. I'd say that 20% increase here is significant (I haven't tried oveclocking it yet)

I also looked at your map.
I would say 90 percent of construction and transport drones are superfluous
However, everyone plays as he personally likes.
I have an average UPS on this map - 18-19: ((And then - in idle.
(on my current map ups 42-43, and even then I find it very low)
DDR-3200 is not very productive for such a map, it would be necessary to raise the frequency to 3600 and higher and lower the timings. if possible.

[Abarel]

Destroying all the locomotives is surely not a good workaround

That was just a joke.
I should have put in bold the "As fun sidenote"...

The real meaning is that the map was using too many of everything, and so the UPS droped a lot. It would work almost the same deleting 70% of trains (10k trains, mostly iddling), 90% of bots (221k logistic + 101k construction bots, mostly iddling), and 40% of inserters (near 100k inserters, half of them iddling most the time). However, Factorio is such a great game that even with this map having 11M entities and 26M connectors, the map is still fun.
I focused the joke on trains as I think the main gaining can be achieved by greatly reducing the amount of trains (1-1 trains are fun, and you can do the same here with just 3k trains, as most the 10k are iddling almot full time), improving the signaling (ok in 3 ways roudabouts, but ugly on 4 ways crossings, stackers and almost everywhere), optimizing the schedules, and even start optimizing inserters as train stations are a big chunk of the inserters used (6 inserters to box, then 6 to belt, to just fill a single blue belt; can replace these 12 inserters with just 2, with a lot less belts and splitters. And there are 3k+ stations... so it is a big task, with big results.

[ptx0]

It would work almost the same deleting 70% of trains (10k trains, mostly iddling)

oh, are you using vehicle equipment grid mod?

[jodokus31]

Destroying all the locomotives is surely not a good workaround

That was just a joke.
I should have put in bold the "As fun sidenote"...

I wondered a bit, if it was really a joke and also wanted to prevent, that somebody blindly copies the command and destroys his save

Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly

[ptx0]

Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly

please cite a source or provide a test case where this is true, because the pathfinder will not search both directions if it's not even possible to go both. it only searches from the locomotive side. RoRo with single loco facing front would be the least expensive.

[jodokus31]

Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly

please cite a source or provide a test case where this is true, because the pathfinder will not search both directions if it's not even possible to go both. it only searches from the locomotive side. RoRo with single loco facing front would be the least expensive.

If the path finder can skip a whole rail contraption, because there is no simple way through it (excluding forwards and then backwards), it should save performance. Assuming the path finder does it, but i have no source or evidence for this...

[ptx0]

If the path finder can skip a whole rail contraption, because there is no simple way through it (excluding forwards and then backwards), it should save performance. Assuming the path finder does it, but i have no source or evidence for this...

the pathfinder is available on github

[jodokus31]

the pathfinder is available on github

Interesting. Takes a bit of time to get into it.

[eradicator]

Not sure if this counts as "performance", but it seems to be the best fit thread on the forum. I've noticed that the mere existance of a lot of prototypes seems to slow down data stage loading of all subsequent mods. I've also noticed that at runtime the name list for Prototype/Entity.additional_pastable_entities seems to be copied for each entity, even if the prototypes are all defined with a reference to the exact same table. This might be fine for short lists, but my mod generates about 5000 prototypes, each of which needs the full list of names of all of them. Resulting in 5000*5000 * (estimate 50 byte) = about 1 gigabyte, but at runtime I'm seeing a ram increase of more than 3 gigabyte. And if the list was stored only once it should be closer to 250 megabyte. I'm not sure why additional_pastable_entities is a list of names and not types, but in the current state it's not realistic to use for this.

(The generation algo as such is not optimized and quite slow by itself.)

eradicators-stockpile_1.9.2.zip

(1.32 MiB) Downloaded 643 times

[eradicator]

I know, not at all a gameplay problem, but maybe there's something to be optimized hiding? I did expect the cpu cost of spawning a single entity to be constant.

For testing something I used LuaSurface.create_entity to spawn in 3 groups of 5000 solar panels each, so the panels in each group all have the same position. I saved the game, and tried to load it, but loading takes ages even after removing all mods. I waited for more than 10 mins and the load bar just stays stuck at 100%. The map is otherwise new and empty. Spawning in the panels was also very slow as if it's trying to detect collision against all existing panels, but I'm not sure if that wasn't mod related.

I used this command 5 times while standing still. Then used editor area clone to copy the group 2 more times, then gave up because it was so slow :D.

Code: Select all

/c
local p = game.player
local ce, args = p.surface.create_entity, {name = 'solar-panel', force = p.force, position = p.position, raise_built = false, 
create_build_effect_smoke = false, spawn_decorations = false, move_stuck_players = false}

for _=1, 1000 do ce(args) end

[Rseding91]

Stacking entities on top of each other causes anything that has to interact with that section of the map to grow linearly more expensive with each entity added. That issue is compounded if they are electric connecting entities because during migrations they disconnect and reconnect to nearby electric poles; but that has to interact with every one of the entiteis in the area as it searches for poles.

That ends up being O(N^2).

[eradicator]

Stacking entities on top of each other causes anything that has to interact with that section of the map to grow linearly more expensive

As for the case of create_entity: As it doesn't do any apparent collision checking I expected it to not care at all about other entities. I added all the {foo = false} options in an attempt to disable as much checking as I could.

during migrations they disconnect and reconnect to nearby electric poles

Aha, interesting. So if migrating poles searched for connectables instead of the other way around it would've work as I expected. Thanks for the explanation!

[mrvn]

On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.

except on linux where iowait is included in system load calculations

Fair point, but I don't consider iowait "CPU usage" (which I'd say is user, system, and perhaps nice time depending on context). As I understand it, it is basically a type (special case) of idle time. On the other hand, threads in iowait state contribute to the load average calculation (if that's what you meant), just like runnable threads, but then load average has a different meaning to CPU usage.

It alternates between the cores and one core is quite high

too bad we can't pin threads to cores

AFAIK you can, and I've often thought it may offer some (small) performance benefit due to caches. But you have to make sure other things are excluded from running on those cores too, so in most cases it seems like micro-management overkill. Generally I think it is better to just ignore per-core usage, which is not very meaningful with "busy" threads flitting around, and look at the total .

There can be some gain with improved cache hits. If you have 2 sockets/cores with separate caches then pining threads to each set of cores/threads that share caches can be beneficial. Even if they get interrupted every now and then. The thread can jump between cpu threads or cores with little cost if they remain in the same cache.

After that it makes a huge difference with NUMA but only if your memory allocation and usage is also NUMA aware. For example you would allocate all inserters in a memory segment in one NUMA domain and pin the thread doing inserter updates to cores in the same NUMA domain. The thread would run a lot faster there. Or split the map into quadrants and place each quadrant into one NUMA domain and pin a thread there. Then each assembler, belt, inserter needs to be allocated in the right memory block and processed by the right thread.

Factorio isn't really designed for that. The core design was really single threaded and since then it only got a few patch ups for special things that could be made multi threaded after the fact. Like the fluid system that could be broken into independent parts. But there isn't a inserter thread that runs in parallel with for example an assembler thread. It's not designed that way. Afaik nothing in factorio would be able to optimize for NUMA and pinned threads without a major overhaul. It's just not designed that way and maybe doesn't fit that memory model at all.

Best approach might be to split things by geography. Split the map into largish tiles and everything in that tile gets pinned to one core and it's closest memory. Of course that requires extra work at the tile boundaries where threads would collide but given large enough tiles that can be minimized (overhead wise). But that would probably only help systems with NUMA as the working set is far to big to make cache effects relevant in that optimization.