Performance optimization - post your saves

Post all other topics which do not belong to any other category.
SoShootMe
Fast Inserter
Fast Inserter
Posts: 145
Joined: Mon Aug 03, 2020 4:16 pm
Contact:

Re: Performance optimization - post your saves

Post by SoShootMe »

AdamK wrote:
Thu May 06, 2021 2:21 pm
I'm not sure if memory latency would show itself in %CPU ussage (on one hand, if CPU instruction execution is stalled by need to wair on memory, it should bee seen as %CPU usage, on the other hand, modern CPUs are so complex I can't be sure here.
CPU usage is calculated by the OS, eg for a given core it is the proportion of time over some period (eg the last second) that the OS was able to schedule a thread on that core. Waiting on RAM (or some other things like execution resources, especially with SMT) is handled by the CPU itself, so a thread that is "stalled" because of this is still "running" from the OS point of view, and the time still counts as CPU usage.

On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1161
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

ptx0 wrote:
Thu May 06, 2021 3:09 pm
jodokus31 wrote:
Thu May 06, 2021 3:02 pm
I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date
i'm on DDR4-2666 Kingston ECC memory + Ryzen 3700x and it's 15 UPS because i'm using mimalloc on Linux as my system-wide memory allocator. if I start it up with glibc malloc there's 11 UPS. i'm sure with 4400MHz memory CL19 and mimalloc you might even get 25UPS! but not 30 UPS. probably never 30 with this map.
I just tested mimalloc:
- Without I get 17-18 UPS, if I stand zoomed-in somewhere
- With I get 19-20 UPS

So, there is a bit of potential, I guess.

BTW:
This is my CPU graph
It alternates between the cores and one core is quite high
If You find typos, my mobile is an uncontrollable beast

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1033
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

SoShootMe wrote:
Fri May 07, 2021 2:23 am
On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.
except on linux where iowait is included in system load calculations
jodokus31 wrote:
Fri May 07, 2021 8:13 am
It alternates between the cores and one core is quite high
too bad we can't pin threads to cores
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

SoShootMe
Fast Inserter
Fast Inserter
Posts: 145
Joined: Mon Aug 03, 2020 4:16 pm
Contact:

Re: Performance optimization - post your saves

Post by SoShootMe »

ptx0 wrote:
Fri May 07, 2021 2:38 pm
SoShootMe wrote:
Fri May 07, 2021 2:23 am
On the other hand, a thread that is "stalled" waiting for data from storage (including paging activity) is different: in this case, the OS gets involved and will suspend the thread, marking it unable to be scheduled again until after the data are available. Suspended threads do not contribute to CPU usage.
except on linux where iowait is included in system load calculations
Fair point, but I don't consider iowait "CPU usage" (which I'd say is user, system, and perhaps nice time depending on context). As I understand it, it is basically a type (special case) of idle time. On the other hand, threads in iowait state contribute to the load average calculation (if that's what you meant), just like runnable threads, but then load average has a different meaning to CPU usage.
ptx0 wrote:
Fri May 07, 2021 2:38 pm
jodokus31 wrote:
Fri May 07, 2021 8:13 am
It alternates between the cores and one core is quite high
too bad we can't pin threads to cores
AFAIK you can, and I've often thought it may offer some (small) performance benefit due to caches. But you have to make sure other things are excluded from running on those cores too, so in most cases it seems like micro-management overkill. Generally I think it is better to just ignore per-core usage, which is not very meaningful with "busy" threads flitting around, and look at the total :).

User avatar
AdamK
Inserter
Inserter
Posts: 30
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

Just to finish my thread-within-a-thread, today I updated my box from I7-8086K with DDR-2166 to I7-11700K with DDR-3200, both have/had decent mobos, memory also with as low CL as I could easily get, and I'm up from 9-10 FPS to 12-13FPS. I'd say that 20% increase here is significant :) (I haven't tried oveclocking it yet)

I forgot one thing: my base uses robots for building and some little logistic. 9-10 FPS on old config was with pretty idle bots, new one is with bots having a bit of work, so I'd say, I may expect +1FPS at lease ;)

Abarel
Burner Inserter
Burner Inserter
Posts: 6
Joined: Wed Mar 13, 2019 10:20 pm
Contact:

Re: Performance optimization - post your saves

Post by Abarel »

AdamK wrote:
Mon May 10, 2021 11:05 am
Just to finish my thread-within-a-thread, [...] so I'd say, I may expect +1FPS at lease ;)
Just in case someone gets here and have similar doubts, Ram bandwidth (not just frequency, but latency) means A LOT difference.
As fun sidenote, save your map before using console commands if you are afraid of achievements, then run this little script (it will take several seconds, be patient):

Code: Select all

/c for key, ent in pairs(game.player.surface.find_entities_filtered({force=game.player.force})) do
	if string.find(ent.name,"locomotive") then
		ent.destroy()
	end
end
My PC went from 13 UPS to 60 UPS on your map :-D

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1161
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

Abarel wrote:
Sat May 29, 2021 1:36 am
AdamK wrote:
Mon May 10, 2021 11:05 am
Just to finish my thread-within-a-thread, [...] so I'd say, I may expect +1FPS at lease ;)
Just in case someone gets here and have similar doubts, Ram bandwidth (not just frequency, but latency) means A LOT difference.
As fun sidenote, save your map before using console commands if you are afraid of achievements, then run this little script (it will take several seconds, be patient):

Code: Select all

/c for key, ent in pairs(game.player.surface.find_entities_filtered({force=game.player.force})) do
	if string.find(ent.name,"locomotive") then
		ent.destroy()
	end
end
My PC went from 13 UPS to 60 UPS on your map :-D
Destroying all the locomotives is surely not a good workaround ;)
And destroying the whole production is not a good way to save UPS ;)
If You find typos, my mobile is an uncontrollable beast

azesmbog
Fast Inserter
Fast Inserter
Posts: 151
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Performance optimization - post your saves

Post by azesmbog »

AdamK wrote:
Mon May 10, 2021 11:05 am
Just to finish my thread-within-a-thread, today I updated my box from I7-8086K with DDR-2166 to I7-11700K with , both have/had decent mobos, memory also with as low CL as I could easily get, and I'm up from 9-10 FPS to 12-13FPS. I'd say that 20% increase here is significant :) (I haven't tried oveclocking it yet)
I also looked at your map.
I would say 90 percent of construction and transport drones are superfluous :)
However, everyone plays as he personally likes.
I have an average UPS on this map - 18-19: ((And then - in idle.
(on my current map ups 42-43, and even then I find it very low)
DDR-3200 is not very productive for such a map, it would be necessary to raise the frequency to 3600 and higher and lower the timings. if possible.

Abarel
Burner Inserter
Burner Inserter
Posts: 6
Joined: Wed Mar 13, 2019 10:20 pm
Contact:

Re: Performance optimization - post your saves

Post by Abarel »

jodokus31 wrote:
Sat May 29, 2021 8:07 am
Destroying all the locomotives is surely not a good workaround ;)
That was just a joke. :-D
I should have put in bold the "As fun sidenote"...

The real meaning is that the map was using too many of everything, and so the UPS droped a lot. It would work almost the same deleting 70% of trains (10k trains, mostly iddling), 90% of bots (221k logistic + 101k construction bots, mostly iddling), and 40% of inserters (near 100k inserters, half of them iddling most the time). However, Factorio is such a great game that even with this map having 11M entities and 26M connectors, the map is still fun.
I focused the joke on trains as I think the main gaining can be achieved by greatly reducing the amount of trains (1-1 trains are fun, and you can do the same here with just 3k trains, as most the 10k are iddling almot full time), improving the signaling (ok in 3 ways roudabouts, but ugly on 4 ways crossings, stackers and almost everywhere), optimizing the schedules, and even start optimizing inserters as train stations are a big chunk of the inserters used (6 inserters to box, then 6 to belt, to just fill a single blue belt; can replace these 12 inserters with just 2, with a lot less belts and splitters. And there are 3k+ stations... so it is a big task, with big results.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1033
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

Abarel wrote:
Sat May 29, 2021 1:23 pm
It would work almost the same deleting 70% of trains (10k trains, mostly iddling)
oh, are you using vehicle equipment grid mod?
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1161
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

Abarel wrote:
Sat May 29, 2021 1:23 pm
jodokus31 wrote:
Sat May 29, 2021 8:07 am
Destroying all the locomotives is surely not a good workaround ;)
That was just a joke. :-D
I should have put in bold the "As fun sidenote"...
I wondered a bit, if it was really a joke and also wanted to prevent, that somebody blindly copies the command and destroys his save

Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly
If You find typos, my mobile is an uncontrollable beast

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1033
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

jodokus31 wrote:
Sun May 30, 2021 10:00 am
Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly
please cite a source or provide a test case where this is true, because the pathfinder will not search both directions if it's not even possible to go both. it only searches from the locomotive side. RoRo with single loco facing front would be the least expensive.
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1161
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

ptx0 wrote:
Sun May 30, 2021 6:08 pm
jodokus31 wrote:
Sun May 30, 2021 10:00 am
Regarding trains: If you make trains double-headed and have dead-end stations, the path finder gets relieved significantly
please cite a source or provide a test case where this is true, because the pathfinder will not search both directions if it's not even possible to go both. it only searches from the locomotive side. RoRo with single loco facing front would be the least expensive.
If the path finder can skip a whole rail contraption, because there is no simple way through it (excluding forwards and then backwards), it should save performance. Assuming the path finder does it, but i have no source or evidence for this...
If You find typos, my mobile is an uncontrollable beast

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1033
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

jodokus31 wrote:
Sun May 30, 2021 7:18 pm
If the path finder can skip a whole rail contraption, because there is no simple way through it (excluding forwards and then backwards), it should save performance. Assuming the path finder does it, but i have no source or evidence for this...
the pathfinder is available on github
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1161
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

ptx0 wrote:
Mon May 31, 2021 2:50 pm
the pathfinder is available on github
Interesting. Takes a bit of time to get into it.
If You find typos, my mobile is an uncontrollable beast

Post Reply

Return to “General discussion”