Performance optimization - post your saves

Post all other topics which do not belong to any other category.
User avatar
Architector_4
Burner Inserter
Burner Inserter
Posts: 13
Joined: Sat Apr 17, 2021 2:04 pm
Contact:

Re: Performance optimization - post your saves

Post by Architector_4 »

Factorio severely loses performance when moving items with inserters between chests with large inventories with (exaggerated example) 65535 slots, even when 65534 of these slots are empty 100% of the time. I have noticed this with one of my modded playthroughs, where UPS goes down to 20 and worse, due to me (having a slow laptop and) using huge chests with inventory size of 1000. I feel like this could be optimized way better, since the game knows at all times that there's only 1 slot used in both chests. I've started another topic detailing the issue: viewtopic.php?f=6&t=97923&p=543996

To not duplicate uploads, I think I better should link to one of my replies in that same topic where I've uploaded a save file that shows the issue loud and clear: viewtopic.php?p=543961#p543961

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

I'm hitting possible performance bottleneck. My UPS is around 10, yet my CPU cores are under utilized, none of them even close to 100% usage. I know that Factorio is not very thready, but I'd expect at least one core close to 100%.

Factorio screenshot: https://www.dropbox.com/s/arkzivruaf4hs ... 0.png?dl=0
CPU usage screenshot: https://www.dropbox.com/s/1i2m94qlbvu7b ... 0.png?dl=0
Save file: https://www.dropbox.com/s/0gro4fo6qe6di ... e.zip?dl=0

User avatar
NotRexButCaesar
Smart Inserter
Smart Inserter
Posts: 1120
Joined: Sun Feb 16, 2020 12:47 am
Contact:

Re: Performance optimization - post your saves

Post by NotRexButCaesar »

AdamK wrote:
Tue May 04, 2021 6:05 pm
I'm hitting possible performance bottleneck. My UPS is around 10, yet my CPU cores are under utilized, none of them even close to 100% usage. I know that Factorio is not very thready, but I'd expect at least one core close to 100%.

Factorio screenshot: https://www.dropbox.com/s/arkzivruaf4hs ... 0.png?dl=0
CPU usage screenshot: https://www.dropbox.com/s/1i2m94qlbvu7b ... 0.png?dl=0
Save file: https://www.dropbox.com/s/0gro4fo6qe6di ... e.zip?dl=0
You are probably running out of memory capacity.
—Crevez, chiens, si vous n'étes pas contents!

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

32GB, so no not really.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1599
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

AdamK wrote:
Wed May 05, 2021 6:01 am
32GB, so no not really.
Depends on the map. I didn't look at yours, but I had a look at somebody else huge map and my 32GB were full and additional 20GB of swap were used.

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

Sure, yes, depends on the map. But I'm not even close to using more than 32GB with my map. My total memory usage is about 28GB with map I posted and a lot of other stuff in the background (5 digital stores, LibreOffice, Discord, the usual).

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1599
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

OK, definitely not a matter of memory size. I have 13 GB usage
If you press F4 and activate "show-time-usage" you see...
this
The biggest chunks are:
- trains(25ms): 10000 trains / 3000 stations is pretty massive. Also you have loops all the way (path finder is 12ms)
- entity update (21s): Lots of moving entities like inserters, assemblers, etc. ways to reduce would be:
* more direct insertions
* the super modular approach probably hurts: if have to have to load/unload for every step, you have a lot of inserters
- transportlines(6s): belts, splitters(aka balancers)

In order to have 60 UPS you have to reduce everything to 16,6666ms

Why is one core not fully used? Idk

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

AdamK wrote:
Wed May 05, 2021 10:05 am
Sure, yes, depends on the map. But I'm not even close to using more than 32GB with my map. My total memory usage is about 28GB with map I posted and a lot of other stuff in the background (5 digital stores, LibreOffice, Discord, the usual).
i took a look. it runs at 15 UPS on my system, and you've got the definition of an inefficient build style there.

everything has to be checked on every tick, and you've got 9000 friggen trains, bro. that's absurd. you need to reduce the number of locomotives by increasing the size of cargo wagons. :?

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1599
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

ptx0 wrote:
Thu May 06, 2021 3:04 am
you've got the definition of an inefficient build style there.
I would say, inefficient mostly in terms of UPS. It's a quite clean approach and has a certain beauty to it :)

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

I never claimed that my base is anywhere close to being optimal. I know it isn't, this is my first serious attempt to build a megabase.

What I claim is that it seems that my UPS is low while CPU seems to be underutilized. I do not claim all cores should be utilized fully, but I expect at least one core to be close to 100% utilization. Memory is not an issue in this case, nor GPU. It seems that there is some other bottleneck.

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

ptx0 wrote:
Thu May 06, 2021 3:04 am
everything has to be checked on every tick, and you've got 9000 friggen trains, bro. that's absurd. you need to reduce the number of locomotives by increasing the size of cargo wagons. :?
This is my experiment with short trains. I like to check 'the other way' of doing things, and seeing many bases with long or extremely long trins, I wanted to do short ones :)

Btw. most of the designs used are not mine, they are Nilauses. There is one originally by me, I call it 'Uranium carousel', it produces steady output of 22 U-235 without any circuritry. You can find it left from the center of the base. It is actually overbuild by a factor of two (concering centrifuges), but I didn't have need to fix it yet :P

User avatar
disentius
Filter Inserter
Filter Inserter
Posts: 694
Joined: Fri May 12, 2017 3:17 pm
Contact:

Re: Performance optimization - post your saves

Post by disentius »

Can't find the discussion, but I remember a discussion where Devs stated that the biggest bottleneck is not the amount of RAM, but the data transfer between RAM and CPU, and RAM speed
This would be an explanation for the CPU usage not maxing out on one core.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Performance optimization - post your saves

Post by orzelek »

There is a chance that main game thread is being scheduled on different cores of CPU.
This will look like no cores are in full use while game is bottlenecked by CPU speed and/or RAM latency.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

AdamK wrote:
Thu May 06, 2021 8:16 am
I never claimed that my base is anywhere close to being optimal. I know it isn't, this is my first serious attempt to build a megabase.

What I claim is that it seems that my UPS is low while CPU seems to be underutilized. I do not claim all cores should be utilized fully, but I expect at least one core to be close to 100% utilization. Memory is not an issue in this case, nor GPU. It seems that there is some other bottleneck.
memory latency. my system runs at 15 UPS and it's still not going to suck down a full core. that's just not how it works :cry:

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

I'm not sure if memory latency would show itself in %CPU ussage (on one hand, if CPU instruction execution is stalled by need to wair on memory, it should bee seen as %CPU usage, on the other hand, modern CPUs are so complex I can't be sure here.

I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

AdamK wrote:
Thu May 06, 2021 2:21 pm
I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.
latency doesn't change as much with frequency as it does if you get high freq DIMMs with low-CAS timings.

User avatar
jodokus31
Smart Inserter
Smart Inserter
Posts: 1599
Joined: Sun Feb 26, 2017 4:13 pm
Contact:

Re: Performance optimization - post your saves

Post by jodokus31 »

I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

jodokus31 wrote:
Thu May 06, 2021 3:02 pm
I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date
i'm on DDR4-2666 Kingston ECC memory + Ryzen 3700x and it's 15 UPS because i'm using mimalloc on Linux as my system-wide memory allocator. if I start it up with glibc malloc there's 11 UPS. i'm sure with 4400MHz memory CL19 and mimalloc you might even get 25UPS! but not 30 UPS. probably never 30 with this map.

User avatar
AdamK
Inserter
Inserter
Posts: 47
Joined: Thu Jul 25, 2019 9:11 am
Contact:

Re: Performance optimization - post your saves

Post by AdamK »

ptx0 wrote:
Thu May 06, 2021 2:58 pm
AdamK wrote:
Thu May 06, 2021 2:21 pm
I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.
latency doesn't change as much with frequency as it does if you get high freq DIMMs with low-CAS timings.
When I was selecting new DIMMs I've chosen ones with the lowest CAS timings I could get.

But I think that memory latency is not the problem here. I'm pretty certain that latency is included in CPU%, as that are cycles CPU is using. @ptx0 malloc comment is interesting, this may move me back to Linux for this :)

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Performance optimization - post your saves

Post by ptx0 »

AdamK wrote:
Thu May 06, 2021 5:23 pm
But I think that memory latency is not the problem here. I'm pretty certain that latency is included in CPU%, as that are cycles CPU is using. @ptx0 malloc comment is interesting, this may move me back to Linux for this :)
don't report any bugs if you decide to do that.

Post Reply

Return to “General discussion”