Performance optimization - post your saves

[Architector_4]

Factorio severely loses performance when moving items with inserters between chests with large inventories with (exaggerated example) 65535 slots, even when 65534 of these slots are empty 100% of the time. I have noticed this with one of my modded playthroughs, where UPS goes down to 20 and worse, due to me (having a slow laptop and) using huge chests with inventory size of 1000. I feel like this could be optimized way better, since the game knows at all times that there's only 1 slot used in both chests. I've started another topic detailing the issue: viewtopic.php?f=6&t=97923&p=543996

To not duplicate uploads, I think I better should link to one of my replies in that same topic where I've uploaded a save file that shows the issue loud and clear: viewtopic.php?p=543961#p543961

[AdamK]

I'm hitting possible performance bottleneck. My UPS is around 10, yet my CPU cores are under utilized, none of them even close to 100% usage. I know that Factorio is not very thready, but I'd expect at least one core close to 100%.

Factorio screenshot: https://www.dropbox.com/s/arkzivruaf4hs ... 0.png?dl=0
CPU usage screenshot: https://www.dropbox.com/s/1i2m94qlbvu7b ... 0.png?dl=0
Save file: https://www.dropbox.com/s/0gro4fo6qe6di ... e.zip?dl=0

[NotRexButCaesar]

I'm hitting possible performance bottleneck. My UPS is around 10, yet my CPU cores are under utilized, none of them even close to 100% usage. I know that Factorio is not very thready, but I'd expect at least one core close to 100%.

Factorio screenshot: https://www.dropbox.com/s/arkzivruaf4hs ... 0.png?dl=0
CPU usage screenshot: https://www.dropbox.com/s/1i2m94qlbvu7b ... 0.png?dl=0
Save file: https://www.dropbox.com/s/0gro4fo6qe6di ... e.zip?dl=0

You are probably running out of memory capacity.

[AdamK]

32GB, so no not really.

[jodokus31]

32GB, so no not really.

Depends on the map. I didn't look at yours, but I had a look at somebody else huge map and my 32GB were full and additional 20GB of swap were used.

[AdamK]

Sure, yes, depends on the map. But I'm not even close to using more than 32GB with my map. My total memory usage is about 28GB with map I posted and a lot of other stuff in the background (5 digital stores, LibreOffice, Discord, the usual).

[jodokus31]

OK, definitely not a matter of memory size. I have 13 GB usage
If you press F4 and activate "show-time-usage" you see...

this

Screenshot from 2021-05-05 20-15-14.png (783.82 KiB) Viewed 8167 times

The biggest chunks are:
- trains(25ms): 10000 trains / 3000 stations is pretty massive. Also you have loops all the way (path finder is 12ms)
- entity update (21s): Lots of moving entities like inserters, assemblers, etc. ways to reduce would be:
* more direct insertions
* the super modular approach probably hurts: if have to have to load/unload for every step, you have a lot of inserters
- transportlines(6s): belts, splitters(aka balancers)

In order to have 60 UPS you have to reduce everything to 16,6666ms

Why is one core not fully used? Idk

[ptx0]

Sure, yes, depends on the map. But I'm not even close to using more than 32GB with my map. My total memory usage is about 28GB with map I posted and a lot of other stuff in the background (5 digital stores, LibreOffice, Discord, the usual).

i took a look. it runs at 15 UPS on my system, and you've got the definition of an inefficient build style there.

everything has to be checked on every tick, and you've got 9000 friggen trains, bro. that's absurd. you need to reduce the number of locomotives by increasing the size of cargo wagons.

[jodokus31]

you've got the definition of an inefficient build style there.

I would say, inefficient mostly in terms of UPS. It's a quite clean approach and has a certain beauty to it

[AdamK]

I never claimed that my base is anywhere close to being optimal. I know it isn't, this is my first serious attempt to build a megabase.

What I claim is that it seems that my UPS is low while CPU seems to be underutilized. I do not claim all cores should be utilized fully, but I expect at least one core to be close to 100% utilization. Memory is not an issue in this case, nor GPU. It seems that there is some other bottleneck.

[AdamK]

everything has to be checked on every tick, and you've got 9000 friggen trains, bro. that's absurd. you need to reduce the number of locomotives by increasing the size of cargo wagons.

This is my experiment with short trains. I like to check 'the other way' of doing things, and seeing many bases with long or extremely long trins, I wanted to do short ones

Btw. most of the designs used are not mine, they are Nilauses. There is one originally by me, I call it 'Uranium carousel', it produces steady output of 22 U-235 without any circuritry. You can find it left from the center of the base. It is actually overbuild by a factor of two (concering centrifuges), but I didn't have need to fix it yet

[disentius]

Can't find the discussion, but I remember a discussion where Devs stated that the biggest bottleneck is not the amount of RAM, but the data transfer between RAM and CPU, and RAM speed
This would be an explanation for the CPU usage not maxing out on one core.

[orzelek]

There is a chance that main game thread is being scheduled on different cores of CPU.
This will look like no cores are in full use while game is bottlenecked by CPU speed and/or RAM latency.

[ptx0]

I never claimed that my base is anywhere close to being optimal. I know it isn't, this is my first serious attempt to build a megabase.

What I claim is that it seems that my UPS is low while CPU seems to be underutilized. I do not claim all cores should be utilized fully, but I expect at least one core to be close to 100% utilization. Memory is not an issue in this case, nor GPU. It seems that there is some other bottleneck.

memory latency. my system runs at 15 UPS and it's still not going to suck down a full core. that's just not how it works

[AdamK]

I'm not sure if memory latency would show itself in %CPU ussage (on one hand, if CPU instruction execution is stalled by need to wair on memory, it should bee seen as %CPU usage, on the other hand, modern CPUs are so complex I can't be sure here.

I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.

[ptx0]

I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.

latency doesn't change as much with frequency as it does if you get high freq DIMMs with low-CAS timings.

[jodokus31]

I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date

[ptx0]

I have DDR4-3600 16-19-19-39 and a Ryzen 5600. And it's still only 16 UPS. I'm sure, there are better rigs out there, but i can't imagine that it gets extremely much better (like 2x) at current date

i'm on DDR4-2666 Kingston ECC memory + Ryzen 3700x and it's 15 UPS because i'm using mimalloc on Linux as my system-wide memory allocator. if I start it up with glibc malloc there's 11 UPS. i'm sure with 4400MHz memory CL19 and mimalloc you might even get 25UPS! but not 30 UPS. probably never 30 with this map.

[AdamK]

I'm about to upgrade my box (CPU, mobo and memory) from DDR4-2133 to DDR4-3200 I should have a bit better latency, a bit more CPU L3 cache. If latency theory is true, I should see nice UPS improvement.

latency doesn't change as much with frequency as it does if you get high freq DIMMs with low-CAS timings.

When I was selecting new DIMMs I've chosen ones with the lowest CAS timings I could get.

But I think that memory latency is not the problem here. I'm pretty certain that latency is included in CPU%, as that are cycles CPU is using. @ptx0 malloc comment is interesting, this may move me back to Linux for this

[ptx0]

But I think that memory latency is not the problem here. I'm pretty certain that latency is included in CPU%, as that are cycles CPU is using. @ptx0 malloc comment is interesting, this may move me back to Linux for this

don't report any bugs if you decide to do that.