0.17.xx optimizing base for ups

[jape3]

Recently I've run into slight ups issues with a large-ish base. I'm considering starting a new base, and was wondering what practices should I use to maximize performance? I like to play with belts / inserters + trains for bulk and only use bots for localized deliveries. I've read through a few threads about this issue, but many seem to be quite old and I suspect they might no longer apply.

-Do idle machines contribute to ups loss? So should I demolish any unused or legacy areas of base? Does powering off stuff relieve the load on ups?
-Saw some threads that minimizing belt segments (= use as much undergrounds as possible) helps keep ups up, is this still the case or has this been optimized? Do long belt transfers eat more cpu resources than moving the same amount of product on a short belt, or is it just the amount of interactions that happen with the belt that eat CPU time?
-Do long pipe runs still load the cpu?

Also, is there a list of things that must be run on the "main" game thread? Any things that can be shifted to another cpu thread obviously won't provide a bottleneck with the availability of crazy number of cpu cores these days..

[Jap2.0]

Quick reply to the main points:
- If the machines aren't doing anything, getting rid of them is the best thing to do, and cutting power is the worst
- Underground belt's don't help any more
- Pipes are somewhat multi-threaded, but still aren't great
- Not much that you can use is threaded

Long unedited semi-speculative ramblings

-Do idle machines contribute to ups loss? So should I demolish any unused or legacy areas of base? Does powering off stuff relieve the load on ups?

This has been debated. I believe the current conclusion is that powering things off is one of the worst things to do ups-wise; the accepted way to stop part of your base you're not using is to back up the output (eg. stop belts coming out and wait for them to back up, or remove output inserters or something). If it's entirely unused, completely removing it would be the best thing to do.

-Saw some threads that minimizing belt segments (= use as much undergrounds as possible) helps keep ups up, is this still the case or has this been optimized? Do long belt transfers eat more cpu resources than moving the same amount of product on a short belt, or is it just the amount of interactions that happen with the belt that eat CPU time?

Using underground belts is no longer more efficient than using regular belts. As for the rest, I'm not entirely sure. The amount of interaction is definitely a part; I'm not sure if the performance impact is linear with belt length or with the amount of "segments" in a belt, although I'm guessing the latter. (The "segments" I'm referring to are a result of an optimization a few versions back - also the reason underground belts are no longer more efficient - where lines of belts are grouped. Each group of connected belts is a segment, and they are further segmented by splitters and changes in belt type; they also segment every few hundred tiles, and sideloading onto a belt does not merge the two groups of belts.)

-Do long pipe runs still load the cpu?

In 0.17, pipes were multithreaded (as well as other miscellaneous optimizations), meaning that they are now much faster. However, I still doubt that they're great for distances; there were discussions of making long segments of pipes act as a single long pipe, but I don't believe that ever happened. As long as that is the case, I'd say that trains would probably be better for long distances.

Also, is there a list of things that must be run on the "main" game thread?

A lot of it. Belts, inserters, etc, etc... all interact very closely, which makes threading hard. (There's mounds of discussion over this I can point you to if you want - also, memory latency becomes a bottleneck before long as well.) Fluids are multi-threaded, but I believe they still stop the main thread while they work; terrain generation looks like it's done on a separate thread; a lot of graphics stuff is multi-threaded; autosaving can be done off-thread on linux; there may be a few things more, but not much that will help with ups.

[jape3]

Thanks for the pointers, especially I had forgotten about the memory latency being a possible bottleneck too. Something to keep in mind while putting together a new rig that hopefully will not choke on a large base Factorio not running smoothly is a perfectly legit reason to upgrade from 4-core haswell to something like ryzen 3700x, righ?

In the meantime I tried completely removing all biters, turn pollution off and uninstalled rampant enemies mod.. That bumped the ups back up to 60 from the 45-50 range with some room to spare, update now running in about 14.5ms.. Going to see what kind of improvement demolishing old portions of the base & defence perimeter will bring.

[eradicator]

I'll add some ramblings too (kinda similar to @Jap2.0).

Pipes: Yes, underground pipes are (still™) better because you need fewer of them for the same distance. If "the big fluid update"™ ever comes this is likely to change.
Pipes: Different "networks" are processed in parralell. So it's (theoretically!) better to not have all your oil/gas connected to one humongous pipe sytem. I don't think you can force seperation of networks though (except maybe with a train wagon)?

Electricity: On the contraty it's best to have a single monolithic network for the whole map.

Belts: Underground belts are no longer better than straight belts (has been optimized).
Belts: Have been heavily optimized so the length of a belt doesn't matter *that* much anymore. But ofc the more belts you have the more items need to be kept in memory.
Belts: Splitter based balancers are worse than loader+large modded chests based balancers. a) because splitters are expensive (compared to straight belt), and b) because splitters also split belt segments.

Robots: Are rumored to be ever-so-slightly worse than belts after the belt optimization. (I.e. they are no longer a no-brainer-always-faster.)

Machines: If you don't need a machine it's best to block it's output (not input or power). Or demolish if you can ofc, stuff that doesn't exist doesn't cost CPU.

Trains: Try to avoid buffer chests. Every item interaction costs something, and chests means two movements for each item to be loaded.
Trains: Pathfinding is quicker the fewer segments you have. I.e. try avoiding needlessly complicated junctions. Place signals sparsely. Mostly relevant if you have a lot of trains (i.e. hundrets).

Generic: You can see belt/rail signals with the debug view (F4/F5).
Generic: You can see how much the rail pathfinder is costing in debug view too.

[eradicator]

Factorio not running smoothly is a perfectly legit reason to upgrade from 4-core haswell to something like ryzen 3700x, righ? :D

You should definetly read https://factorio.com/blog/post/fff-315 for "Ryzen vs Ram Speed". Personally i'm waiting for DDR5 to hit the consumer market as it's supposed to be like at leat 50% faster. According to wiki that'll be about 2021...

[jape3]

Good pointers, thanks for those.. Especially about RAM I already had 3200MHz modules on the shopping cart but might want to go faster.

I checked the debug list and the biggest time hog is entity update at about 6.8ms.. Transport lines are about 4ms, does that mean all belts or?

Also had a look into big chest mods.. Are there any mods with 4x4 chests? That would be the minimum size that works well for 8 belt lanes.

[Zavian]

Also be aware that Rseding has recommended against using chests with large capacities (which most of the big chest mods have) since that large capacity is actually bad from a performance point of view. (I think the issue is that every time an inserter interacts with the chest, it potentially needs to examine every slot of that chest, so a chest with 200 slots is potentially 10x worse than a chest with 20 slots, from a ups point of view).

[BlueTemplar]

https://mods.factorio.com/mod/LB-Modular-Chests


Merging Chests

(With inventory size multiplier.)

[quyxkh]

Sure, I love a good ramble:

Inserter swings matter.

Fluid and heat junctions matter. Straight-line connections matter less, but enough that needing ~10x fewer ug pipes matters.

Travelling-locomotive count matters.

Active assembler/smelter count matters (max-boost assemblers are very good for UPS).

In-flight bot count matters, bots suck for ups over long routes. Ideally you use bots for the shortest delivery routes possible, like to hop a beacon. I've done things like circuit-control production lines so bots concentrate on one request, just loading copper buffers then just loading cable buffers then just loading chip buffers, like that. You can control your requests so the supplies are always available at the shortest hop, do the refill to half what's in the supply buffers and the circuitry's simple.

Circuit signal changes are individually too small to care about, but the cost is per entity that has to check the new value, so one wire running to hundreds of enable-if inserters/belt segments/whatever should have the results reduced to a single pulse, so all those testing entities don't wake up for nothing. (Yes, MadZuri's balanced-loading stunt is awesome early, later you just use bars and drain your buffers on pickup to maintain balance).

Belts are cheap. You can run a monster yellow-belt ammo distribution network all the way around your defenses and just keep extending it.

[mrvn]

Good pointers, thanks for those.. Especially about RAM I already had 3200MHz modules on the shopping cart but might want to go faster.

I checked the debug list and the biggest time hog is entity update at about 6.8ms.. Transport lines are about 4ms, does that mean all belts or?

Also had a look into big chest mods.. Are there any mods with 4x4 chests? That would be the minimum size that works well for 8 belt lanes.

The frequency (MHz) of your ram isn't the only factor. The clock times are a huge factor too. You can get 3200MHz RAM with CL20 or CL15. The former is 1/3 slower. That's a much bigger factor than 3000MHz vs 32000MHz.

[Koub]

Hi @OP,

Have you seen these topics :
viewtopic.php?f=49&t=72080
viewtopic.php?f=49&t=74585

They might be of some help.

[mrvn]

Trains: Avoid a uniform grid pattern.

The path finding uses the A* algorithm. In an uniform grid there are many different paths from A to B that have exactly the same cost and the A* algorithm will explore all of them in parallel until one of them reaches the destination. It can't exclude any of them from consideration since none of them have a cost advantage over others.

Trains: Intermediate stops

When a train is stopped by a chain signal it will try to find an alternate path. The longer the path the longer this takes. In the grid pattern above the number of alternate paths cost time. Even if the network isn't a grid every junction potentially has to explore both ways. A* is pretty good at only exploring paths leading to the destination but it still has to explore around the perfect path trying to see if something is a shortcut or not. And the simplest fact is exploring a long path costs more time than a short path.

If your network consists of separate clusters with long tracks between them you might consider putting train stops at the entry to each cluster and setting them as intermediate stops in the schedule. The path finding will then only search until it reaches the intermediate stop instead of continuing and branching into all the possible ways inside the next cluster.

[BlueTemplar]

Trains: Avoid a uniform grid pattern.

The path finding uses the A* algorithm. In an uniform grid there are many different paths from A to B that have exactly the same cost and the A* algorithm will explore all of them in parallel until one of them reaches the destination. It can't exclude any of them from consideration since none of them have a cost advantage over others.

Oh, wow, any benchmarks ? A lot of megabase-builders certainly seem to love uniform grids !
https://www.youtube.com/watch?v=dY2nxVNBHQs

[eradicator]

https://www.youtube.com/watch?v=dY2nxVNBHQs

Watching that video i question my understanding of how facotrio is played. At 4:39 ("138h") he starts terraforming an area of 26*28 blocks, each block being 3*3 chunks, for a total of 6.7 million tiles. Terraforming completes after 9 hours. That's 207 tiles per second or 5 seconds per chunk going from natural forest to perfectly floored railworld. Is this a creative run where infrastructure+concrete is free? Do construction bots move faster than sound? (Serious questions, i've never played any of these these super-microcrafting modpacks).

Building that thing in just 500 hours is to unreal for my brain to comprehend.
It's a shame that there seems to be no map download.

[BlueTemplar]

Py has alternate recipes for concrete (and ores) and 3 levels of additional con&logibots (BTW Py Industries that adds them is available as a standalone), and the first level of conbots, available in the science1 era (but very expensive at that point), is already 50% faster than vanilla.

[mrvn]

Trains: Avoid a uniform grid pattern.

The path finding uses the A* algorithm. In an uniform grid there are many different paths from A to B that have exactly the same cost and the A* algorithm will explore all of them in parallel until one of them reaches the destination. It can't exclude any of them from consideration since none of them have a cost advantage over others.

Oh, wow, any benchmarks ? A lot of megabase-builders certainly seem to love uniform grids !
https://www.youtube.com/watch?v=dY2nxVNBHQs

No benchmarks but comments from devs on it in other threads. It's kind of hard to build your megabase so you can turn on/off equal cost paths in a grid pattern to compare the two.

Note that trains repath when they get stoped at chain signals. As your network gets congested this becomes more pronounced. If your network is mostly clear then it's probably irelevant.

[PunPun]

The path finding uses the A* algorithm. In an uniform grid there are many different paths from A to B that have exactly the same cost and the A* algorithm will explore all of them in parallel until one of them reaches the destination. It can't exclude any of them from consideration since none of them have a cost advantage over others.

If your A* implementation does this then you either have a bug or your heuristics are bad. In a uniform grid a properly implemented A* will not explore any alternate paths and goes straight to the goal. That is the whole point of the A* algorithm.

[Bilka]

I'll just leave this here: https://mulark.github.io/test-index.html

[mrvn]

The path finding uses the A* algorithm. In an uniform grid there are many different paths from A to B that have exactly the same cost and the A* algorithm will explore all of them in parallel until one of them reaches the destination. It can't exclude any of them from consideration since none of them have a cost advantage over others.

If your A* implementation does this then you either have a bug or your heuristics are bad. In a uniform grid a properly implemented A* will not explore any alternate paths and goes straight to the goal. That is the whole point of the A* algorithm.

A* expands the path where |path| + H() is the cheapest. The heuristic H() must not overestimate the cost to reach the goal. It usually underestimates it relative to the amount of path remaining. With many paths of identical cost that means the paths that have been least explored will have the lowest estimates cost and will be expanded first. The problem is that in a grid there are many way of going straight. Going 5 north and 5 west is the same as going 5 west and 5 north.

Now in factorio trains can't turn 90° on a spot but have to take a curve. And the curve has a diagonal part. So I'm not 100% sure that in a grid all the paths between 5N+5W and 5W+5N would be the exact same cost. Going NWNWNWNWNW probably differs a bit from NNNNNWWWWW. But the presence of other trains will change those costs too, leading to many path being explored until they hit a train blocking the way and then A* will try all the other "straight" paths that don't yet have a train blocking it.

Anyway, I haven't seen any benchmarks on the effect of grid on train pathing costs but the devs have claimed they are costly. Make of that what you will or make a test map.

[DaleStan]

Anyway, I haven't seen any benchmarks on the effect of grid on train pathing costs but the devs have claimed they are costly.

Do you have a source for that, or are we just supposed to believe you?

The only mention of "grid" in Bilka's link is talking about cars and the collision grid, not trains and a grid-based rail system.