Factorio Forums

Hunt for more stable benchmarking results

- Factorio benchmarks seems to have 1%-3% accuracy
- I've tried using Windows Safe mode - expected more stable numbers
- Also turning Debug on/off (F5 before save) and setting the speed to normal or 64x - just to see if there is any effect - expected no effect
- Run the test saves (100 iron smelting lanes) 8x, 50.000 ticks each

Results

- Windows Safe Mode does not give more stability (just a little bit better)
- Safe mode gives +3.78% UPS, proportional - fine, actually I've expected more
- Debug mode and speed set in the savegame has no effect on the measurements
- No idea why one of these saves has outlier numbers

Saves

- 100 lanes of iron smelting - see this and this thread
- "d" for debug more: "dn": no "dy":yes
- "s" for speed: "s1": normal speed "s64": saved with 64x speed set
- saves are created from the same savegame, setting the parameters and saving immediately

Notes

- Still looking for methods making the benchmarks in Factorio more stable
- I'll move back to 500 lanes - these tests had 100 lanes leading to 1000+ UPS
- Saves, results attached
- the 1% bar is... ...roughly

I think the best benchmark would be headless Factorio on Linux with no GUI and all unnecessary processes turned off. For example Arch linux or Ubuntu server is pretty close to that out of the box.

I find it interesting that you expected better performance out of Factorio in Safe mode vs not.

While yes, Safe mode loads without a lot of the extra processes and overhead, it also loads with generic drivers for the bare necessities. So while on the one hand, yes, there’s less stuff running to compete with it, it’s also going to be working with unoptimized drivers for everything.

joonazan wrote: ↑

Fri Aug 05, 2022 12:26 pm

I think the best benchmark would be headless Factorio on Linux with no GUI and all unnecessary processes turned off. For example Arch linux or Ubuntu server is pretty close to that out of the box.

Yah, this one. a linode running arch gets to just about the minimal set of overhead processes if anybody needs it, but on almost all maps I get within 1% repeatability by just closing the browser and steam, everything else uses basically zero cpu; in one, maybe two cases(?) I have had to shut off the wifi too, no idea if that was transitory/driver issues or the maps driving cache bandwidth right to my rig's limits.

But that's the thing: benchmark-land is not reality-land. It's a good rough comparison if none of the competitors is trying to optimize for that particular metric, but once anybody's actually relying on the benchmark metric the system has already, and literally, been gamed. Best not to worry about a few percent unless everybody knows it's mostly for shits and giggles.

FuryoftheStars wrote: ↑

Sat Aug 06, 2022 5:44 pm

I find it interesting that you expected better performance out of Factorio in Safe mode vs not.

While yes, Safe mode loads without a lot of the extra processes and overhead, it also loads with generic drivers for the bare necessities. So while on the one hand, yes, there’s less stuff running to compete with it, it’s also going to be working with unoptimized drivers for everything.

CPUs are pretty much "whatever" in terms of Driver support. it's baked into the OS binaries.

though I agree about the GPU, for sure. and audio, unless it's completely disabled in Safe Mode.