UPS optimization - 100vs500 lanes (spoiler: 500 lanes results are more stable)

[Belter]

Problem

- Seeking for more stable benchmarking results I'd expect a bigger load to give more accurate results
- Like a smelting benchmark testing w/100 lanes (1000 UPS) vs w/500 lanes (160 UPS): 500 lanes should be better

Test

- I took 3 blueprints and ran them 5 x 25.000 ticks for each (see this interesing thread about smelting opt)
- Turned off running apps and let it run alone, using Win10 and an old i5 CPU

Results

- 500 lanes tests give better accuracy and separates the results a bit better from each other
- I recommend scaling the benchmarks to the 100 UPS range instead of 1000 UPS to have more reliable results

Details

- Some calculations 500 lanes/100 lanes follow:
- UPS Max difference in Avg UPS withing the 5 runs of the same BP: 0.77% / 1.39% (smaller is better. 500 wins)
- UPS difference between the avg UPS between the different BPs: 4.75% / 3.24% (higher is better, 500 wins)
- Y axis: marks are set to 1% - hence the odd numbers. The 1% yellow rectangle is accurate this time

[Tallywort]

Interesting, though I wonder how those test results stack up if you take into account that you could run that 100 lane test 6 times longer than the 500 lane one.

[Khagan]

I wonder how those test results stack up if you take into account that you could run that 100 lane test 6 times longer than the 500 lane one.

This assumes that Factorio is ergodic.

[Tallywort]

How is the system we're looking at non-ergodic? We can get nice means out of it, AND expect those means to be the same between tests.

If the system were non-ergodic, what even are we measuring?

[Khagan]

How is the system we're looking at non-ergodic?

I fear you are taking my tongue-in-cheek comment more seriously than intended.

The assumption of ergodicity is entirely reasonable. Major mathematical kudos if you can actually prove it ...