UPS optimization - benchmarking w/ or w/o mods

This board is to show, discuss and archive useful combinator- and logic-creations.
Smart triggering, counters and sensors, useful circuitry, switching as an art :), computers.
Please provide if possible always a blueprint of your creation.
Post Reply
Belter
Fast Inserter
Fast Inserter
Posts: 128
Joined: Sun Mar 14, 2021 12:38 pm
Contact:

UPS optimization - benchmarking w/ or w/o mods

Post by Belter »

Disabling mods? Suggestion: disable, benchmark vanilla only
- I was not that happy to disable mods for benchmarking tests but it came up there
- How cooler is Editor Extension’s underground belt for creating ore / dumping plates? Yes, cooler than the vanilla exp loader & the inf chest
- But as measured there is a negative UPS impact
- To avoid further issues (like to run into mod which IS bad for benchmarking) I’d suggest testing vanilla. It is just simpler.
- Stability of the results are about the same: 1% (yellow boxes)
The Savegames
- ZM: Zero Mod - Only vanilla elements. Mods disalbed before save.
- EE: Editor Extensions on - same as ZM, so only vanilla elements
- EET: Editor Extensions on, using the Ungerground Belt from EE to produce ore/dispose plates
All attached here.
Results
- ZM and EE is the same savegane and just tested to prove that having EE alone has not impact on the results. That's proven
- Vanilla saves (ZM, EE) run at 187 UPS, EET runs at 166 UPS
- Results shows the same ~1% accurancy in both cases
- There is one outlier result, my bad: I've disabled AV and other tools and the test was already running, ignore that blue bar
- As a bonus you can see 10k, 25k and 50k tick runs - 50k runs give a bit better accurnacy but the differnce is really small
- Another graph from this dataset is posted here
Csv file details
Attachments
UPSvsTicks-ZMEE.png
UPSvsTicks-ZMEE.png (62.66 KiB) Viewed 867 times
UPSvsTicks-EET.png
UPSvsTicks-EET.png (53.23 KiB) Viewed 867 times
UPS-B05-ModOff.png
UPS-B05-ModOff.png (61.83 KiB) Viewed 867 times
UPS-B04-ModOn.png
UPS-B04-ModOn.png (52.54 KiB) Viewed 867 times
B004 - Saves.zip
(11.47 MiB) Downloaded 220 times
B005 results.csv
(429 Bytes) Downloaded 216 times
B004 Results All.csv
(13.01 KiB) Downloaded 218 times
Last edited by Belter on Sat Feb 26, 2022 6:28 pm, edited 1 time in total.

User avatar
disentius
Filter Inserter
Filter Inserter
Posts: 659
Joined: Fri May 12, 2017 3:17 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by disentius »

Ha Belter,

Will you take this save in your bechmark?
It beats everything we made up unto now :)
It is from the ups master Stevetrov.
Attachments
Stevetrov_2feed-v1 500.zip
(3.95 MiB) Downloaded 221 times

fallenghostplayer
Manual Inserter
Manual Inserter
Posts: 4
Joined: Wed Mar 16, 2022 11:47 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by fallenghostplayer »

The one other concern I have with benchmarking with mod entities is that it is unclear whether it takes in the same ups as an equivalent batch of miners or even infinite chests+loader in vanilla. After testing, I know for a fact that there is a difference between those two vanilla methods, so I would expect similar differences to occur with modded entities.

The impact is a slight bias towards increasing or decreasing the input lane count to optimize the last few percent, as the UPS per item produced changes whether you are at 7.5 or 22.5 items per lane. As such, testing with different number of source items would produce different UPS results than a vanilla version of the same thing.

I would expect that other entities could have similar behaviour.

User avatar
disentius
Filter Inserter
Filter Inserter
Posts: 659
Joined: Fri May 12, 2017 3:17 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by disentius »

All ups measures are relative.
Hardware, OS, configuration, mods, all differ between all players.
So it doesn't matter if you use editor extensions or not. it is far more important that all the maps for testing have the same configurations, and the same amount of mod elements, whether it is the EE loader/unloader or the vanilla version.

fallenghostplayer
Manual Inserter
Manual Inserter
Posts: 4
Joined: Wed Mar 16, 2022 11:47 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by fallenghostplayer »

Agreed, all UPS is relative to a number of variables. Let's assume for a moment that we are talking about the same design, on the same machine. What I mean is that when comparing varied designs, the impact of the mod elements such as ore sources, may* have different trends than vanilla elements with different item rates, causing variants with different amounts of the ore sources (though with the same total ore rate) to behave differently UPS-wise.

This is not an issue for the comparison at a conceptual stage when all players agree to use a common mod setup and a common ore source (ex: EE, infinite chests), but when translating the design back to the vanilla ore sources (ex: miners or train unloading), the UPS-optimal number of ore sources vs transport line designs maybe different. True, it's a very small effect, and can be neglected most of the time, but will be noticeable trying to get the last (fractions of a) percent out of very optimized designs in a factory settings. As evidence, I submit the following comparison of miners vs infinite chests (https://www.reddit.com/r/technicalfacto ... te_chests/)

*haven't tested EE, but I would expect some effect. How much depends on many factors, as you said. Since mods rely on lua scripts (is that correct?), I don't know what that means for performance.

SoShootMe
Filter Inserter
Filter Inserter
Posts: 294
Joined: Mon Aug 03, 2020 4:16 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by SoShootMe »

disentius wrote:
Mon Mar 21, 2022 1:18 pm
All ups measures are relative.
Absolutely, but for comparison purposes you want to:
  • Minimise the overhead of things you are not testing (so the relative UPS of different but functionally equivalent designs is closest to their relative UPS effect).
  • Maximise ease for others to repeat tests.
  • Minimise potential sources of variation affecting consistency of results.
All these favour benchmarking without mods.

User avatar
disentius
Filter Inserter
Filter Inserter
Posts: 659
Joined: Fri May 12, 2017 3:17 pm
Contact:

Re: UPS optimization - benchmarking w/ or w/o mods

Post by disentius »

agreed:) that is why we stopped using EE.

Post Reply

Return to “Combinator Creations”