Mod Benchmark / Performance Comparison: Project Cybersyn/Logistics Train Network/Rail Logistics Dispatcher

Looking for a mod? Have a review on a mod you'd like to share?
Post Reply
WristWatch
Manual Inserter
Manual Inserter
Posts: 2
Joined: Tue Dec 13, 2022 1:05 am
Contact:

Mod Benchmark / Performance Comparison: Project Cybersyn/Logistics Train Network/Rail Logistics Dispatcher

Post by WristWatch »

Hi everyone,

I've just completed a benchmark for these three mods:
Project Cybersyn (I'll simply refer to this mod as Cybersyn from here on) by mami
Logistic Train Network (LTN) by Optera
Rail Logistics Dispatcher (RLD) by Viidi
I also contribute to English locale for Rail Logistics Dispatcher.

Setup:
For the test, I downloaded a copy of farcast's map, Completely Combinator Controlled Trains. (If you're reading this, farcast, you are insane to have built something like this, and you've inspired me to do more sushi belts in the future :D) You can go here (FactorioBox) to see what the map looks like even if you don't wish to download it.

I then spent some a lot of time converting the map such that each mod had its own dedicated save, with supply/request thresholds synced up as much possible. Settings were similarly synced up. (If you find an inconsistency between maps/settings please let me know, as it's likely an oversight on my part.) The stations helpfully marked legacy/incompatible on the map tags were not converted for usage by any of the mods. The map itself is mostly unchanged: I did disable the combinators' ability to control the signals & stations, otherwise the trains would get stuck at a permanently red signal/s. I also swapped the signals on the stackers from rail signals (which were also circuit controlled) to chains so that the trains wouldn't block each other.

Fortunately, the extremely extensive circuitry - There are over 35,000 combinators! - on the map allowed to easily deduce appropriate request threshold, as each station had a train load and buffer size and train load combinator to help me puzzle it out.
I simply applied the following formulae:
Station Limit = Ceiling(Buffer/2)
(The station limit for outposts was instead based on the number of available stackers)

Request = Buffer + 1/2Train load
This had some exceptions due to higher thoughput/consumption:
  • Oil intake= Buffer x 3 + 1/2 Train Load
  • (Southern) One Stone Requester in the central furnace = Buffer * 2 + 1/2 Train Load
  • Low Density structures similarly had 1 Copper Plate requester = Buffer * 2 + 1/2 Train Load
  • Each of the Petroleum Requesters = Buffer + 3/4 Train Load
Now, the map uses exclusively 2-4 trains running on nuclear fuel. There are 110 train in total on the map, though only 106 are left enabled for the mods to utilize. With the exceptions of the Science packs and U-235, all trains run a max train load. Apart from Space science, those resources run half a train load.

Each mod's map was then further divided into 4 with the following:
  • Using both Priority and Train limits
  • Using Priority but not Train limits.
  • Using Trains Limits but not Priority.
  • Using neither.
Priority (when present) is only present on central smelter inputs & outputs, as well as the crude oil requester station. Depots still had Train Limits regardless of the tests.

One thing that could not be perfectly replicated across the mods was a fuel station. Both Cybersyn and RLD have options for it. However, LTN does not have this option. Instead a Priority Depot was used as a psuedo-fuel station. These stations also kept their priority signal regardless of the test.

All tests were performed with the following:
Hardware info
Factorio Version: 1.1.74 build 60256
Mod Versions
  • Cybersyn: 1.2.12
  • LTN: 1.18.2
  • RLD: 0.0.18
  • flib 0.12.4
settings
The listed mods are the only ones enabled during the tests. Flib is only enabled with Cybersyn and LTN as it a dependency for both of those mods. Each save begins right around the time the trains pull into the depot stations for each mod. Each test map/save was run for 108000 ticks (30 minutes), three times. Factorio's built-in --benchmark command line parameter was used to record performance data.

The only performance data I was initially interested in was that of the mods themselves. That falls under the ScriptUpdate tab, which unfortunately cannot be further narrowed down to a per-mod metric to my knowledge. On advice from Walter over on the technical factorio discord, I decided to add Trains and Train Pathfinder to the comparison as well. Total UPS (wholeUpdate) was mostly ignored, but did still prove useful for discarding certain tests for inconsistencies not found on others. Including at least one that was contaminated by, of all things, Windows Update. spoiler:
Yes, Windows Update is bad for UPS. What did you expect?
.

All the test data, as well as the saves, can be found here: (Google Drive link). The gathered data alone close to 1 GB in size, so keep that in mind if you're on a metered/limited connection of some kind. There's also production statistics for each map at the end of the 30 minutes test period in the relevant folders. (Note: production stats for RLD test were using the 0.17 version of that mod)

Results:
Let's start with Script Update, which is as close as I've been able to get to raw mod performance.
Script Update.png
Script Update.png (13.96 KiB) Viewed 6896 times
All of these are the 30 min average across all 3 tests for that map. Except for Total, which is the 30 min average across all 12 tests.

Looking at these metrics, Cybersyn has already pulled a clear lead ahead of LTN, averaging nearly 0.01ms faster. While it isn't the ~3x performance gap Mami claims can be achieved, it's quite likely that this base simply isn't able to push the mods that far apart. (Mami has also mentioned to me that she wasn't using the --benchmark test to measure her mods performance when I reached out to her, as she wasn't aware of the command line parameter until I mentioned it to her. That claim was based on statistics from a profiler.) RLD on the other hand trails behind, at close to a 67% increase in UPS cost from Cybersyn.

While I'm no data scientist, I do still find that averages alone don't tell the whole story, or even simply enough of the story, so let's dive a little deeper:
These graphs are all taken from the data I've linked elsewhere in this post, altered here for better visibility and clarity. Blue is the actual data, while Black lines are moving average trendlines with a period of 60.
Cybersyn NPNL 3.png
Cybersyn NPNL 3.png (21.46 KiB) Viewed 6896 times
The initial dip in ms per tick is one thing that stands out to me. But we don't have anything to compare Cybersyn's graph to, so let's proceed to the next one.

LTN NPNL 2.png
LTN NPNL 2.png (20.03 KiB) Viewed 6896 times
We can clearly see that despite performing worse overall, LTN is by far the more consistent option, with almost no large spikes, and few spikes in general.

RLD PNL 1.png
RLD PNL 1.png (24.84 KiB) Viewed 6896 times
RLD meanwhile, gets even less appealing; It spikes high, and spikes often, literally going off the charts. It spikes so often in fact, that one can almost argue it has a second 'base' level right around the 1.3-1.4 ms per tick mark.

Incidentally, these tests were one of the ones that contributed to LTN's 'worst' average, and the same to RLD and Cybersyn's 'best'. Graphs for the averages themselves weren't used for the same reason i didn't just leave it at the overall averages: it smooths the graphs out too much to accurately show what it would look like in a real game.

Let's proceed to the the entities that these mods manage: Trains. (Remember that Train Pathfinder is a subset of the Trains metric)
Trains.png
Trains.png (12.64 KiB) Viewed 6896 times
Train Pathfinder.png
Train Pathfinder.png (12.97 KiB) Viewed 6896 times
Now this certainly was interesting to look at: Cybersyn again manages to handily claim the lowest ms per tick average on the trains. And the real surprise, and certainly not a result I was expecting; LTN lost all the UPS it saved over RLD in the Script Update phase, and then lost even more. Across all 12 tests, LTN averages more than double Cybersyn's train pathfinder ms time, and almost double RLD's. However that only translates to about 0.25-0.29 ms more on the trains themselves.

When I saw this result, I initially attributed it to the fact that with both Cybersyn and RLD, trains can receive new schedules without having to park at the depots. But then I realized it was still much too high. And then I recalled that LTN draws from its provider stations far more evenly than the other two mods; In fact it will happily send trains to collect resources from the northeasternmost outpost, whereas RLD and Cybersyn will prioritize providers closer to the requesters. In fact, if you'll look back to my setup, I only have "priority" on the central smelter input, and on the oil requester station. Given that information, I believe the high train values are a factor of both longer journeys for trains, and a higher number of trains in general. (Determining which of the two is higher is beyond the scope of my experiment) I'm quite certain better use of prioritization on requesters and providers will allow LTN to close the gap between them significantly, potentially allowing it to retake the #2 spot. However, I find it highly unlikely that LTN will be able to claim #1 even under the best of circumstances.

With all that said, let me know if you see more in the data, and especially if what you see leads you to disagree with my conclusions. :)


Summary/TLDR:

Combined.png
Combined.png (14.27 KiB) Viewed 6896 times
Project Cybersyn is the unambiguous leader in UPS performance, with RLD being a somewhat distant second. LTN is last on this test, but has the most consistent Script Update of the three. It does appears to suffer a significant cost when one doesn't take full advantage Priority signals, while Cybersyn and RLD are 'smart' enough to mitigate that. I am fairly certain that the gap between LTN and its two new competitors can be closed significantly with proper use of Station Priority, potentially allowing it to take #2 from RLD in ideal circumstances. Unfortunately, as Cybersyn already performs better script-wise, I don't see LTN taking #1 from Cybersyn.

Future Recommendations:
For this particular map and setup, expanding the test to see how all three mods (LTN in particular) handle a more extensive and better use of station priority would be a excellent test to perform. One test that is probably better performed on a different map would be seeing how well they scale up to 200-300 trains and beyond. Seeing how the mods cope with different train sizes, different networks, different surfaces, multi-item stations etc. would be other tests that yield useful results.

Special thanks to Xorimuth, Curiosity, and also Walter and the rest of the Technical Factorio discord for advice and assistance on setting up the test/data recording.
Last edited by WristWatch on Sun Jan 15, 2023 12:19 pm, edited 1 time in total.

User avatar
Optera
Smart Inserter
Smart Inserter
Posts: 2920
Joined: Sat Jun 11, 2016 6:41 am
Contact:

Re: Mod Benchmark / Performance Comparison: Project Cybersyn/Logistics Train Network/Rail Logistics Dispatcher

Post by Optera »

Nice to see my efforts preventing micro-stutter are this visible.
Apart from that I'm really surprised to see LTN doing this well on average update time.
WristWatch wrote:
Sun Jan 15, 2023 3:21 am
I recalled that LTN draws from its provider stations far more evenly than the other two mods; In fact it will happily send trains to collect resources from the northeasternmost outpost, whereas RLD and Cybersyn will prioritize providers closer to the requesters. In fact, if you'll look back to my setup, I only have "priority" on the central smelter input, and on the oil requester station. Given that information, I believe the high train values are a factor both longer journeys for trains, and a higher number of trains in general.
Correct, LTN only picks closest Depot to Provider. It felt better to equally drain all providers by going for the highest capacity rather than quickly draining the closest mines.
Whether it's from number of trains or travel distance doesn't really matter, both are only dependent on rail network and base mechanics.
What matters is that without manually optimization by splitting the map into regions with NetworkID LTN does worse in this metric.
I'm curious to hear from others if this difference is big enough to warrant changing picking closest providers over highest capacity ones.

PS: Perhaps this should be moved into "Questions, reviews and ratings": viewforum.php?f=88

Post Reply

Return to “Questions, reviews and ratings”