I've just completed a benchmark for these three mods:
Project Cybersyn (I'll simply refer to this mod as Cybersyn from here on) by mami
Logistic Train Network (LTN) by Optera
Rail Logistics Dispatcher (RLD) by Viidi
I also contribute to English locale for Rail Logistics Dispatcher.
Setup:
For the test, I downloaded a copy of farcast's map, Completely Combinator Controlled Trains. (If you're reading this, farcast, you are insane to have built something like this, and you've inspired me to do more sushi belts in the future
) You can go here (FactorioBox) to see what the map looks like even if you don't wish to download it.I then spent
Fortunately, the extremely extensive circuitry - There are over 35,000 combinators! - on the map allowed to easily deduce appropriate request threshold, as each station had a train load and buffer size and train load combinator to help me puzzle it out.
I simply applied the following formulae:
Station Limit = Ceiling(Buffer/2)
(The station limit for outposts was instead based on the number of available stackers)
Request = Buffer + 1/2Train load
This had some exceptions due to higher thoughput/consumption:
- Oil intake= Buffer x 3 + 1/2 Train Load
- (Southern) One Stone Requester in the central furnace = Buffer * 2 + 1/2 Train Load
- Low Density structures similarly had 1 Copper Plate requester = Buffer * 2 + 1/2 Train Load
- Each of the Petroleum Requesters = Buffer + 3/4 Train Load
Each mod's map was then further divided into 4 with the following:
- Using both Priority and Train limits
- Using Priority but not Train limits.
- Using Trains Limits but not Priority.
- Using neither.
One thing that could not be perfectly replicated across the mods was a fuel station. Both Cybersyn and RLD have options for it. However, LTN does not have this option. Instead a Priority Depot was used as a psuedo-fuel station. These stations also kept their priority signal regardless of the test.
All tests were performed with the following:
Hardware info
Factorio Version: 1.1.74 build 60256Mod Versions
- Cybersyn: 1.2.12
- LTN: 1.18.2
- RLD: 0.0.18
- flib 0.12.4
settings
The listed mods are the only ones enabled during the tests. Flib is only enabled with Cybersyn and LTN as it a dependency for both of those mods. Each save begins right around the time the trains pull into the depot stations for each mod. Each test map/save was run for 108000 ticks (30 minutes), three times. Factorio's built-in --benchmark command line parameter was used to record performance data. The only performance data I was initially interested in was that of the mods themselves. That falls under the ScriptUpdate tab, which unfortunately cannot be further narrowed down to a per-mod metric to my knowledge. On advice from Walter over on the technical factorio discord, I decided to add Trains and Train Pathfinder to the comparison as well. Total UPS (wholeUpdate) was mostly ignored, but did still prove useful for discarding certain tests for inconsistencies not found on others. Including at least one that was contaminated by, of all things, Windows Update. spoiler:
Yes, Windows Update is bad for UPS. What did you expect?
.All the test data, as well as the saves, can be found here: (Google Drive link). The gathered data alone close to 1 GB in size, so keep that in mind if you're on a metered/limited connection of some kind. There's also production statistics for each map at the end of the 30 minutes test period in the relevant folders. (Note: production stats for RLD test were using the 0.17 version of that mod)
Results:
Let's start with Script Update, which is as close as I've been able to get to raw mod performance.
- Script Update.png (13.96 KiB) Viewed 7296 times
Looking at these metrics, Cybersyn has already pulled a clear lead ahead of LTN, averaging nearly 0.01ms faster. While it isn't the ~3x performance gap Mami claims can be achieved, it's quite likely that this base simply isn't able to push the mods that far apart. (Mami has also mentioned to me that she wasn't using the --benchmark test to measure her mods performance when I reached out to her, as she wasn't aware of the command line parameter until I mentioned it to her. That claim was based on statistics from a profiler.) RLD on the other hand trails behind, at close to a 67% increase in UPS cost from Cybersyn.
While I'm no data scientist, I do still find that averages alone don't tell the whole story, or even simply enough of the story, so let's dive a little deeper:
These graphs are all taken from the data I've linked elsewhere in this post, altered here for better visibility and clarity. Blue is the actual data, while Black lines are moving average trendlines with a period of 60.
- Cybersyn NPNL 3.png (21.46 KiB) Viewed 7296 times
- LTN NPNL 2.png (20.03 KiB) Viewed 7296 times
- RLD PNL 1.png (24.84 KiB) Viewed 7296 times
Incidentally, these tests were one of the ones that contributed to LTN's 'worst' average, and the same to RLD and Cybersyn's 'best'. Graphs for the averages themselves weren't used for the same reason i didn't just leave it at the overall averages: it smooths the graphs out too much to accurately show what it would look like in a real game.
Let's proceed to the the entities that these mods manage: Trains. (Remember that Train Pathfinder is a subset of the Trains metric)
- Trains.png (12.64 KiB) Viewed 7296 times
- Train Pathfinder.png (12.97 KiB) Viewed 7296 times
When I saw this result, I initially attributed it to the fact that with both Cybersyn and RLD, trains can receive new schedules without having to park at the depots. But then I realized it was still much too high. And then I recalled that LTN draws from its provider stations far more evenly than the other two mods; In fact it will happily send trains to collect resources from the northeasternmost outpost, whereas RLD and Cybersyn will prioritize providers closer to the requesters. In fact, if you'll look back to my setup, I only have "priority" on the central smelter input, and on the oil requester station. Given that information, I believe the high train values are a factor of both longer journeys for trains, and a higher number of trains in general. (Determining which of the two is higher is beyond the scope of my experiment) I'm quite certain better use of prioritization on requesters and providers will allow LTN to close the gap between them significantly, potentially allowing it to retake the #2 spot. However, I find it highly unlikely that LTN will be able to claim #1 even under the best of circumstances.
With all that said, let me know if you see more in the data, and especially if what you see leads you to disagree with my conclusions.
Summary/TLDR:
- Combined.png (14.27 KiB) Viewed 7296 times
Future Recommendations:
For this particular map and setup, expanding the test to see how all three mods (LTN in particular) handle a more extensive and better use of station priority would be a excellent test to perform. One test that is probably better performed on a different map would be seeing how well they scale up to 200-300 trains and beyond. Seeing how the mods cope with different train sizes, different networks, different surfaces, multi-item stations etc. would be other tests that yield useful results.
Special thanks to Xorimuth, Curiosity, and also Walter and the rest of the Technical Factorio discord for advice and assistance on setting up the test/data recording.
