[0.17.x] --benchmark resulting map-saves

[someone1337]

I failed to find a way to dump the resulting map of one or all --benchmark run(s).

If I run: bin/x64/factorio --disable-audio --benchmark saves/smelter.zip --benchmark-ticks 1000000 --benchmark-runs 3

It will happily run the map 3 times for 1m ticks, but once its done, there dont seem to be any relevant outputs.
I work-arounded the missig periodic and final production stats with: https://mods.factorio.com/mod/some-benchmark which dumps production stats every 5k ticks, but there also seems to be no way to know what the final tick number will be within lua which makes these stats unreliable.

The reason for this missing-feature-report - its not really a bug - is that I was debugging a single benchmark run where ups seemed a little tooo good, to be true, but production lacked way behind, which doesnt make any sense, while the other runs were "fine". I can only guess what happened: likely a temp. train deadlock.

What I expect
Some commandline option that will allow me to enable saving of the resulting map(s). For example: --benchmark-save which has for example %n as expansion for the benchmark's run number.

bin/x64/factorio --disable-audio --benchmark saves/smelter.zip --benchmark-ticks 1000000 --benchmark-runs 3 --benchmark-save saves/smelter_result_%n.zip
-> Would run the map 3 times and create (so I can load them and look into issues)
saves/smelter_result_1.zip
saves/smelter_result_2.zip
saves/smelter_result_3.zip

What I expect 2
A way to read from lua what the last tick of the game will be, if factorio is started with --benchmark-ticks or --until-tick

[Bilka]

The reason for this missing-feature-report - its not really a bug

Moved to ideas and suggestions

[Uthrom]

While doing some benchmarking, I realized it would be nice to be able to "fast-forward" some time either between benchmarks, or as a way to prime a production chain before performing a benchmark, and essentially being able to save a map after it's elapsed benchmark has finished, would be the (seemingly) easiest way to do it.

So I second the feature request of being able to save a map after it's been benchmarked, both for purpose of validating a map after the benchmark has been run, and to be able to do something like the following:

Code: Select all

# make map however you want
$ ./factorio --benchmark FILE --benchmark-ticks N --benchmark-save <filename optional as far as I care>   #fast forwards
$ ./factorio --benchmark FILE --benchmark-ticks N --benchmark-runs M --benchmark-verbose                  #... as you would normal.

[Uthrom]

TL;DR

Add --save-benchmark or --save-map option, so we can save the map after running the map with --benchmark --benchmark-ticks

What ?

I would like a commandline to factorio that allows me to save the map (optionally, specifying a filename, please?) after running the map in headless mode with --benchmark FILE --benchmark-ticks N.

Why ?

There are times when you need to "prime" a map before benchmarking, and instead of having to sit there idling for however long, it would be nice to "fast forward" through the phase by adding the ability to run the map using the benchmark options, and then tell factorio to save the resulting map.

This makes life for those of us who are starting to (or have been for a long time) delve into the data and performance side of things to do so a little easier, and from a code perspective, it seems simple enough to force a save right before exiting the game that's been run through a benchmark.

At least in theory -- I haven't seen your code.

[Uthrom]

For completeness, this was already suggested here: viewtopic.php?f=6&t=73374

But I never heard anything more about it.

[Allaizn]

Are you aware of the editor and it's time tool? It allows you to specify a tick number and runs the map for exactly that many ticks, which should be sufficient for your use case of priming maps.

This has the additional benefit of you actually being ingame and being able to observe that everything happens as expected, and also allows you to verify that the map ends in a state that is suitable for benchmarking.

If you for some reason don't like that, it's also possible to make a save via LUA - the only caveat is that the output filename will get an autosave prepended, which shouldn't be a problem for your usecase link to the lua doc of the command. Defering the save to the end of the simulation can be done by registering it to one of the ontick events (my knowledge is worse here, I'd guess this should work, and you almost surely want to if the save behind a test to only save at the correct mpa tick.

As a side note, since I made the feature and you seem interested in benchmarking: the benchmark commandline option has more settings than that, most notably a --benchmark-runs option to run multiple times, and a --benchmark-verbose one that outputs the debug timers (that are also visible ingame) of your choosing for each tick that the bench ran. The latter is mentioned here, and I will ask Bilka shortly to add the former one, too, since it seems to have been missed.

[Koub]

[Koub] Merged both topics with same suggestion.

[Uthrom]

Are you aware of the editor and it's time tool? It allows you to specify a tick number and runs the map for exactly that many ticks, which should be sufficient for your use case of priming maps.

This has the additional benefit of you actually being ingame and being able to observe that everything happens as expected, and also allows you to verify that the map ends in a state that is suitable for benchmarking.

I was not aware of the time tool, this is useful. However, since the game still then has to render everything, et cetera, it will be significantly slower than being able to burn through simulated time by just having the game engine run without having to do any I/O to a client.

Since the game is deterministic, it shouldn't matter if I'm in-game or not, if all I want to do is run a build for an hour of game time as fast as possible to make sure everything is full enough.

If you for some reason don't like that, it's also possible to make a save via LUA - the only caveat is that the output filename will get an autosave prepended, which shouldn't be a problem for your usecase link to the lua doc of the command. Defering the save to the end of the simulation can be done by registering it to one of the ontick events (my knowledge is worse here, I'd guess this should work, and you almost surely want to if the save behind a test to only save at the correct mpa tick.

It's unfortunate that everything is already in place except for the save, in order to be able to quickly prep and run benchmarks just using various commandline invocations. And I haven't even tested whether you can do RCON to a benchmarking instance of factorio. (Yes, I could do priming on a headless server using RCON, but again -- benchmarking instances run much faster than non-benchmarking instances, possibly by orders of magnitude).

As a side note, since I made the feature and you seem interested in benchmarking: the benchmark commandline option has more settings than that, most notably a --benchmark-runs option to run multiple times, and a --benchmark-verbose one that outputs the debug timers (that are also visible ingame) of your choosing for each tick that the bench ran.

I am aware: github.com/Uthrom/factorio-tools/tree/master/script-tools.

The latter is mentioned here, and I will ask Bilka shortly to add the former one, too, since it seems to have been missed.

Yeah, I got signed up with a Wiki account so I could make updates that were missing. I found several cases where the documentation does not match reality.

[quyxkh]

You can achieve this by setting the auto-save interval, running the map normally, then benchmarking the autosaves.

[Uthrom]

You can achieve this by setting the auto-save interval, running the map normally, then benchmarking the autosaves.

Factorio benchmark mode does not trigger autosaves, at least not in my testing:

Code: Select all

$ grep autosave-interval ../config/config.ini 
autosave-interval=5
$ ls *autosave*.zip
ls: *autosave*.zip: No such file or directory
$ benchmark-factorio.sh -d -m 10 solo-17.zip 
RUNNING: /Applications/factorio-0-17-66.app/Contents/MacOS/factorio --benchmark solo-17.zip --benchmark-ticks 36000 --benchmark-runs 1
solo-17.zip,27322.905
$ ls *autosave*.zip
ls: *autosave*.zip: No such file or directory
$

I'm running the game for 10 minutes of game time (incidentally, in just over 27 seconds, for an incredible UPS), and autosave-interval is set to 5 minutes as you can see above.

No autosave is generated.

[Uthrom]

There is already the --create option, why not have a variant of a --save option to go with it?

https://wiki.factorio.com/Command_line_parameters:

Code: Select all

--create FILE	create a new map