Factorio data-dump mode

Post your ideas and suggestions how to improve the game.
Post Reply
User avatar
sonaxaton
Burner Inserter
Burner Inserter
Posts: 8
Joined: Sat Dec 30, 2017 12:18 am
Contact:

Factorio data-dump mode

Post by sonaxaton » Tue Aug 06, 2019 11:54 pm

It would be fantastic if Factorio provided a command-line option which did the following:
  • Start the application in headless mode.
  • Run the normal game startup process, loading mods and running through the stages described in the docs, except sprites aren't loaded.
  • After all the game data is loaded, dump data.raw to stdout in some recognizable format (JSON ideally, or just what serpent does would probably be fine).
  • Exit.
Personally I would find this incredibly useful as I have been trying to develop a desktop application (completely separate from the game, so not a mod) that provides various tools for calculating recipe ratios and planning factories outside of the game. In order to do this, I need the game data from data.raw for the base game as well as any mods you might have installed. What I have right now is some code which actually emulates the entire startup process of the game (thanks to the fantastic documentation) using a Lua engine and the real game files, and it actually works pretty well. However I've started running into problems because I realized the Lua interpreter I'm using runs Lua 5.3, but Factorio uses Lua 5.2 and there are some subtle differences which cause issues. I could modify the library I'm working with to support Lua 5.2, but it would be so much easier if I didn't have to try to emulate the game startup process at all, and could just use the actual Factorio binary to do it. There may be a hacky way to do this currently by temporarily installing a mod that dumps data.raw to the Factorio logs using serpent, then killing the game process once it detects the presence of the data dump. But it would be much easier if there was a built-in option for this.

I think in general having an easy way to export the game data would be super useful to lots of people who make external Factorio tools. I definitely consider Factorio a very data-driven game, especially with the way mods can add and modify content, and having a way to automatically process that data is the highest level of Factorio game-play. In an abstract sense, the application I'm developing is meant to automate the playing of the game itself!

Happy to answer questions about use-cases for this, and thanks for making a fantastically fun and well-documented game!

Darinth
Filter Inserter
Filter Inserter
Posts: 321
Joined: Wed Oct 17, 2018 12:17 pm
Contact:

Re: Factorio data-dump mode

Post by Darinth » Wed Aug 07, 2019 3:20 pm

Sounds like an ideal candidate for a tiny mod. Once everything is done loading, I'd expect that a mod could dump all of the data to some point. Either stdout or to a file somewhere. Either would then allow a 3rd party application to read the data to do calculations.

User avatar
sonaxaton
Burner Inserter
Burner Inserter
Posts: 8
Joined: Sat Dec 30, 2017 12:18 am
Contact:

Re: Factorio data-dump mode

Post by sonaxaton » Wed Aug 07, 2019 11:01 pm

It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.

quyxkh
Filter Inserter
Filter Inserter
Posts: 727
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: Factorio data-dump mode

Post by quyxkh » Thu Aug 08, 2019 2:16 am

name your mod `~~~` and give it a data-final-fixes.lua, it'll see the finished data, it can be just `print(serpent.block(data))`, I just tried it.

Atlas caches load _fast_ on a hot restart.
Attachments
~~~dumper_0.0.0.zip
(663 Bytes) Downloaded 2 times

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 3250
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Factorio data-dump mode

Post by eradicator » Thu Aug 08, 2019 5:00 am

A custom mod is much better because it allows you to dump exactly the things you want, as everyone as different needs. You can essentially output a pre-formatted dump that is trivial to import into your external application. Additionally if you dump the data in control-stage you get a) access to game.write_file so you don't have to parse the log, and more importantly b) the data from i.e. game.recipe_prototypes is already unified into a standard format, and you don't need to do that yourself anymore (as data.raw in some cases has many different viable definition styles). Via command line you can auto-load a prepared savegame (or two if you need normal+expensive), and if you're so inclined you could make the game crash itself once it's done.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

User avatar
sonaxaton
Burner Inserter
Burner Inserter
Posts: 8
Joined: Sat Dec 30, 2017 12:18 am
Contact:

Re: Factorio data-dump mode

Post by sonaxaton » Thu Aug 08, 2019 1:31 pm

eradicator wrote:
Thu Aug 08, 2019 5:00 am
A custom mod is much better because it allows you to dump exactly the things you want, as everyone as different needs. You can essentially output a pre-formatted dump that is trivial to import into your external application. Additionally if you dump the data in control-stage you get a) access to game.write_file so you don't have to parse the log, and more importantly b) the data from i.e. game.recipe_prototypes is already unified into a standard format, and you don't need to do that yourself anymore (as data.raw in some cases has many different viable definition styles). Via command line you can auto-load a prepared savegame (or two if you need normal+expensive), and if you're so inclined you could make the game crash itself once it's done.
Thanks for the info! I'll try out some of these (TIL about game.write_file). I haven't considered using the in-game API to access data, because I figured loading up an actual save would take too long, but I could maybe do it with a degenerate save file. The in-game API looks to have more consistent ways of accessing data, as you say.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 3250
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Factorio data-dump mode

Post by eradicator » Thu Aug 08, 2019 1:45 pm

sonaxaton wrote:
Thu Aug 08, 2019 1:31 pm
because I figured loading up an actual save would take too long
I have no clue about what you consider "too long" etc, but limiting the height/width of the map makes loading significantly faster. I use a 100x100 map when i need to do a quick test of a change in my mods. Also loading a savegame is faster than starting a new map, thus loading a saved size-limited map would be double speedup.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

mrvn
Smart Inserter
Smart Inserter
Posts: 3520
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Factorio data-dump mode

Post by mrvn » Thu Aug 08, 2019 3:18 pm

sonaxaton wrote:
Wed Aug 07, 2019 11:01 pm
It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.
Simply use the headless server. It doesn't load any sprites. And you can set it to profile the game for 1 tick so it would automatically stop as well. Or add an error in the lua script after the dump so it aborts there.

Darinth
Filter Inserter
Filter Inserter
Posts: 321
Joined: Wed Oct 17, 2018 12:17 pm
Contact:

Re: Factorio data-dump mode

Post by Darinth » Fri Aug 09, 2019 1:36 pm

mrvn wrote:
Thu Aug 08, 2019 3:18 pm
sonaxaton wrote:
Wed Aug 07, 2019 11:01 pm
It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.
Simply use the headless server. It doesn't load any sprites. And you can set it to profile the game for 1 tick so it would automatically stop as well. Or add an error in the lua script after the dump so it aborts there.
That really sounds like the way to do this. Load game with mods, loading a save file profiling for 1 tick and then stopping. No sprite sheet load time, just load mods, dump data, run the game for a tick, and shut down automatically. Easily repeated using a batch file.

Post Reply

Return to “Ideas and Suggestions”

Who is online

Users browsing this forum: No registered users