Factorio data-dump mode

[sonaxaton]

It would be fantastic if Factorio provided a command-line option which did the following:

• Start the application in headless mode.
• Run the normal game startup process, loading mods and running through the stages described in the docs, except sprites aren't loaded.
• After all the game data is loaded, dump data.raw to stdout in some recognizable format (JSON ideally, or just what serpent does would probably be fine).
• Exit.

Personally I would find this incredibly useful as I have been trying to develop a desktop application (completely separate from the game, so not a mod) that provides various tools for calculating recipe ratios and planning factories outside of the game. In order to do this, I need the game data from data.raw for the base game as well as any mods you might have installed. What I have right now is some code which actually emulates the entire startup process of the game (thanks to the fantastic documentation) using a Lua engine and the real game files, and it actually works pretty well. However I've started running into problems because I realized the Lua interpreter I'm using runs Lua 5.3, but Factorio uses Lua 5.2 and there are some subtle differences which cause issues. I could modify the library I'm working with to support Lua 5.2, but it would be so much easier if I didn't have to try to emulate the game startup process at all, and could just use the actual Factorio binary to do it. There may be a hacky way to do this currently by temporarily installing a mod that dumps data.raw to the Factorio logs using serpent, then killing the game process once it detects the presence of the data dump. But it would be much easier if there was a built-in option for this.

I think in general having an easy way to export the game data would be super useful to lots of people who make external Factorio tools. I definitely consider Factorio a very data-driven game, especially with the way mods can add and modify content, and having a way to automatically process that data is the highest level of Factorio game-play. In an abstract sense, the application I'm developing is meant to automate the playing of the game itself!

Happy to answer questions about use-cases for this, and thanks for making a fantastically fun and well-documented game!

[Darinth]

Sounds like an ideal candidate for a tiny mod. Once everything is done loading, I'd expect that a mod could dump all of the data to some point. Either stdout or to a file somewhere. Either would then allow a 3rd party application to read the data to do calculations.

[sonaxaton]

It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.

[quyxkh]

name your mod `~~~` and give it a data-final-fixes.lua, it'll see the finished data, it can be just `print(serpent.block(data))`, I just tried it.

Atlas caches load _fast_ on a hot restart.

[eradicator]

A custom mod is much better because it allows you to dump exactly the things you want, as everyone as different needs. You can essentially output a pre-formatted dump that is trivial to import into your external application. Additionally if you dump the data in control-stage you get a) access to game.write_file so you don't have to parse the log, and more importantly b) the data from i.e. game.recipe_prototypes is already unified into a standard format, and you don't need to do that yourself anymore (as data.raw in some cases has many different viable definition styles). Via command line you can auto-load a prepared savegame (or two if you need normal+expensive), and if you're so inclined you could make the game crash itself once it's done.

[sonaxaton]

A custom mod is much better because it allows you to dump exactly the things you want, as everyone as different needs. You can essentially output a pre-formatted dump that is trivial to import into your external application. Additionally if you dump the data in control-stage you get a) access to game.write_file so you don't have to parse the log, and more importantly b) the data from i.e. game.recipe_prototypes is already unified into a standard format, and you don't need to do that yourself anymore (as data.raw in some cases has many different viable definition styles). Via command line you can auto-load a prepared savegame (or two if you need normal+expensive), and if you're so inclined you could make the game crash itself once it's done.

Thanks for the info! I'll try out some of these (TIL about game.write_file). I haven't considered using the in-game API to access data, because I figured loading up an actual save would take too long, but I could maybe do it with a degenerate save file. The in-game API looks to have more consistent ways of accessing data, as you say.

[eradicator]

because I figured loading up an actual save would take too long

I have no clue about what you consider "too long" etc, but limiting the height/width of the map makes loading significantly faster. I use a 100x100 map when i need to do a quick test of a change in my mods. Also loading a savegame is faster than starting a new map, thus loading a saved size-limited map would be double speedup.

[mrvn]

It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.

Simply use the headless server. It doesn't load any sprites. And you can set it to profile the game for 1 tick so it would automatically stop as well. Or add an error in the lua script after the dump so it aborts there.

[Darinth]

It's doable with a mod, but I wouldn't say it's ideal. The game would still be wasting time loading all the sprite sheets which aren't needed since the game can just exit right after startup and never needs to render any sprites at all.

If the devs don't implement something like this, I'll probably end up doing it with a mod, I'm just hopeful for some first-class support for this kind of use-case.

Simply use the headless server. It doesn't load any sprites. And you can set it to profile the game for 1 tick so it would automatically stop as well. Or add an error in the lua script after the dump so it aborts there.

That really sounds like the way to do this. Load game with mods, loading a save file profiling for 1 tick and then stopping. No sprite sheet load time, just load mods, dump data, run the game for a tick, and shut down automatically. Easily repeated using a batch file.

[TubbyBoy419]

name your mod `~~~` and give it a data-final-fixes.lua, it'll see the finished data, it can be just `print(serpent.block(data))`, I just tried it.

Atlas caches load _fast_ on a hot restart.

Ok, color me stupid, installed the mod, and setup steam to run from a command window. The mod loads, but it doesn’t seem to put anything out. At least anything that looks like a data.raw file. Do I need to redirect the print somehow?

Thanks in advance!

[quyxkh]

name your mod `~~~` and give it a data-final-fixes.lua, it'll see the finished data, it can be just `print(serpent.block(data))`, I just tried it.

Atlas caches load _fast_ on a hot restart.

Ok, color me stupid, installed the mod, and setup steam to run from a command window. The mod loads, but it doesn’t seem to put anything out. At least anything that looks like a data.raw file. Do I need to redirect the print somehow?

Thanks in advance!

not sure when it happened, but Factorio doesn't like the name of that mod any more. change the name to `zdumper` ... there's one on the mod portal for this, Bilka's "data raw serpent".

[mrvn]

and use log() instead of print() and the data will end up in the factorio-current.log.

[moon69]

Implemented Factorio v1.1.77:

Added a command line flag (dump-data) to dump data-raw to the script output folder as json.
Added a command line flag (dump-icon-sprites) to dump prototype icons to the script output folder.
Added a command line flag (dump-prototype-locale) to dump prototype locale to the script output folder.

(Necro since this thread is high in google results for factorio dump data).

[Koub]

Implemented Factorio v1.1.77:

Thanks a lot for spotting this
[Koub] Moving this to implemented.

[mrvn]

Thanks for implementing this.