Format of level.dat and blueprint.dat?

[sillyfly]

Is there any info on the format of the level.dat and blueprint.dat (scenario) files?
I am mostly interested in the format of the chunks and resource layers - as the map editor is very lacking at the moment, and I want to be able to do more advanced stuff with much larger maps than it's currently possible.

So - anybody god any hints? Anyone tried to RE some parts of the files and wants to share their progress?
Maybe the developers would like to chime in and help us out?

Thanks!

(Side note: Not sure if this is the correct sub-forum for this, so please move it elsewhere if there is a better one)

[Oxyd]

They are serialised internal data structures. The format is described by the source code – that is the only description of the format that I know of.

Trying to RE the format by hand sounds very painful. So, good luck & have fun.

[sillyfly]

I'll try to

Can you just tell me this - are there any checksums in these files, and if so what checksum format you are using? Those specifically can be a real pain to RE.
Also, do you have any objections to me trying to RE and publishing my results? I think it would be nice to have community-made map editors (Minecraft sure has a lot of these!), but I wouldn't want to do this if you guys are against it...

[Zeblote]

You've got factorio.exe and a matching factorio.pdb, reverse engineering the file format wouldn't be all that hard.

[tamat]

I was also thinking about making a web visualizer of the maps, so you can inspect from the browser the state of a map, zoomin, add comments, etc, something similar to overviewer for minecraft. The only info I need are the chunks and the codification of the cells. I understand that being a serialization of the memory is not going to be easy, but if I can get just the memory region with the chunks and the size of every chunk from the level.dat I could make a nice render.

Thanks!

[asheiduk]

I've decoded the blueprint-storage.dat format -- you can see the code here: GitHub: Factorio Blueprint Decoder. The tool transforms the file into a JSON structure of a blueprint-book. The JSON can be encoded into a standard import/export string.

The file format for level.dat seems to use similar datastructures but with much more details. So my code might be a base for digging deeper into level.dat

[asdff45]

The level.dat header is really good reverse engineered, i'm currently working on getting the new fields in 1.1. (no new fields in 1.1)
The project where we read the header: https://github.com/OpenFactorioServerMa ... ave.go#L79

Also a python-script as available, where the old code, that was used is posted: viewtopic.php?f=5&t=8568&p=277892&hilit ... on#p277892

For reading/writing the mod-settings.dat (not tested for 0.18/1.0/1.1): https://github.com/knoxfighter/factorio ... dat.go#L24. I will work on that, when the factorio-server-manager released v1.0.

[asheiduk]

The level.dat header is really good reverse engineered, [...] The project where we read the header: https://github.com/OpenFactorioServerMa ... ave.go#L79

Just to make sure the difference is clear: The linked project (Go) and the linked Python scripts (outdated) only parse a minuscule part of the save file -- the header. On the hand: The "Factorio Blueprint Decoder" can decode the complete blueprints file down to the details of each entity. So -- for example -- the circuit connections and configured circuit conditions of an inserter are parsed.

I suspect that this level of detail can help to better understand the format of level.dat beyond the header.

[morsk]

I've decoded the blueprint-storage.dat format -- you can see the code here: GitHub: Factorio Blueprint Decoder. The tool transforms the file into a JSON structure of a blueprint-book. The JSON can be encoded into a standard import/export string.

I just heard about this from reddit! I spent a lot of time doing almost the same thing, but didn't finish inserter, train-stop, upgrade planner, or the snap to grid data. (Or any infinity types, or tags.) I especially wasted time trying to generate 100% identical json and base64 to Factorio, which is a pain because the floats are unnecessarily long, and the 64-bit ones have errors. And I hadn't used Python in years so it's quite horrible and all needs to be restructured. https://github.com/morsk/factorio-blueprint-dat

I will probably still finish. Even if it's something of a mess, I made an automated way to find the binary size of every entity type, and this will make detecting changes after patches easier. (A lua mod makes a special blueprint, with 2 of every entity, and 2 more if it can be wired. The python can find the sizes when it knows they're always two in a row.) I'm also trying to make an automated way to mass-update .dat files when the game updates. I will probably finish that before anything else.

[asheiduk]

I just heard about this from reddit!

Interesting!. Do you have a link?

I spent a lot of time doing almost the same thing, but didn't finish inserter, train-stop, upgrade planner, or the snap to grid data. (Or any infinity types, or tags.) I especially wasted time trying to generate 100% identical json and base64 to Factorio, which is a pain because the floats are unnecessarily long, and the 64-bit ones have errors.

My JSON is identical to the original JSON after sorting the order of the keys within the JSON objects. Of course this means that the original bas64 is not bytewise identical. But this level is practically enough and it helped me quite much do eliminate the differences one by one.

Side note: I didn't have much problems with floats and doubles. Python seems to produce similar output Factorio for these types. You may take a look at the code.

And I hadn't used Python in years so it's quite horrible and all needs to be restructured. https://github.com/morsk/factorio-blueprint-dat

I will probably still finish. Even if it's something of a mess, I made an automated way to find the binary size of every entity type, and this will make detecting changes after patches easier. (A lua mod makes a special blueprint, with 2 of every entity, and 2 more if it can be wired. The python can find the sizes when it knows they're always two in a row.)

But your link does not show a repo in GitHub Is it a private repo?

The Lua mod is an interesting idea. My problem was not about finding the length of entities though (using a similar technique by hand) but enumerating all the possible variations and configurations. These can change the length of each entity. Wiring is just one example. But it would be very helpful to query things like "can be wired", "has direction", ... from Lua. But I'm totally blank on that level. So looking at your mod for this I might get some ideas how to proceed.