[Technical] Save Format Questions

[N3X15]

Right off the bat, I know the devs are likely under NDA and won't be able to answer this, so this is more pointed at the community.

I've managed to figure out the general structure of the level.dat file format (and, by extension, level-init.dat). I enjoy poking around at stuff like this, and am currently working on a tool to manage mods, as well as get various pieces of information about saves, like game version and mods used. Eventually, I'd like to build it up to a save editor, but right now it's more for screwing around.

So, here's my dumb question, for those of you who are more familiar with reverse-engineering binary files: What is the big blob of data right after the ore list? Here's the overall structure I've managed to put together, for context:

Code: Select all

SaveFormat_00_15 extends BaseSave {
0:
  // Version info. 0.15.9-0 would be:
  uint16 major; // 0
  uint16 minor; // 15
  uint16 patch; // 9
  uint16 build; // 0
  // 63 00 00 00 00 00 in singleplayer
  // 00 00 00 00 00 00 in MP.  Flags?
  byte[6] unknown_1;
  bigstring gamemode; // "freeplay", usually.
  bigstring base; // Not sure why this is needed, always "base"
  biglist<Mod_00_15> mods;
  biglist<littlestring> players;
  // 29 A8 81 02 01 02 on MP (0.15.9)
  // F8 26 08 00 01 04 on SP (0.15.0) Settings/flags?
  byte[6] unknown_2;
  biglist<OreDef_00_15> ores;
  // And what follows is a big block of gibberish. 259-260ish bytes long, variable size, no discernible size header.
  // It has various repeating patterns, like 47 E1 7A and 80 84 1E 00.
  // Then there's a few zero bytes (at least 4), and then some bigstrings that look like entity names.
}

OreDef_00_15 extends object {
  bigstring name;
  byte[3] unknown_1; // NOT RGB color.  While coal is 00 00 00, oil is 02 02 03, and enemy-base is 01 01 01. Best I can think of is perlin noise adjustments.
}

Mod_00_15 extends BaseMod {
  // 08 41 69 72 63 72 61 66 74 01 03 00 26 08 89 B7
  // Aircraft v1.3.0, CRC 0x260889B7
  littlestring name;
  uint8 major;
  uint8 minor;
  uint8 patch;
  uint32 crc;
}

What I'd like to do, at the very least, is figure out the size of the object so I can skip it in the reader and get to the stuff that follows it. Anything helps.

[DaveMcW]

What programming language are you writing the save editor in?

If you are undecided, I recommend Lua 5.2 inside the factorio engine. It has a working save reader built in!

[quinor]

Dave, you are totally no fun. N1X1S, if you ask nicely (I recommend irc) devs might tell you a thing or two. They are very fun people and generally open and not under any strong NDA as far as I know

[N3X15]

Dave, you are totally no fun. N1X1S, if you ask nicely (I recommend irc) devs might tell you a thing or two. They are very fun people and generally open and not under any strong NDA as far as I know :)

Already discussed it in IRC, and the conclusion was that the devs don't want to discuss save file format.

[SQLek]

Maybe "known content" attack? There is a mod that generates all water world https://mods.factorio.com/mods/FuzzCat927/waterworld . I can't find 0.15 version of it, but i think it exists (i saw it once inside a game mod list). Maybe scenario editor and prepare some maps and analyze it afterwards?

[N3X15]

Maybe "known content" attack? There is a mod that generates all water world https://mods.factorio.com/mods/FuzzCat927/waterworld . I can't find 0.15 version of it, but i think it exists (i saw it once inside a game mod list). Maybe scenario editor and prepare some maps and analyze it afterwards?

Well, to me, it seems too small to be a chunk, and its position in the header doesn't lend it to being any other sort of positional level data. I'd expect it to be included with all the other map data further down in the file. This seems to be metadata of some sort.

[Zeblote]

Why not try to decompile the exe and see exactly how the file is written? It comes with a pdb file with all the symbols

[Daid]

Already discussed it in IRC, and the conclusion was that the devs don't want to discuss save file format.

As I understood from other topics on this subject, the file format changes with each update. So they don't want people to mess with the saves if they have a perfectly scripting support from within the game itself.

[Rseding91]

We don't have a fixed-size format - everything is serialized. So you have to know what a piece of data means before you can say how many bytes you should read.

For strings (as you've already found) that works out as 4 bytes for size then size-bytes of the actual string. For everything else you'd have to have the full source code as we change it almost every game version when we do bug fixes/additions/removals.

It's not a matter of making it "hard" to decompile - it's just efficient to do it this way and the nature of it means it changes frequently also meaning there's no way you can describe the format without the current source code.

[N3X15]

We don't have a fixed-size format - everything is serialized. So you have to know what a piece of data means before you can say how many bytes you should read.

For strings (as you've already found) that works out as 4 bytes for size then size-bytes of the actual string. For everything else you'd have to have the full source code as we change it almost every game version when we do bug fixes/additions/removals.

It's not a matter of making it "hard" to decompile - it's just efficient to do it this way and the nature of it means it changes frequently also meaning there's no way you can describe the format without the current source code.

I understand your position, and I fully expected this to be the case when I waded into this. I also understand that it takes time away from development when you have to come in here and explain something extremely complicated in a way that someone who hasn't worked with the systems would understand. This is partly why I'm trying not to involve you guys, since you have a lot on your plate and I know from experience how much time documentation takes away from development.

As I understood from other topics on this subject, the file format changes with each update. So they don't want people to mess with the saves if they have a perfectly scripting support from within the game itself.

Yeah, but what I'm trying to do at the moment is extract information from the saves from outside of the game without involving firing up the game and the Lua VM. It's handy, for instance, to know a particular save's game version and associated mods for use in mod managers, which is one project I'm working on. For instance, I have a save from 0.12 that my scripts can pull modding information from so I can load it up and play it without opening Factorio itself and manually writing download all the mods I need to find. I could probably inject a DLL into Factorio and grab the information that way, but that's even more difficult to keep in line with updates and doesn't work too well on non-Windows platforms for obvious reasons, not to mention the antivirus warnings and possible issues with Steam. Plus, there's currently some stuff that's not implemented in Factorio, such as telling it to use newer ores when generating veins in new chunks, or tweaking generation parameters after-the-fact. This isn't implemented because it could cause inconsistent behaviour (like chunks not lining up), so it's not really something I'd expect the devs to implement, which creates a niche for save editors.

[Daid]

Or, you could have a script fire up the headless version, with the save file, and use the RPC mechanism that is in place already to query the game, and then shut down the server again.

[Rseding91]

... such as telling it to use newer ores when generating veins in new chunks, or tweaking generation parameters after-the-fact. This isn't implemented because it could cause inconsistent behaviour (like chunks not lining up), ...

That's actually on our to-do for 0.16. It's something I've wanted to add support for for a while now but other things were more important at the time.

[sparr]

it's just efficient to do it this way and the nature of it means it changes frequently also meaning there's no way you can describe the format without the current source code.

I've played games where you had the option of two save formats, the first being effectively .bin.zip and the other being .json/.xml/.readable, all coming from the same source just passed through a different serializer.

[Rseding91]

it's just efficient to do it this way and the nature of it means it changes frequently also meaning there's no way you can describe the format without the current source code.

I've played games where you had the option of two save formats, the first being effectively .bin.zip and the other being .json/.xml/.readable, all coming from the same source just passed through a different serializer.

We've talked about such a thing but it drastically slows the game saving process to support that + bloats all of the save methods by requiring every single value you write out have an associated name.

[N3X15]

Or, you could have a script fire up the headless version, with the save file, and use the RPC mechanism that is in place already to query the game, and then shut down the server again.

Yes, and I already have a working RCON connection in another project of mine. However, Lua doesn't have everything I'd need access to. A lot of the game code is in C++, and in order to make something like that accessible to Lua, you'd have to explicitly write a binding in the C++ code (or SWIG, if they chose to use that), which is time-consuming and somewhat painful. So, some things aren't available from RCON or the console.

it's just efficient to do it this way and the nature of it means it changes frequently also meaning there's no way you can describe the format without the current source code.

I've played games where you had the option of two save formats, the first being effectively .bin.zip and the other being .json/.xml/.readable, all coming from the same source just passed through a different serializer.

As Rseding91 mentions, that can cause performance issues. When your software knows exactly what it expects when it reads a file, you don't have to add extra hints to the reader when writing it in the first place and therefore can avoid overhead and bloat. Then again, if the blob I mentioned above is indeed of a static size, they must have lots of save readers/migrators floating around in their code in order to support backwards compatibility.

[nobeerleft]

That chunk of data from map-settings.json in Float64/Uint32/Uint8 all separated by 01 bytes values

[LordMZTE]

i know i'm a bit late tho this but this was a really interesting read!
did your save editor project go anywhere?