level.dat format

[barkface]

Hello!

In my quest to create a mod-managing utility, I'm trying to read information about the mods used from a savegame.

Looking at level.dat, the header seems to have what I want.
I can see the mod names stored in a single-width character set (One of the forms of ISO-8859-1?).
Thanks to Rseding, I know the 4 bytes in front of the mod names are an uint32 describing the length of the mod name in byte.

So the section with the mod names looks like
4 bytes uint length_of_name
length_of_name string modname
6 byes unknown
and repeating.

I can't make heads or tails of the header before the mod names, though.

Can a dev shed some light on the level.dat format?
What's in that header, and how do I find the length of the entire thing? Is my assumption about the mod names section correct?
Thanks~

[dee-]

AFAICT the format is still WIP.
For example there has been a major data redesign (float -> fixed int) some weeks ago.

Somewhere in the header must also be the game version that created the save as older game versions refuse to open newer savegames and can tell which game version created the save.

[barkface]

The game version could actually be the very first thing. Assuming uint16, the first 12 bytes spell out 0 11 15 for my recent saves, and 0 11 10 for an old one I had lying around (could easily have been 0.11.10).

[dee-]

The game version could actually be the very first thing. Assuming uint16, the first 12 bytes spell out 0 11 15 for my recent saves, and 0 11 10 for an old one I had lying around (could easily have been 0.11.10).

one more down

[kovarex]

The 6 unknown bytes is version of the mod (2 bytes per number).

Code: Select all

void ScenarioExecutionContext::save(MapSerialiser& output) const
{
  output << this->location;
  output << this->difficulty;
  output << this->finished;
  output << this->playerWon;
  output << this->nextLevel;
  output << this->savingReplay;
  output << ApplicationVersion::currentVersion;
  output << this->allowedCommands;
  output << this->stats;
  output << global->modManager->getActiveModThatChangeTheGameStateIDList();
}

[barkface]

Squeee, Kovarex-senpai noticed me~

Thanks a bunch, with some more pointers from Rseding, I made it up to and including allowedCommands. I'm pretty sure I have those values right, and my offset is correct after I've read those (at least the version number always comes out matching the one at the very beginning of the file)

The stats are a mystery to me, though. According to Rseding, it's a sequence of three variable-length complex types, but I cannot make heads or tails of how to determine their length. Can you help out once more?

[mickael9]

After some reverse engineering I wrote this parser in Python, it should work with all saves that can be loaded by Factorio 0.15.

https://gist.github.com/mickael9/5dbdb9 ... f0cc1d5a9f

Sample output:

Code: Select all

/home/mickael/.factorio/saves/test.zip:

base_mod: base
campaign: ''
current_version: 0.15.11
difficulty: 1
finished: false
mods:
  base: 0.15.11
  creative-mode: 0.3.3
  cut-and-paste: 0.1.2
name: freeplay
next_level: ''
player_won: false
stats: {}
version: 0.15.11.2

This only parses mods and everything that comes before in the save file.

[mickael9]

Updated for 0.16

Code: Select all

from zipfile import ZipFile
from struct import Struct


class Deserializer:
    u16 = Struct('<H')
    u32 = Struct('<I')

    def __init__(self, stream):
        self.stream = stream
        self.version = tuple(self.read_u16() for i in range(4))

    def read(self, n):
        return self.stream.read(n)

    def read_fmt(self, fmt):
        return fmt.unpack(self.read(fmt.size))[0]

    def read_u8(self):
        return self.read(1)[0]

    def read_bool(self):
        return bool(self.read_u8())

    def read_u16(self):
        return self.read_fmt(self.u16)

    def read_u32(self):
        return self.read_fmt(self.u32)

    def read_str(self, dtype=None):
        if self.version >= (0, 16, 0, 0):
            length = self.read_optim(dtype or self.u32)
        else:
            length = self.read_fmt(dtype or self.u32)

        return self.read(length).decode('utf-8')

    def read_optim(self, dtype):
        if self.version >= (0, 14, 14, 0):
            byte = self.read_u8()
            if byte != 0xFF:
                return byte
        return self.read_fmt(dtype)

    def read_optim_u16(self):
        return self.read_optim(self.u16)

    def read_optim_u32(self):
        return self.read_optim(self.u32)

    def read_optim_str(self):
        length = self.read_optim_u32()
        return self.read(length).decode('utf-8')

    def read_optim_tuple(self, dtype, num):
        return tuple(self.read_optim(dtype) for i in range(num))


class SaveFile:
    def __init__(self, filename):
        zf = ZipFile(filename, 'r')
        datfile = None

        for f in zf.namelist():
            if f.endswith('/level.dat'):
                datfile = f
                break

        if not datfile:
            raise IOError("level.dat not found in save file")

        ds = Deserializer(zf.open(datfile))
        self.version = self.version_str(ds.version)

        self.campaign = ds.read_str()
        self.name = ds.read_str()
        self.base_mod = ds.read_str()

        # 0: Normal, 1: Old School, 2: Hardcore
        self.difficulty = ds.read_u8()

        self.finished = ds.read_bool()
        self.player_won = ds.read_bool()
        self.next_level = ds.read_str()  # usually empty

        if ds.version >= (0, 12, 0, 0):
            self.can_continue = ds.read_bool()
            self.finished_but_continuing = ds.read_bool()

        self.saving_replay = ds.read_bool()

        if ds.version >= (0, 16, 0, 0):
            self.allow_non_admin_debug_options = ds.read_bool()

        self.loaded_from = self.version_str(ds.read_optim_tuple(ds.u16, 3))
        self.loaded_from_build = ds.read_u16()

        self.allowed_commands = ds.read_u8()
        if ds.version <= (0, 13, 0, 87):
            if not self.allowed_commands:
                self.allowed_commands = 2
            else:
                self.allowed_commands = 1

        self.stats = {}
        if ds.version <= (0, 13, 0, 42):
            num_stats = ds.read_u32()
            for i in range(num_stats):
                force_id = ds.read_u8()
                self.stats[force_id] = []
                for j in range(3):
                    st = {}
                    length = ds.read_u32()
                    for k in range(length):
                        k = ds.read_u16()
                        v = ds.read_u32()
                        st[k] = v
                    self.stats[force_id].append(st)


        self.mods = {}
        if ds.version >= (0, 16, 0, 0):
            num_mods = ds.read_optim_u32()
        else:
            num_mods = ds.read_u32()

        for i in range(num_mods):
            name = ds.read_optim_str()
            version = ds.read_optim_tuple(ds.u16, 3)
            if ds.version > (0, 15, 0, 91):
                ds.read_u32()  # CRC
            self.mods[name] = self.version_str(version)

    @staticmethod
    def version_str(ver):
        return '.'.join(str(x) for x in ver)


if __name__ == '__main__':
    import sys
    try:
        from yaml import safe_dump
    except ImportError:
        print('Install PyYAML for pretty printing')

        def safe_dump(s, **kw):
            return repr(s)

    for name in sys.argv[1:]:
        sf = SaveFile(name)
        print('%s:' % name)
        print()
        print(safe_dump(sf.__dict__, default_flow_style=False))
        print('---')

[Siorai]

Oh -awesome- ! Great job on this btw. ( Verified working on Python3.7 in case you were wondering ). By chance gonna update it for .17 ? ( Once it's out that is... )

Updated for 0.16

Code: Select all

from zipfile import ZipFile
from struct import Struct


class Deserializer:
    u16 = Struct('<H')
    u32 = Struct('<I')

    def __init__(self, stream):
        self.stream = stream
        self.version = tuple(self.read_u16() for i in range(4))

    def read(self, n):
        return self.stream.read(n)

    def read_fmt(self, fmt):
        return fmt.unpack(self.read(fmt.size))[0]

    def read_u8(self):
        return self.read(1)[0]

    def read_bool(self):
        return bool(self.read_u8())

    def read_u16(self):
        return self.read_fmt(self.u16)

    def read_u32(self):
        return self.read_fmt(self.u32)

    def read_str(self, dtype=None):
        if self.version >= (0, 16, 0, 0):
            length = self.read_optim(dtype or self.u32)
        else:
            length = self.read_fmt(dtype or self.u32)

        return self.read(length).decode('utf-8')

    def read_optim(self, dtype):
        if self.version >= (0, 14, 14, 0):
            byte = self.read_u8()
            if byte != 0xFF:
                return byte
        return self.read_fmt(dtype)

    def read_optim_u16(self):
        return self.read_optim(self.u16)

    def read_optim_u32(self):
        return self.read_optim(self.u32)

    def read_optim_str(self):
        length = self.read_optim_u32()
        return self.read(length).decode('utf-8')

    def read_optim_tuple(self, dtype, num):
        return tuple(self.read_optim(dtype) for i in range(num))


class SaveFile:
    def __init__(self, filename):
        zf = ZipFile(filename, 'r')
        datfile = None

        for f in zf.namelist():
            if f.endswith('/level.dat'):
                datfile = f
                break

        if not datfile:
            raise IOError("level.dat not found in save file")

        ds = Deserializer(zf.open(datfile))
        self.version = self.version_str(ds.version)

        self.campaign = ds.read_str()
        self.name = ds.read_str()
        self.base_mod = ds.read_str()

        # 0: Normal, 1: Old School, 2: Hardcore
        self.difficulty = ds.read_u8()

        self.finished = ds.read_bool()
        self.player_won = ds.read_bool()
        self.next_level = ds.read_str()  # usually empty

        if ds.version >= (0, 12, 0, 0):
            self.can_continue = ds.read_bool()
            self.finished_but_continuing = ds.read_bool()

        self.saving_replay = ds.read_bool()

        if ds.version >= (0, 16, 0, 0):
            self.allow_non_admin_debug_options = ds.read_bool()

        self.loaded_from = self.version_str(ds.read_optim_tuple(ds.u16, 3))
        self.loaded_from_build = ds.read_u16()

        self.allowed_commands = ds.read_u8()
        if ds.version <= (0, 13, 0, 87):
            if not self.allowed_commands:
                self.allowed_commands = 2
            else:
                self.allowed_commands = 1

        self.stats = {}
        if ds.version <= (0, 13, 0, 42):
            num_stats = ds.read_u32()
            for i in range(num_stats):
                force_id = ds.read_u8()
                self.stats[force_id] = []
                for j in range(3):
                    st = {}
                    length = ds.read_u32()
                    for k in range(length):
                        k = ds.read_u16()
                        v = ds.read_u32()
                        st[k] = v
                    self.stats[force_id].append(st)


        self.mods = {}
        if ds.version >= (0, 16, 0, 0):
            num_mods = ds.read_optim_u32()
        else:
            num_mods = ds.read_u32()

        for i in range(num_mods):
            name = ds.read_optim_str()
            version = ds.read_optim_tuple(ds.u16, 3)
            if ds.version > (0, 15, 0, 91):
                ds.read_u32()  # CRC
            self.mods[name] = self.version_str(version)

    @staticmethod
    def version_str(ver):
        return '.'.join(str(x) for x in ver)


if __name__ == '__main__':
    import sys
    try:
        from yaml import safe_dump
    except ImportError:
        print('Install PyYAML for pretty printing')

        def safe_dump(s, **kw):
            return repr(s)

    for name in sys.argv[1:]:
        sf = SaveFile(name)
        print('%s:' % name)
        print()
        print(safe_dump(sf.__dict__, default_flow_style=False))
        print('---')