level.dat format

Post all other topics which do not belong to any other category.
Post Reply
barkface
Burner Inserter
Burner Inserter
Posts: 9
Joined: Thu Dec 25, 2014 2:02 pm
Contact:

level.dat format

Post by barkface »

Hello!

In my quest to create a mod-managing utility, I'm trying to read information about the mods used from a savegame.

Looking at level.dat, the header seems to have what I want.
I can see the mod names stored in a single-width character set (One of the forms of ISO-8859-1?).
Thanks to Rseding, I know the 4 bytes in front of the mod names are an uint32 describing the length of the mod name in byte.

So the section with the mod names looks like
4 bytes uint length_of_name
length_of_name string modname
6 byes unknown
and repeating.

I can't make heads or tails of the header before the mod names, though.

Can a dev shed some light on the level.dat format?
What's in that header, and how do I find the length of the entire thing? Is my assumption about the mod names section correct?
Thanks~

dee-
Filter Inserter
Filter Inserter
Posts: 413
Joined: Mon Jan 19, 2015 9:21 am
Contact:

Re: level.dat format

Post by dee- »

AFAICT the format is still WIP.
For example there has been a major data redesign (float -> fixed int) some weeks ago.

Somewhere in the header must also be the game version that created the save as older game versions refuse to open newer savegames and can tell which game version created the save.

barkface
Burner Inserter
Burner Inserter
Posts: 9
Joined: Thu Dec 25, 2014 2:02 pm
Contact:

Re: level.dat format

Post by barkface »

The game version could actually be the very first thing. Assuming uint16, the first 12 bytes spell out 0 11 15 for my recent saves, and 0 11 10 for an old one I had lying around (could easily have been 0.11.10).

dee-
Filter Inserter
Filter Inserter
Posts: 413
Joined: Mon Jan 19, 2015 9:21 am
Contact:

Re: level.dat format

Post by dee- »

barkface wrote:The game version could actually be the very first thing. Assuming uint16, the first 12 bytes spell out 0 11 15 for my recent saves, and 0 11 10 for an old one I had lying around (could easily have been 0.11.10).
one more down Image

kovarex
Factorio Staff
Factorio Staff
Posts: 8052
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: level.dat format

Post by kovarex »

The 6 unknown bytes is version of the mod (2 bytes per number).

Code: Select all

void ScenarioExecutionContext::save(MapSerialiser& output) const
{
  output << this->location;
  output << this->difficulty;
  output << this->finished;
  output << this->playerWon;
  output << this->nextLevel;
  output << this->savingReplay;
  output << ApplicationVersion::currentVersion;
  output << this->allowedCommands;
  output << this->stats;
  output << global->modManager->getActiveModThatChangeTheGameStateIDList();
}

barkface
Burner Inserter
Burner Inserter
Posts: 9
Joined: Thu Dec 25, 2014 2:02 pm
Contact:

Re: level.dat format

Post by barkface »

Squeee, Kovarex-senpai noticed me~

Thanks a bunch, with some more pointers from Rseding, I made it up to and including allowedCommands. I'm pretty sure I have those values right, and my offset is correct after I've read those (at least the version number always comes out matching the one at the very beginning of the file)

The stats are a mystery to me, though. According to Rseding, it's a sequence of three variable-length complex types, but I cannot make heads or tails of how to determine their length. Can you help out once more?

User avatar
mickael9
Fast Inserter
Fast Inserter
Posts: 112
Joined: Mon Mar 14, 2016 4:04 am
Contact:

Re: level.dat format

Post by mickael9 »

After some reverse engineering I wrote this parser in Python, it should work with all saves that can be loaded by Factorio 0.15.

https://gist.github.com/mickael9/5dbdb9 ... f0cc1d5a9f

Sample output:

Code: Select all

/home/mickael/.factorio/saves/test.zip:

base_mod: base
campaign: ''
current_version: 0.15.11
difficulty: 1
finished: false
mods:
  base: 0.15.11
  creative-mode: 0.3.3
  cut-and-paste: 0.1.2
name: freeplay
next_level: ''
player_won: false
stats: {}
version: 0.15.11.2
This only parses mods and everything that comes before in the save file.

User avatar
mickael9
Fast Inserter
Fast Inserter
Posts: 112
Joined: Mon Mar 14, 2016 4:04 am
Contact:

Re: level.dat format

Post by mickael9 »

Updated for 0.16

Code: Select all

from zipfile import ZipFile
from struct import Struct


class Deserializer:
    u16 = Struct('<H')
    u32 = Struct('<I')

    def __init__(self, stream):
        self.stream = stream
        self.version = tuple(self.read_u16() for i in range(4))

    def read(self, n):
        return self.stream.read(n)

    def read_fmt(self, fmt):
        return fmt.unpack(self.read(fmt.size))[0]

    def read_u8(self):
        return self.read(1)[0]

    def read_bool(self):
        return bool(self.read_u8())

    def read_u16(self):
        return self.read_fmt(self.u16)

    def read_u32(self):
        return self.read_fmt(self.u32)

    def read_str(self, dtype=None):
        if self.version >= (0, 16, 0, 0):
            length = self.read_optim(dtype or self.u32)
        else:
            length = self.read_fmt(dtype or self.u32)

        return self.read(length).decode('utf-8')

    def read_optim(self, dtype):
        if self.version >= (0, 14, 14, 0):
            byte = self.read_u8()
            if byte != 0xFF:
                return byte
        return self.read_fmt(dtype)

    def read_optim_u16(self):
        return self.read_optim(self.u16)

    def read_optim_u32(self):
        return self.read_optim(self.u32)

    def read_optim_str(self):
        length = self.read_optim_u32()
        return self.read(length).decode('utf-8')

    def read_optim_tuple(self, dtype, num):
        return tuple(self.read_optim(dtype) for i in range(num))


class SaveFile:
    def __init__(self, filename):
        zf = ZipFile(filename, 'r')
        datfile = None

        for f in zf.namelist():
            if f.endswith('/level.dat'):
                datfile = f
                break

        if not datfile:
            raise IOError("level.dat not found in save file")

        ds = Deserializer(zf.open(datfile))
        self.version = self.version_str(ds.version)

        self.campaign = ds.read_str()
        self.name = ds.read_str()
        self.base_mod = ds.read_str()

        # 0: Normal, 1: Old School, 2: Hardcore
        self.difficulty = ds.read_u8()

        self.finished = ds.read_bool()
        self.player_won = ds.read_bool()
        self.next_level = ds.read_str()  # usually empty

        if ds.version >= (0, 12, 0, 0):
            self.can_continue = ds.read_bool()
            self.finished_but_continuing = ds.read_bool()

        self.saving_replay = ds.read_bool()

        if ds.version >= (0, 16, 0, 0):
            self.allow_non_admin_debug_options = ds.read_bool()

        self.loaded_from = self.version_str(ds.read_optim_tuple(ds.u16, 3))
        self.loaded_from_build = ds.read_u16()

        self.allowed_commands = ds.read_u8()
        if ds.version <= (0, 13, 0, 87):
            if not self.allowed_commands:
                self.allowed_commands = 2
            else:
                self.allowed_commands = 1

        self.stats = {}
        if ds.version <= (0, 13, 0, 42):
            num_stats = ds.read_u32()
            for i in range(num_stats):
                force_id = ds.read_u8()
                self.stats[force_id] = []
                for j in range(3):
                    st = {}
                    length = ds.read_u32()
                    for k in range(length):
                        k = ds.read_u16()
                        v = ds.read_u32()
                        st[k] = v
                    self.stats[force_id].append(st)


        self.mods = {}
        if ds.version >= (0, 16, 0, 0):
            num_mods = ds.read_optim_u32()
        else:
            num_mods = ds.read_u32()

        for i in range(num_mods):
            name = ds.read_optim_str()
            version = ds.read_optim_tuple(ds.u16, 3)
            if ds.version > (0, 15, 0, 91):
                ds.read_u32()  # CRC
            self.mods[name] = self.version_str(version)

    @staticmethod
    def version_str(ver):
        return '.'.join(str(x) for x in ver)


if __name__ == '__main__':
    import sys
    try:
        from yaml import safe_dump
    except ImportError:
        print('Install PyYAML for pretty printing')

        def safe_dump(s, **kw):
            return repr(s)

    for name in sys.argv[1:]:
        sf = SaveFile(name)
        print('%s:' % name)
        print()
        print(safe_dump(sf.__dict__, default_flow_style=False))
        print('---')

Siorai
Manual Inserter
Manual Inserter
Posts: 4
Joined: Thu Oct 27, 2016 5:11 pm
Contact:

Re: level.dat format

Post by Siorai »

Oh -awesome- ! Great job on this btw. ( Verified working on Python3.7 in case you were wondering ). By chance gonna update it for .17 ? ( Once it's out that is... )
mickael9 wrote:
Sun Apr 15, 2018 1:20 pm
Updated for 0.16

Code: Select all

from zipfile import ZipFile
from struct import Struct


class Deserializer:
    u16 = Struct('<H')
    u32 = Struct('<I')

    def __init__(self, stream):
        self.stream = stream
        self.version = tuple(self.read_u16() for i in range(4))

    def read(self, n):
        return self.stream.read(n)

    def read_fmt(self, fmt):
        return fmt.unpack(self.read(fmt.size))[0]

    def read_u8(self):
        return self.read(1)[0]

    def read_bool(self):
        return bool(self.read_u8())

    def read_u16(self):
        return self.read_fmt(self.u16)

    def read_u32(self):
        return self.read_fmt(self.u32)

    def read_str(self, dtype=None):
        if self.version >= (0, 16, 0, 0):
            length = self.read_optim(dtype or self.u32)
        else:
            length = self.read_fmt(dtype or self.u32)

        return self.read(length).decode('utf-8')

    def read_optim(self, dtype):
        if self.version >= (0, 14, 14, 0):
            byte = self.read_u8()
            if byte != 0xFF:
                return byte
        return self.read_fmt(dtype)

    def read_optim_u16(self):
        return self.read_optim(self.u16)

    def read_optim_u32(self):
        return self.read_optim(self.u32)

    def read_optim_str(self):
        length = self.read_optim_u32()
        return self.read(length).decode('utf-8')

    def read_optim_tuple(self, dtype, num):
        return tuple(self.read_optim(dtype) for i in range(num))


class SaveFile:
    def __init__(self, filename):
        zf = ZipFile(filename, 'r')
        datfile = None

        for f in zf.namelist():
            if f.endswith('/level.dat'):
                datfile = f
                break

        if not datfile:
            raise IOError("level.dat not found in save file")

        ds = Deserializer(zf.open(datfile))
        self.version = self.version_str(ds.version)

        self.campaign = ds.read_str()
        self.name = ds.read_str()
        self.base_mod = ds.read_str()

        # 0: Normal, 1: Old School, 2: Hardcore
        self.difficulty = ds.read_u8()

        self.finished = ds.read_bool()
        self.player_won = ds.read_bool()
        self.next_level = ds.read_str()  # usually empty

        if ds.version >= (0, 12, 0, 0):
            self.can_continue = ds.read_bool()
            self.finished_but_continuing = ds.read_bool()

        self.saving_replay = ds.read_bool()

        if ds.version >= (0, 16, 0, 0):
            self.allow_non_admin_debug_options = ds.read_bool()

        self.loaded_from = self.version_str(ds.read_optim_tuple(ds.u16, 3))
        self.loaded_from_build = ds.read_u16()

        self.allowed_commands = ds.read_u8()
        if ds.version <= (0, 13, 0, 87):
            if not self.allowed_commands:
                self.allowed_commands = 2
            else:
                self.allowed_commands = 1

        self.stats = {}
        if ds.version <= (0, 13, 0, 42):
            num_stats = ds.read_u32()
            for i in range(num_stats):
                force_id = ds.read_u8()
                self.stats[force_id] = []
                for j in range(3):
                    st = {}
                    length = ds.read_u32()
                    for k in range(length):
                        k = ds.read_u16()
                        v = ds.read_u32()
                        st[k] = v
                    self.stats[force_id].append(st)


        self.mods = {}
        if ds.version >= (0, 16, 0, 0):
            num_mods = ds.read_optim_u32()
        else:
            num_mods = ds.read_u32()

        for i in range(num_mods):
            name = ds.read_optim_str()
            version = ds.read_optim_tuple(ds.u16, 3)
            if ds.version > (0, 15, 0, 91):
                ds.read_u32()  # CRC
            self.mods[name] = self.version_str(version)

    @staticmethod
    def version_str(ver):
        return '.'.join(str(x) for x in ver)


if __name__ == '__main__':
    import sys
    try:
        from yaml import safe_dump
    except ImportError:
        print('Install PyYAML for pretty printing')

        def safe_dump(s, **kw):
            return repr(s)

    for name in sys.argv[1:]:
        sf = SaveFile(name)
        print('%s:' % name)
        print()
        print(safe_dump(sf.__dict__, default_flow_style=False))
        print('---')

Post Reply

Return to “General discussion”