The size of every terrain chunk

[tamat]

Im trying to implement a web-viewer of the maps of Factorio, a web tool that allows to scroll and see your map in detail from your browser.
To do that I need to extract the info from the level.dat but as far as I know there is no public info about the file format due to the way the data is stored from the ingame memory.

I plan do reverse engineer it but it would help to have the chunk dimensions (how many cells in X and Y have every block of the terrain, I asume 32 or 64 ) and also how many bytes does every cell have to store the info. This way I could map them easily.

Thanks.

[Phillip_Lynx]

Have you had a glimpse in this Mod?

[sillyfly]

This: https://forums.factorio.com/forum/vie ... =14&t=7243 may also help.

As for your original question - the chunks are 32x32 tiles, but sadly this does not map directly into data in the level.dat file. As you know from this thread: https://forums.factorio.com/forum/vie ... 69&t=16844 I have started to RE the blueprint.dat file, and it seems there are a few things about the way the maps are serialized that will make it not too straight-forward - for one thing, Factorio tiles (when serialized) may be 1x1, 2x2 or even 4x4 tiles in size, so chunks are not consistent in size inside the file. Furthermore, those multi-tile tiles may span few chunks (up to four), which means you must read and parse all chunks to be able to read them correctly.
Finally, it seems that entities (players, biters, machines etc.) are stored in-line, which makes the chunk data even more irregular.

[tamat]

Thanks for the link, that seems like a very nice project, but it implies that you have to execute the MOD from within the game everytime you want to update the map and you only have the images.
My idea is that it loads the info directly from the saved game so you have always the last version, and in the long run my idea is to extract other info from the savegame (like power consumtion graphs or stocks).
Also I would like to play with the visualization (like showing layers with different info). For all those think I cannot use a MOD.

Thanks again for the link.

[tamat]

This: https://forums.factorio.com/forum/vie ... =14&t=7243 may also help.

As for your original question - the chunks are 32x32 tiles, but sadly this does not map directly into data in the level.dat file. As you know from this thread: https://forums.factorio.com/forum/vie ... 69&t=16844 I have started to RE the blueprint.dat file, and it seems there are a few things about the way the maps are serialized that will make it not too straight-forward - for one thing, Factorio tiles (when serialized) may be 1x1, 2x2 or even 4x4 tiles in size, so chunks are not consistent in size inside the file. Furthermore, those multi-tile tiles may span few chunks (up to four), which means you must read and parse all chunks to be able to read them correctly.
Finally, it seems that entities (players, biters, machines etc.) are stored in-line, which makes the chunk data even more irregular.

Thanks Sillyfly, I was expecting something like this. Are the bytes per tile constant? what I mean is if every chunk has the same size or the amount of bytes per chunk depends on the type of tiles? In case they have different size do they encode the amount of bytes?, I only need the spatial info (what is where), so if there is some extra bytes that could change in length I dont care as far as the level geography is always 32x32xN

[sillyfly]

The chunks are not the same size in bytes, and are almost never 32x32xN. As far as I could tell there is no explicit field specifying the length (in bytes) of a specific chunk, so each chunk has to be processed in full.

[tamat]

what a difficult task then. Then the only option is to export the data from a MOD to a more easy-to-work-with file format, like a JSON dump or XML. What a pity.
Anyway, thanks a lot for your info, you saved me a lot of time. If you manage to parse just the basic structure then let me know. Good luck with your colosal task!

[Zeblote]

The chunks are not the same size in bytes, and are almost never 32x32xN. As far as I could tell there is no explicit field specifying the length (in bytes) of a specific chunk, so each chunk has to be processed in full.

I assume for every chunk it writes down all entities, which have different length. An assembler would have extra bytes about the receipt and modules, while a conveyor belt doesn't, etc...

If that's the case then there might be no value indicating the length at all, because virtual save()/load() methods on entity classes are supposed to handle the data.

Just a though, could be wrong. You can always use something like ida/hex-rays to find out what's written in what order The pdb file makes this really easy as it can automatically reconstruct all names and structs/types with it.

[sillyfly]

...

I assume for every chunk it writes down all entities, which have different length. An assembler would have extra bytes about the receipt and modules, while a conveyor belt doesn't, etc...

If that's the case then there might be no value indicating the length at all, because virtual save()/load() methods on entity classes are supposed to handle the data.

Just a though, could be wrong. You can always use something like ida/hex-rays to find out what's written in what order The pdb file makes this really easy as it can automatically reconstruct all names and structs/types with it.

Yep, that's what I assume is happening.
Anyway, I tried disassembling the Linux binary (which is also compiled with debug symbols), but it took ages (the file is ~55MB!). Do you think processing the Windows executable/pdb would be faster?
Anyway, if you want to try and help I'd be happy to join forces!

[Zeblote]

Anyway, I tried disassembling the Linux binary (which is also compiled with debug symbols), but it took ages (the file is ~55MB!). Do you think processing the Windows executable/pdb would be faster?

It takes a few minutes to disassemble the exe and another few minutes to load the pdb info, then it's very fast. Also lets you use the hex-rays decompiler to get almost-useful code.

However, I don't think the devs like us doing things like this?

[sillyfly]

However, I don't think the devs like us doing things like this?

Well, they are reading this forum. If asked not to do it I won't pursue. I did ask if they mind in the first thread, which at least on of the devs replied in...

[Zeblote]

If they don't mind people reverse engineering it, it would definitely be easier for one of them to just copy-paste whatever documentation they have in the source

[sillyfly]

Not exactly. I think there is a difference between saying - "We don't mind if people take time and effort to do something probably not everyone has the tools and knowledge to do, to get an understanding of the inside-workings of our product" versus "Here, everyone can have a clear look at the way we do it - first hand info!".

But that's just how I feel. Of course, the first thing I asked was if the developers themselves wanted to fill us in on the details of how saved games and scenarios are serialized

[Oxyd]

If they don't mind people reverse engineering it, it would definitely be easier for one of them to just copy-paste whatever documentation they have in the source

We have any such documentation in the source?

[Zeblote]

If they don't mind people reverse engineering it, it would definitely be easier for one of them to just copy-paste whatever documentation they have in the source

We have any such documentation in the source?

You didn't document your file format?

[Oxyd]

If they don't mind people reverse engineering it, it would definitely be easier for one of them to just copy-paste whatever documentation they have in the source

We have any such documentation in the source?

You didn't document your file format?

Why would we do that? That would be completely insane. Also if we did, it would be out-of-date by now, as it literally changes with almost every minor version, and I bet no one would be arsed to keep it up-to-date.

[tamat]

If they don't mind people reverse engineering it, it would definitely be easier for one of them to just copy-paste whatever documentation they have in the source

We have any such documentation in the source?

You didn't document your file format?

Why would we do that? That would be completely insane. Also if we did, it would be out-of-date by now, as it literally changes with almost every minor version, and I bet no one would be arsed to keep it up-to-date.

Then the only solution I see for people to do stuff with the savegames would be if there was a tool compiled by the Factorio Staff that takes the savegame and convert it to a easy to read sintax, verbose and all, and also the opposite process.
This way people would be able to do editors and visualizers, and the tool could be added to your build system so you dont have to mantain it. But I understand that it takes some effort to do that in the first place.

[tamat]

I was checking the level.dat file in a hex editor and Im surprised to see that besides some data that seems like headers there are big chunks of the file filled with 00 or the sequence 20 07, maybe is for the terrain info without the tiles.

About opening the PDB, what software did you use? I tryed with VS 2012 but didnt work, I guess I should use 2015.

[Zeblote]

About opening the PDB, what software did you use? I tryed with VS 2012 but didnt work, I guess I should use 2015.

Ida pro with hex-rays plugin

[tamat]

About opening the PDB, what software did you use? I tryed with VS 2012 but didnt work, I guess I should use 2015.

Ida pro with hex-rays plugin

Thanks