Better filesystem storage of Blueprints

Post your ideas and suggestions how to improve the game.

Moderator: ickputzdirwech

Post Reply
knifeymcknifeface
Manual Inserter
Manual Inserter
Posts: 3
Joined: Thu Jul 30, 2020 6:53 pm
Contact:

Better filesystem storage of Blueprints

Post by knifeymcknifeface »

TL;DR
With the new updates to blueprints, it would be great to store blueprints in a json format somewhere on disk. This would allow sharing and version control of blueprints via git/github or third party applications vs having to copy and paste JSON strings.

What ?
Right now, my understanding is that blueprints are stored in blueprint-storage.dat. I haven't taken a look at the file format, but it looks like a custom data format.

It would be great for to either

a) Store these blueprints on disk in a json format instead of the blueprint-storage.dat format
or
b) Allow users to define a directory where json files can be synced (imported/exported) with their blueprints

There are a number of things that would want to include in addition to the core functionality:

1) Ability to have blueprint books based off directory
2) Having the game recognize file changes and automatically import changes to those blueprints
3) Sorting the content in some sane way (probably based off grid positions) so that version control can be effective
4) Having the json broken out by lines (so its easier to manage in version control)
5) Have this work on a server
Why ?
For multiplayer/collaborative projects it would be awesome to use something like GitHub or GitLab to manage large blueprint libraries. It would also be awesome to be able to have third party apps interact with the blueprints and keep them up to date as the game runs.

I personally would see myself using it with a dedicated server, hooked up to a GitHub repo and using it to have a master blueprint library. If the server could pick up file changes while it runs, then I could just push new blueprints into the directory and it would just "work".

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12170
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Better filesystem storage of Blueprints

Post by ssilk »

I like the idea of having a second blueprint format. But it would be a mess. :)
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

knifeymcknifeface
Manual Inserter
Manual Inserter
Posts: 3
Joined: Thu Jul 30, 2020 6:53 pm
Contact:

Re: Better filesystem storage of Blueprints

Post by knifeymcknifeface »

Yea, TBH it might be best to just to have a single format (JSON), but broken out on the filesystem under a blueprints folder.

User avatar
jamiechi1
Fast Inserter
Fast Inserter
Posts: 134
Joined: Wed Jan 03, 2018 10:12 pm

Re: Better filesystem storage of Blueprints

Post by jamiechi1 »

I like the idea of having this information in some type of easy to read format, but I would prefer XML.

SoShootMe
Fast Inserter
Fast Inserter
Posts: 161
Joined: Mon Aug 03, 2020 4:16 pm
Contact:

Re: Better filesystem storage of Blueprints

Post by SoShootMe »

knifeymcknifeface wrote:
Thu Jul 30, 2020 10:51 pm
1) Ability to have blueprint books based off directory
2) Having the game recognize file changes and automatically import changes to those blueprints
3) Sorting the content in some sane way (probably based off grid positions) so that version control can be effective
4) Having the json broken out by lines (so its easier to manage in version control)
5) Have this work on a server
From #3 and #4 are you thinking of version control of parts of a blueprint? From what I've seen of the JSON, I think diffs aren't always useful as a change could update "unchanged" positions everywhere and merging changes within blueprints would often break them.

#1, #3 and #4 can theoretically be implemented now with some scripts but the required manual import/export is limiting.

With the new hierarchical blueprint books, books represented by directories and blueprints by files is a natural fit, and being able to version control that (eg by making the top-level directory a git repository) is an obvious use case to me.

An "External blueprints" tab (in addition to "My blueprints" and "Game blueprints") which is essentially a list of directory paths might be the easiest way to support this.
jamiechi1 wrote:
Sun Aug 09, 2020 5:29 am
I like the idea of having this information in some type of easy to read format, but I would prefer XML.
JSON is used for the existing blueprint import/export in Factorio, so I can't imagine XML being considered.

asheiduk
Inserter
Inserter
Posts: 23
Joined: Sat Dec 05, 2020 9:46 am
Contact:

Re: Better filesystem storage of Blueprints

Post by asheiduk »

I've written a tool to decode blueprint-storage.dat format into JSON and from there into the standard import/export string. With a tool like like jq you could split the JSON file into separate books/blueprints and store them in an directory and put them in version control separatly. Restoring would encompass merging them back and convert them into import/export strings.

That's by far not so convenient as requested but it is sufficient to put the blueprints into version control.

If you are using Linux you can give it a try.

Zaflis
Filter Inserter
Filter Inserter
Posts: 355
Joined: Sun Apr 24, 2016 12:51 am
Contact:

Re: Better filesystem storage of Blueprints

Post by Zaflis »

The most important thing about blueprint storage that relates to 99.9% of players is that it should be compatible when downgrading game version. This kind of futureproofing is really required for a file that contains all your delicately made creations over the years. Just to be wiped out for simply testing 1.1 and going back to 1.0.

asheiduk
Inserter
Inserter
Posts: 23
Joined: Sat Dec 05, 2020 9:46 am
Contact:

Re: Better filesystem storage of Blueprints

Post by asheiduk »

Zaflis wrote:
Tue Dec 22, 2020 9:09 am
The most important thing about blueprint storage that relates to 99.9% of players is that it should be compatible when downgrading game version.
AFAIK the import/export strings are quite stable. The mentioned tool uses that format (in uncompressed form).

Zaflis
Filter Inserter
Filter Inserter
Posts: 355
Joined: Sun Apr 24, 2016 12:51 am
Contact:

Re: Better filesystem storage of Blueprints

Post by Zaflis »

"Quite" but not really. There was reddit post after 1.1 where one lost his library after downgrading back to 1.0 but i can't find link anymore.

Problems in the past:
https://www.reddit.com/r/factorio/comme ... 10_update/
https://www.reddit.com/r/factorio/comme ... gone_help/
And this being widely known now:
you downgraded. you can't downgrade and keep your blueprints. Alas
And some less knowledgeable people are still using even cloud sync, causing even more crazy situations:
https://www.reddit.com/r/factorio/comme ... team_sync/

Anyways on topic... The library file is already fairly big for some people in the compressed form. Text/XML formats are much more "fluffy", bigger and slower to load. But indeed they can be more futureproof.

asheiduk
Inserter
Inserter
Posts: 23
Joined: Sat Dec 05, 2020 9:46 am
Contact:

Re: Better filesystem storage of Blueprints

Post by asheiduk »

I think we miss each other: All those posts seem to be about version incompatibilities with the original blueprint-storage.dat. I consider that accepted knowledge, too.

The "quite" part was about the import/export strings which - AFAIK - have not changed in an incompatible way. Even if that format should change in the future -- it is basically a text format which can be transformed into the hypothetical future format more easily than the original binary blueprint-storage.dat.

Therefore I think this JSON (uncompressed) is a much better format for long-term storage. For transfer they can be compressed.

Post Reply

Return to “Ideas and Suggestions”