Better filesystem storage of Blueprints

[knifeymcknifeface]

TL;DR

With the new updates to blueprints, it would be great to store blueprints in a json format somewhere on disk. This would allow sharing and version control of blueprints via git/github or third party applications vs having to copy and paste JSON strings.

What ?

Right now, my understanding is that blueprints are stored in blueprint-storage.dat. I haven't taken a look at the file format, but it looks like a custom data format.

It would be great for to either

a) Store these blueprints on disk in a json format instead of the blueprint-storage.dat format
or
b) Allow users to define a directory where json files can be synced (imported/exported) with their blueprints

There are a number of things that would want to include in addition to the core functionality:

1) Ability to have blueprint books based off directory
2) Having the game recognize file changes and automatically import changes to those blueprints
3) Sorting the content in some sane way (probably based off grid positions) so that version control can be effective
4) Having the json broken out by lines (so its easier to manage in version control)
5) Have this work on a server

Why ?

For multiplayer/collaborative projects it would be awesome to use something like GitHub or GitLab to manage large blueprint libraries. It would also be awesome to be able to have third party apps interact with the blueprints and keep them up to date as the game runs.

I personally would see myself using it with a dedicated server, hooked up to a GitHub repo and using it to have a master blueprint library. If the server could pick up file changes while it runs, then I could just push new blueprints into the directory and it would just "work".

[ssilk]

I like the idea of having a second blueprint format. But it would be a mess.

[knifeymcknifeface]

Yea, TBH it might be best to just to have a single format (JSON), but broken out on the filesystem under a blueprints folder.

[jamiechi1]

I like the idea of having this information in some type of easy to read format, but I would prefer XML.

[SoShootMe]

1) Ability to have blueprint books based off directory
2) Having the game recognize file changes and automatically import changes to those blueprints
3) Sorting the content in some sane way (probably based off grid positions) so that version control can be effective
4) Having the json broken out by lines (so its easier to manage in version control)
5) Have this work on a server

From #3 and #4 are you thinking of version control of parts of a blueprint? From what I've seen of the JSON, I think diffs aren't always useful as a change could update "unchanged" positions everywhere and merging changes within blueprints would often break them.

#1, #3 and #4 can theoretically be implemented now with some scripts but the required manual import/export is limiting.

With the new hierarchical blueprint books, books represented by directories and blueprints by files is a natural fit, and being able to version control that (eg by making the top-level directory a git repository) is an obvious use case to me.

An "External blueprints" tab (in addition to "My blueprints" and "Game blueprints") which is essentially a list of directory paths might be the easiest way to support this.

I like the idea of having this information in some type of easy to read format, but I would prefer XML.

JSON is used for the existing blueprint import/export in Factorio, so I can't imagine XML being considered.

[asheiduk]

I've written a tool to decode blueprint-storage.dat format into JSON and from there into the standard import/export string. With a tool like like jq you could split the JSON file into separate books/blueprints and store them in an directory and put them in version control separatly. Restoring would encompass merging them back and convert them into import/export strings.

That's by far not so convenient as requested but it is sufficient to put the blueprints into version control.

If you are using Linux you can give it a try.

[Zaflis]

The most important thing about blueprint storage that relates to 99.9% of players is that it should be compatible when downgrading game version. This kind of futureproofing is really required for a file that contains all your delicately made creations over the years. Just to be wiped out for simply testing 1.1 and going back to 1.0.

[asheiduk]

The most important thing about blueprint storage that relates to 99.9% of players is that it should be compatible when downgrading game version.

AFAIK the import/export strings are quite stable. The mentioned tool uses that format (in uncompressed form).

[Zaflis]

"Quite" but not really. There was reddit post after 1.1 where one lost his library after downgrading back to 1.0 but i can't find link anymore.

Problems in the past:
https://www.reddit.com/r/factorio/comme ... 10_update/
https://www.reddit.com/r/factorio/comme ... gone_help/
And this being widely known now:

you downgraded. you can't downgrade and keep your blueprints. Alas

And some less knowledgeable people are still using even cloud sync, causing even more crazy situations:
https://www.reddit.com/r/factorio/comme ... team_sync/

Anyways on topic... The library file is already fairly big for some people in the compressed form. Text/XML formats are much more "fluffy", bigger and slower to load. But indeed they can be more futureproof.

[asheiduk]

I think we miss each other: All those posts seem to be about version incompatibilities with the original blueprint-storage.dat. I consider that accepted knowledge, too.

The "quite" part was about the import/export strings which - AFAIK - have not changed in an incompatible way. Even if that format should change in the future -- it is basically a text format which can be transformed into the hypothetical future format more easily than the original binary blueprint-storage.dat.

Therefore I think this JSON (uncompressed) is a much better format for long-term storage. For transfer they can be compressed.