Better compression for savegames

[Byproduct]

After noticing it took a while to sync my savegames, I got curious about the format, and gave different compression methods a try.

14.404 mb factorio version of my largest game (=80 hours in, completed one rocket)
13.499 mb "normal quality" zip using winrar
11.218 mb "normal quality" rar
11.188 mb "best quality" rar
10.923 mb "normal quality" rar5
10.872 mb "best quality" rar5
[edit:] <8mb 7zip!

In a 14 mb file it of course doesn't matter much. But I've heard megabase saves can get as large as 100mb? And cause steam sync hiccups? So if the size difference scales and you can cut the size down to 60-80% by just changing the compression method, maybe that's worth looking into?

I also suspect the zip library used in factorio might be out of date, since zipping it using default winrar settings produces a smaller file, while still being really fast.

I know nothing of file compression though, or if it needs licensing or whatnot, or if this has been already discussed, so sorry if I'm talking out of my ass.

Cheers!

[darkfrei]

Another example:
zip archive from the game folder: 171 MB
unpacked: 382 MB
packed to 7z ("normal" compression): 97 MB; packaging time: 49 s
packed with 7-Zip to zip ("normal" compression): 150 MB; packaging time: 21 s

[Byproduct]

Oh yeah, I forgot about 7zip. Looks like it's even better than rar5. I don't have the exact same save anymore, but one of the four 7zip compression options (PPMd on Ultra) squeezed the save down to 8.7mb instead of factorio's 15.0mb zip. Nice!

That one took a bit of time, but other 7zip methods managed under 10mb too.

[Tekky]

Some games don't apply compression to autosaves (or only minimal compression), whereas apply heavy compression to manually saved games.

I'm not sure if this already is the case with Factorio, but, if not, I would be in favor of this being implemented. This would be meaningful, because, when autosaving, speed is most important, whereas when manually saving, space is most important.

When manually saving, having a checkbox with the option of applying heavy compression (which is enabled by default) might also be nice.

[bobingabout]

I believe Factorio uses the Zip format in it's save games. Part of the reason the compression is so crap is that it's optimised more for speed, to minimise the interuption during gameplay.

[Optera]

I like to have cake and also eat it so my solution to minimal interruption for backups is to make a snapshot/shadow copy and have that compressed to the maximum by a background process/thread.
Worst case my backup is already a few seconds to minutes old when created.

[keyboardhack]

Apparently you can se the compression level in the config file viewtopic.php?f=6&t=56287&p=331759&hili ... ip#p331759

I added support to set the compression level for autosaves and for multiplayer peer joining in the config for the next version of 0.16.

The main reason we don't currently use any third party compression library is so anyone on any OS can simply open the zip and modify/see it as they please. Also better compression methods are slower (such as increasing the compression level in the zip we use now).

[Byproduct]

Some games don't apply compression to autosaves (or only minimal compression), whereas apply heavy compression to manually saved games.

Useful approach, but I think it's still a problem if you have a 100mb savegame for example. You'll be uploading half a gigabyte to Steam every time just to send 5 autosaves.

As Optera said, savegame compression can usually be done in a separate thread. Make a quick uncompressed save first, then run a separate thread to compress the savegame. Doesn't interfere with the game cpu much, only uses some extra RAM. Factorio may already be doing something like this, but the compression could be much heavier in my opinion - looks like switching to 7zip could cut the current save file sizes to half.

The main reason we don't currently use any third party compression library is so anyone on any OS can simply open the zip and modify/see it as they please. Also better compression methods are slower (such as increasing the compression level in the zip we use now).

For example 7zip or rar aren't as common as zip, but surely anyone who wants to poke inside save files knows what a .7z or .rar is? It doesn't sound like a problem to me. I mean sure, .zip is absolutely the most popular one, but if the other options are also popular enough and can halve the file size – sounds worth it to me!

Speed is also not a problem, if compressed in a separate thread while the game is running. May want to avoid the most extreme settings, but for example 7zip does great using the default ones.

[quyxkh]

zstd seems to deserve that moniker they picked for it, it gets better compression than zip -9 in less time than zip -1.

[Darinth]

My only concern for this would be 'amount of time to implement'. The initial save functionally requires a game pause to get all of the data from one 'frame' of the game written to a file. Once it's written to a file, there's no reason that a background process can't operate to do fairy high level of compression. So I think it *can* be done, I'm just not certain it's worth the extra effort.

Especially when you take into account things like: Not compressing the original save may actually cause the save process to take more time, depending on where the bottleneck is on the save time. If the biggest bottleneck is simply in hard drive speed, compression actually cuts down on the amount of time to save which means the background process should go through the pain to uncompress the saved game before recompressing it.

Another option would be to write the saved game to a variable in memory (even 100 megs of ram now that factorio run on x64 architecture isn't huge), and once it has a snap-shot of the game saved in RAM another thread can do a high level of compression before/as it writes to disk. That might actually be worth some more time to look into. If the hard-drive can be completely bypassed for the initial save and it's the biggest bottleneck, that could *drastically* reduce save times.

[Byproduct]

So I think it *can* be done, I'm just not certain it's worth the extra effort.

Certainly not a critical issue nor top priority, but the huge cloud-synced save files are a bit of a problem, enough that reduction seems like a substantial benefit to me. As for effort, well, again I might be talking out of my ass and sorry if I am, but what's suggested is probably easy, if not trivial, to someone who can code something as amazing as factorio. Even I might be able to approach a task like that, and I suck at coding.

About the bottleneck...

Yeah it would make sense to create the uncompressed save in RAM. But even if you had to do it on the disk for some reason, it still doesn't sound like a problem if it's a background process while the game is running. SSDs can write hundreds of megabytes per second, and slow drives should also manage it in a few seconds tops.

[Jeeto]

I also want the Steam version to sync savefiles to Steam more often. A lot of times I will play on my desktop, save, exit game, and shutdown computer and the next day my laptop only has savefiles from like weeks ago.

[Darinth]

I also want the Steam version to sync savefiles to Steam more often. A lot of times I will play on my desktop, save, exit game, and shutdown computer and the next day my laptop only has savefiles from like weeks ago.

My base instinct is that this isn't something Wube can control. I think that this is done automatically by steam when the game is exited; I've never seen a game that performs it's syncing while the game is running. I'd love to be corrected if I'm wrong though, it'd be cool to know that steam has capabilities to do cloud syncs in the background while a game is running.

[Jeeto]

My base instinct is that this isn't something Wube can control. I think that this is done automatically by steam when the game is exited; I've never seen a game that performs it's syncing while the game is running. I'd love to be corrected if I'm wrong though, it'd be cool to know that steam has capabilities to do cloud syncs in the background while a game is running.

Yeah but how long does it take to sync, maybe there's a lag. Cause I will close the game on my desktop, reply to an email or two or watch a YouTube video, and then shutdown the desktop and then the next day my laptop doesn't have any of the savefiles from yesterday on it. I don't know if its something Wube can put into the application closure script or if it's some setting I need to tweak on Steam. I'm sick of using a thumbdrive to copy-paste savefile zips from %appdata%

[Byproduct]

It should sync every time I think, but it's missed it a few times for me too, so I use dropbox if I know I'm going to play on the laptop next.

I wonder if its hiccups are related to the unusually large saves?

[Jeeto]

Well I do remember the first time I installed it on my laptop, I copied over my current savefile before opening it and then there was a sync error notification and I told it to use local data instead of the cloud data... and since then it's never loaded the cloud data ever again. So maybe is there a way in Steam to tell it to forget that sync preference and use cloud data?