Use ZSTD for savegame compression (much faster, smaller)

[hansinator]

Hi,

I've got a map that produces a 58mb zip file savegame. It takes around 5 seconds to do this and autosave becomes quite annoying. It looks like you are using deflate already with the fastest setting for compression. Letting deflate run on the raw data with its fastest settings gives me a compression time of 4 seconds.
I've got a hexa-core broadwell-e overclocked to 4,2GHz and NVMe SSDs so the bottleneck is most likely not the CPU+disk and not the game's save routines but the compression.

I suggest you use Zstd instead (http://facebook.github.io/zstd/). It compresses the same data in 1 second while producing a 53mb file (using level 3, 1 is lowest and even faster). Zstd is in most aspects superior to deflate. I've tested a couple of large savegames all with consistent results. This would reduce the time it takes to produce a savegame roughly by factor 4.

I suppose you are not able to snapshot the game state in-memory and let the game continue while the snapshot is compressed and written out to disk on a separate thread?

[ssilk]

Hm. Not against this. But I need to give you the discussion from that time.
I think I remember the arguments:
1. Zip is standard. Zip-Standard (ZSTD) is not!
2. Faster plays no role: The unpacking is done by the second CPU in parallel. The serialization/deserialisation is the problem (memory-speed!).
3. Less size of files. Always good. But i it worth the adfford?

[Rseding91]

I suppose you are not able to snapshot the game state in-memory and let the game continue while the snapshot is compressed and written out to disk on a separate thread?

Currently the game state is copied out in one thread while another thread does the compression and writing to disk. 5 seconds for a 50 MB save file sounds like it's waiting on your computer. When I implemented the threaded compression during saving for 0.13.0 I was testing with a 65 MB save file and it would save in 3.2 seconds. Of that 3.2 seconds 2.5~ worth was copying the memory around and the remaining 0.7~ was spent compressing and writing to disk.

So, compression is not near as much time as you'd think.

[Supercheese]

So, compression is not near as much time as you'd think.

Perhaps the compression ratio could then be improved, since performance is already quite nice.

[Rseding91]

So, compression is not near as much time as you'd think.

Perhaps the compression ratio could then be improved, since performance is already quite nice.

Increasing the compression increases the time spent so it does have an impact. It's just the percentage spent compressing right now is quite small compared to the total time spent.

[hansinator]

Currently the game state is copied out in one thread while another thread does the compression and writing to disk. 5 seconds for a 50 MB save file sounds like it's waiting on your computer. When I implemented the threaded compression during saving for 0.13.0 I was testing with a 65 MB save file and it would save in 3.2 seconds. Of that 3.2 seconds 2.5~ worth was copying the memory around and the remaining 0.7~ was spent compressing and writing to disk.

So, compression is not near as much time as you'd think.

I am pretty sure it is not waiting for my computer.. I made sure my CPUs+disk are not busy and the system itself is just one of the fastest you can get for a desktop PC right now so it should be on par with your results or faster, unless you got an even faster system which is unlikely.

How did you measure the time that was spent on compression?
Did you try uncompress a save file to a ramdisk and then just compress it using a zlib based cmdline utilty and measure the time?
That's what I did and it repeatedly takes about 4 seconds (a little less) using the fastest deflate setting. The uncompressed size of my test data is 122MB.
Either there must be something else to it or one of us obtained bogus results.

It's really not hard to implement zstd, you're done in less than an hour. Then you would know if it improves the concrete situation when saving a game.
I don't really see an argument against it.

[hansinator]

Hm. Not against this. But I need to give you the discussion from that time.
I think I remember the arguments:
1. Zip is standard. Zip-Standard (ZSTD) is not!
2. Faster plays no role: The unpacking is done by the second CPU in parallel. The serialization/deserialisation is the problem (memory-speed!).
3. Less size of files. Always good. But i it worth the adfford?

1. Zip is a de-facto standard, not a standard
1.1 Zip is a file format, not a compression algorithm, the algorithm is called deflate and it merely has an RFC
2. I have got quad channel DDR4 memory with a peak bandwidth of 45GByte/s. I benchmarked the memory speed so its no theoretical value. It can't be the memory speed.
3. It is basically no effort to integrate Zstd. It supports single-call interfaces as well as a zlib-style API for drop-in replacement. I strongly believe it already took more effort discussing it here.

It even doesn't matter if Zstd is a standard or not - why should it? The compression format is stable right now so it probably won't change a lot in the future. And even if it does it is no problem, because Factorio savegames only need to be read by Factorio and the devs have full control over which library version they use.

[hansinator]

I've run some additional benchmarks.
My disk can read with 2.8GByte/s and write with ~880MByte/s.
Deflate (zlib) achieves around 90MByte/s from RAM to disk when compressing a 1GB file (source code). It reaches only 55MByte/s when I compress ~600MB of raw Factorio savegame data.
Zstd reaches 220Mbyte/s.

My results for deflate throughput are consistent with this paper from Google employees: https://cran.r-project.org/web/packages ... -09-22.pdf
My results for Zstd (and deflate, too) are somewhat consistent with the "official" benchmarks by Zstd. The used lzbench and only did in-memory compression.

@rseding
If you say you had a 65MB save let's assume it is at least 100MB uncompressed. With roughly 90MB/s we get a theoretical compression time of 1.1 seconds.
You said you have measured 0.7 seconds. Either you have a special hand-tweaked deflate algorithm that is faster than what everybody else is using or your measurement method is flawed or there is something special with Factorio data that makes it speedier to compress. The latter case is unlikely, because my results show less than ideal throughput on Factorio data. The numbers contradict yours.

With Zstd you can still cut the time by a factor of approx 3.5. That would mean instead of 1s compression time we get 280ms _and_ smaller files!

[hansinator]

Going through the debug symbols it shows you are indeed using Zlib. As I said, Zstd has a Zlib-style API for drop-in replacement. It'll take you a couple of minutes to try using that, including download and compilation. No effort.

Edit:
I've used the undecorator mod to remove all decorations from my largest savegame. It the size reduced approximately in half from currently 58MB to 26MB. The time needed to save the game reduced from roughly 5 seconds to ~1 second. Thats odd, right? One might suspect that serialization is the slowest component.. but I can confirm an increase in compression throughput on undecorated savegames.

I've used several decoration-stripped savegames and made a folder with 60MB of uncompressed save data. The compression speed on this data increases and I can measure about 1.1 seconds from ramdisk to disk using a command line utility.

It seems to me that savegames with a lot of decorations make deflate have less throughput. Could it be that you have benchmarked the speeds with a version that produces less decorated savegames?

[ssilk]

I added this thread to
viewtopic.php?f=80&t=21367 Game Save File ans -Speed Related Suggestions

Please avoid double (triple) postings. No good style.

One might suspect that serialization is the slowest component. but I can confirm an increase in compression throughput on undecorated savegames.

That lets room for interpretatoin. IMHO: The serialization is much slower than the compression.

Without the tools to measure this correctly, it is a dig in the mud.

And I'm sure you know much about compression etc. But to improve save-speed (load-speed), it is needed, that you find the smallest bottleneck, and this (and other) discussion came to the conclusion, that that isn't compression.

[DarkShadow44]

I tested four saves I consider big:

• big: Just an empty widely explored world
• reactor: A massive nuclear reactor setup, otherwise empty world
• vanilla: My save that started as vanilla, biggest one I have so far
• se: Space exploration save from me, lots of explored planets

How tests were done:

• Save files as uncompressed (factorio extra settings, autosaves compression level)
• Let Factorio compress then as .zip at fast / maximum compression (same setting)
• Compress the uncompressed saves with various compression alorithms

The factorio created saves are used as baseline, to show the limit of how small saves can currently get. Time gotten from Factorio log.

External compression is timed using the "time" command (real time).
The times are pretty different from Factorio, since AFAIK Factorio uses multiple threads, and the time includes gathering the data as well. Factorio also compresses chunks individually AFAIK. Those tools I used are all single threads and compress the whole zip, that's what I added zip (deflate) as comparison.

All data gathered on my Linux Ryzen 3800X, on an SSD. All elements are "Size in MB @ time taken to compress".
Single core use forced with "taskset -c 1"

Data:

Code: Select all

                                    se                     vanilla            reactor             big
0) raw                              686.3                  153.1              182.1               217.7
1) Factorio fast                    290.5 @    3.8s         63.8 @  1.2s       61.8 @  1.0s       101.2 @   1.2s
2) Factorio max                     266.7 @    9.6s         56.6 @  3.4s       56.4 @  1.8s        92.0 @   3.6s
Single core
3) 7z a out.zip in.zip              255.7 @   64.4s         54.2 @ 15.6s       54.0 @ 17.7s        87.4 @  24.5s
4) 7z a out.7z in.zip               212.7 @  300.0s         44.7 @ 68.5s       41.7 @ 75.8s        74.3 @ 120.2s
5) zstd -3 in.zip -o out.zstd       285.0 @    4.6s         57.7 @  1.0s       55.9 @  0.9s        98.9 @   1.6s
6) zstd -9 in.zip -o out.zstd       264.3 @   14.3s         53.1 @  3.4s       52.0 @  2.9s        90.8 @   5.6s
Multicore
7) zstd -T4 -9 in.zip -o out.zstd   264.3 @    4.3s         53.1 @  1.2s       52.0 @  1.0s        90.8 @   1.6s
8) zstd -T8 -9 in.zip -o out.zstd   264.3 @    3.5s         53.1 @  0.9s       52.0 @  0.9s        90.8 @   1.4s
9 xz -k -T8 -1 in.zip               244.1 @    6.6s         50.7 @  1.4s       48.2 @  1.5s        86.7 @   1.4s

Test 4 shows what I consider the limits of compression, the smallest size you can get.
In Test 8 with 8 threads we're most likely IO bound, should be a lot faster in memory

Overall zstd seems to beat "Factorio fast", even single core is almost as fast as multithreaded factorio save, while being smaller.
And the jump from "Factorio fast" to "Factorio slow" is also bigger than the entire multicore zstd compression, so there's that.

Still not sure what exactly to make out of that, your interpretation?

[azesmbog]

What do you know about saving the game))
p-f-f-f-f-f
My save (vanilla clean) current time is 65 seconds, and this can be said almost instantly. Therefore, the interval between autosaves is 1 hour. And even then, I save more in order not to get into the "zero problem" / For me, the main thing is that the data is not distorted. Alas, in this game such a nuisance happens ...

[Rseding91]

Factorio has a ZSTD implementation that is used for some cache files if compression of those is enabled. So I can just swap the compression used for saves to test it. These are my results:

Raw save file size (disabling compression): 259 MB

Save time with factorio-threaded zstd compression level 1:

• 1.17468
• 1.13124
• 1.14024
• 1.14004

Compressed size: 98.3 MB

Save time with Factorio-threaded deflate compression level 1:

• 1.21332
• 1.18647
• 1.18466
• 1.20887

Compressed size: 96 MB

Save time just skipping compression and skipping writing to disk:

• 1.16021
• 1.11839
• 1.12461
• 1.13545

Compressed size: 0 MB

[Rseding91]

So in conclusion (today); it would be faster - by a near insignificant amount of time. The *vast* amount of time is just spent preparing the data for the "compression + write-to-disk logic."

[DarkShadow44]

Well, you're testing the fasted settings of the compression alorithms, no? What are the results with e.g. zlib -9 vs zstd -9? It only becomes interesting if you're not IO bound, I think. Especially for MP download saves, IMHO.

One more thing, it's still chunked into levelXYZ.dat, right? I do believe you get better results if you compress above all (which zstd seems to support). Is probably a bigger rework, though...

[FuryoftheStars]

Well, you're testing the fasted settings of the compression alorithms, no? What are the results with e.g. zlib -9 vs zstd -9? It only becomes interesting if you're not IO bound, I think. Especially for MP download saves, IMHO.

Was this thread not about save times to begin with? Why would you test on the slower settings if the goal is faster saves?

[DarkShadow44]

I thought it was about file sizes as well. And since zstd ist so much faster, you can run it at higher settings for the same time and get smaller saves basically for free.

[FuryoftheStars]

My interpretation of it was the save time was their biggest beef, and that the save size was just an added bonus.

[lyvgbfh]

And since zstd ist so much faster

I think in this case it isn't a huge difference, both in rseding's and the previous benchmarks

[Rseding91]

Using the same save file:

ZSTD Compression 9: 1.52269 - 84.8 MB

ZSTD Compression 22: 5.46916 - 77.6 MB

Zip compression 9: 3.53715 - 85.9