`zstd -1` on atlas caches could double even SSD read speed

Suggestions that were added to the game.
Post Reply
quyxkh
Filter Inserter
Filter Inserter
Posts: 727
Joined: Sun May 08, 2016 9:01 am
Contact:

`zstd -1` on atlas caches could double even SSD read speed

Post by quyxkh » Sat Sep 29, 2018 5:19 pm

It occurrred to me to check how zstd would do on the atlas caches, the vanilla one's about 3GB. `zstd -1` compresses at ~350MB/s on my 5-year-old midrange box to 22% of the original size, ~3GB to ~0.66GB, which is enough to help even SSDs some when writing, but the decompress performance is ridiculous, I get ~1GB/s decompression, it takes less than three seconds to decompress it with hot caches and less than five seconds straight off my 150MB/s HDD. That'd cut cold-start time noticeably even on SSDs, on HDDs the difference would be really gratifying.

posila
Factorio Staff
Factorio Staff
Posts: 3788
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by posila » Wed Jul 10, 2019 11:40 am

In 0.17.56, there will be config.ini setting "compress-sprite-atlas-cache". At the moment it uses compression level 1, we may change it to -1 in the future. Also it doesn't utilize parallel compression/decompression, so I am not sure if it'll make 350 MB/s resp. 1 GB/s on your PC, but it seemed pretty fast on mine.

quyxkh
Filter Inserter
Filter Inserter
Posts: 727
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by quyxkh » Wed Jul 10, 2019 5:26 pm

the `zstd -1` was meant to be a command quote, that gets the zstd level 1 compression you implemented. _thanks_! (and also, btw, thanks for whatever magic you worked that made loading the atlas from a hot OS buffer cache so close to instantaneous, that still gets a little internal smile of gratification when it hits).

dimm
Burner Inserter
Burner Inserter
Posts: 11
Joined: Tue Jun 06, 2017 12:04 pm
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by dimm » Fri Jul 12, 2019 7:17 pm

The game now loads very quickly. Thanks!

quyxkh
Filter Inserter
Filter Inserter
Posts: 727
Joined: Sun May 08, 2016 9:01 am
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by quyxkh » Fri Jul 26, 2019 2:04 pm

Lol just checked out the data rates on the new PCIe SSDs, they're stupid fast. Thanks for indulging us older-hw users on this.

slippycheeze
Filter Inserter
Filter Inserter
Posts: 482
Joined: Sun Jun 09, 2019 10:40 pm
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by slippycheeze » Wed Jul 31, 2019 11:09 pm

quyxkh wrote:
Fri Jul 26, 2019 2:04 pm
Lol just checked out the data rates on the new PCIe SSDs, they're stupid fast. Thanks for indulging us older-hw users on this.
You are not wrong, but... to quote from Latency at a human scale -- which scales one CPU cycle to one second, and reflects modern system latency:

L3 cache is about one minute away from the CPU, main memory is about four minutes, and the super-fast Optane stuff? 15 minutes for persistent memory, 7 hours for the DC SSD version. 17-ish hours for the average NVMe SSD.

CPU power is vastly, vastly cheaper than any sort of I/O, to the point that as long as you don't need a huge dictionary, compressed data is almost always a performance win even in main memory, so long as you don't need random access inside pages.

Darinth
Filter Inserter
Filter Inserter
Posts: 321
Joined: Wed Oct 17, 2018 12:17 pm
Contact:

Re: `zstd -1` on atlas caches could double even SSD read speed

Post by Darinth » Thu Aug 01, 2019 1:15 pm

slippycheeze wrote:
Wed Jul 31, 2019 11:09 pm
quyxkh wrote:
Fri Jul 26, 2019 2:04 pm
Lol just checked out the data rates on the new PCIe SSDs, they're stupid fast. Thanks for indulging us older-hw users on this.
You are not wrong, but... to quote from Latency at a human scale -- which scales one CPU cycle to one second, and reflects modern system latency:

L3 cache is about one minute away from the CPU, main memory is about four minutes, and the super-fast Optane stuff? 15 minutes for persistent memory, 7 hours for the DC SSD version. 17-ish hours for the average NVMe SSD.

CPU power is vastly, vastly cheaper than any sort of I/O, to the point that as long as you don't need a huge dictionary, compressed data is almost always a performance win even in main memory, so long as you don't need random access inside pages.
That's actually a really cool and useful way to try and explain the gap between the different components in a human-readable way. Thank you for providing that.

Post Reply

Return to “Implemented Suggestions”

Who is online

Users browsing this forum: No registered users