Zipping the core files

[katastic]

As a fellow dev, I have a question, and I don't mean this is any kind of judgy way. Is there a reason the core files, game/space-age files aren't zipped with Factorio, the way mods are? In terms of loading time, loading 16,000 files with the operating system checking permissions on every one, is going to be so much slower than using an archive.

Thanks for the insight,
--Kat

[Rseding91]

Zip files are actually slower to load from than plain directories. The time to load from disk is irrelevant compared to the time to decompress the PNG files into RAM and then VRAM (99% of the time is spent decompressing PNG files in RAM long after they've been loaded from disk).

[katastic]

To be clear, I'm not talking about compressed archives. You can run zips with no compression where they function as simple archives.

Fun side note: There was a time when even compression was used because disks were so much slower, compressing it, sending it, and later uncompressing in CPU was faster than straight access to a disk. This was the reasoning behind NTFS compression, a low yield, fast variant of Lempel-Ziv. But I digress.

[Rseding91]

I’m aware, I was referring to storage based zip files. The image files themselves use a compression format to reduce the file size on disk. Decompressing that data into raw 4 byte per pixel data takes virtually all of the startup time.

[Rseding91]

You can try it yourself I believe. If you zip base and or core you should be able test how it performs.

[katastic]

Ah! Sorry for the misconception on my part!

And it looks like PNG decompression is multi-threaded. Perhaps that's just the nature of the beast barring some sort of faster PNG decompression library.

Hmmm.... perhaps I'm mining a dry well here.

[edit]

> You can try it yourself I believe. If you zip base and or core you should be able test how it performs.

I wasn't sure if Factorio was hardcoded to look for base/space-age in non-zipped form. I'll give that a shot for the heck of it!

[katastic]

Well, that was fun! Confirming what you said it would be:

~22.48 seconds zipped (5 - normal zip compression)
~22.48 seconds zipped (0 - STORE)
~22.10 seconds normal open directory

And of course none of those differences really matter compared to sampling error.

Also for the fun of it, I max zip compressed them at got:

~23.48s, saving a "whopping" 58 MB out of the 4 GB of files.

Which of course does nothing because zip isn't really going to compress PNG and OGG files. But interesting how little impact it has on the total load time!

So in summary, you are absolutely correct. Open files has almost impact on total load time compared to extracting the sprite sheets.

[Muche]

If you have spare free disk space, you could try advanced hidden debug setting cache-sprite-atlas.
On my old PC a warm startup takes 150s. With the cache enabled it takes only 20s.

[katastic]

OOOOOH, thanks! (if someone stumbles on this post from google, hold control and alt while clicking settings to get the extra/hidden settings)

For a pittance of 3.2 GB of cache dumped into a file (not using the compressed atlas option, only stock 'mods')

9.21 seconds! More than double the speed.