Game download files: Slow to verify and unpack on macOS

[squiddog]

This is more of an observation than a bug or help report.

When I download a new DMG from the Factorio download site, the DMG file itself goes through an integrity check that is VERY slow. We are allowed to skip this in macOS. But then the unpacking from the DMG to Applications is just as slow. This is do to it consuming 100% of a single CPU in diskimagehelper which unpacks it.

My guess is that the Factorio team are using the highest level of compression in order to get the smallest possible package for downloads (reduces CDN costs, download times).

My other guess is that if they used a slightly LESS maximal compression level, that the file size would not be that much bigger, and that macOS could unpack it more quickly.

Thoughts?

[willis936]

I am under the impression that compression level has little influence over the decompression time. The primary thing that changes in compression level is dictionary size, which has very little influence over decompression time (for most computers + data, I'm certain corner cases can be found). Much more important is the data itself. Factorio has is 1.8 GB over nearly 11000 files. Large numbers of files introduces signficant overhead in directory operations.

https://quixdb.github.io/squash-benchma ... ompression

[squiddog]

Sorry for not seeing your response on a timely basis!

I did some testing and have some results that may surprise you.

If I take the latest DMG (factorio_alpha_1.1.35) and copy the Factorio.app out of it into a folder, it takes multiple minutes to do so. The `diskimages-helper` process is pegged near 100% CPU (i.e. single core, must be single threaded). The file is written out at 10s of MB/s.

If I then create a new DMG using Disk Utility (File > New image > Image from folder), read-only with no compression, the file is larger (1.78GB vs. 1.48GB) but the copy from the DMG happens in just a few (about 5) seconds.

Next I created a new DMG using Disk Utility (File > New image > Image from folder), compressed, and the file is marginally larger than the official image at 1.52GB, but also decompresses quickly in about 5 seconds.

Official image

Code: Select all

hdiutil imageinfo factorio_alpha_1.1.35.dmg
Format Description: UDIF read-only compressed (bzip2)
Class Name: CUDIFDiskImage
Checksum Type: CRC32
Size Information:
	CUDIFEncoding-bytes-total: 1476498456
	Total Empty Bytes: 134227968
	Sector Count: 3735208
	Total Bytes: 1912426496
	CUDIFEncoding-bytes-wasted: 0
	Total Non-Empty Bytes: 1778198528
	Compressed Ratio: 0.83033385122676251
	CUDIFEncoding-bytes-in-use: 1476498456
	Compressed Bytes: 1476498456
Checksum Value: $B48817E3

My test compressed image

Code: Select all

hdiutil imageinfo FactorioTest-compressed.dmg
Format Description: UDIF read-only compressed (zlib)
Class Name: CUDIFDiskImage
Checksum Type: CRC32
Size Information:
	CUDIFEncoding-bytes-total: 1523498235
	Total Empty Bytes: 572953088
	Sector Count: 4606812
	Total Bytes: 2358687744
	CUDIFEncoding-bytes-wasted: 18679022
	Total Non-Empty Bytes: 1785734656
	Compressed Ratio: 0.84268913914162169
	CUDIFEncoding-bytes-in-use: 1504819213
	Compressed Bytes: 1504819213
Checksum Value: $ED5F316B

So it's apparent that the tool used by the official build system is using bzip2 and while it makes a slightly smaller compressed image, it's MUCH slower to decompress.

If it's possible for the team to try it with zlib vs. bzip2, I think they'll see similar results.


System:

iMac Pro (2017)
3.2GHz 8-core Xeon
64GB RAM
1TB NVMe SSD.