Better compress Saves

[Hundifan33]

I tried unpacking a Save and then recompressing it and noticed, that with 7zip or other compression algorithms that use lzma the file size was reduced by up to 50% (even when compressing it as a normal zip, I saved 10% in space) while this wouldn't be very important when playing alone (Drive Space is pretty much unlimited today) it would make a huge difference in Multiplayer (especially when you have a slow upstream).

[golfmiketango]

those are very slow compression techniques (even though they are fast to decompress). The new hotness for stuff like this is zstd.

[Rseding91]

I added support to set the compression level for autosaves and for multiplayer peer joining in the config for the next version of 0.16.

The main reason we don't currently use any third party compression library is so anyone on any OS can simply open the zip and modify/see it as they please. Also better compression methods are slower (such as increasing the compression level in the zip we use now).

[eradicator]

As someone often plagued by slow connections i can say that in the special case of multiplayer joining even using a significantly slower compression algorithm (let's say one that takes 10 seconds to compress a 100MB map but makes it 50% smaller) would for many people still result in a significant net gain in joining speed due to the smaller download requirement.

Example:
Map Size 50MB, Download speed: 200kb/s
Without compression: 4m10s
With 50% compression and 10s de/compression overhead each: (4m10s)/2 + 20s = 2m25s ~40% reduced join duration. (Which also reduces catch up duration further)

Of course it's kinda difficult to heuristically determine if for a given join attempt strong compression would be faster. But a config (or even server hosting gui) choice for a stronger algorithm for joining would definetly be appreciated. The normal savegames could still be saved as zip files for easy opening.

[Zeblote]

200kb/s? What year is this?

[ske]

200kb/s? What year is this?

Not everybody has fast internet. Sad but true.

[BlakeMW]

Maybe one possibility would be (a config option for) enabling lzma for zip files (I guess this would be easier if the zip library being used natively supports lzma as a compression option). Typically lzma at the lowest compression setting results in zip files about 25% smaller than deflate, and is faster than deflate at max compression and still only about 4x slower than the moderate settings. Zip files using lzma compression aren't as widely supported as normal zip files but they're still pretty normal.

[Rseding91]

You don't get 50% better compression using higher compression settings on normal factorio saves. Which is one of the reasons we didn't do it before. It's also significantly slower than fast compression.

[eradicator]

You don't get 50% better compression using higher compression settings on normal factorio saves. Which is one of the reasons we didn't do it before. It's also significantly slower than fast compression.

I was suggesting using a different algorithm for multiplayer joining, not higher settings for deflate. I did a compression test using Xterminators 4K SPM base as a sample because it's 98.3MB size is so nicely close to 100MB. With 7z "fastest" settings (LZMA2, 64KB dictionary, word size 32, solid block 8MB, limited to 2 threads) it takes 11 seconds to compress and 7 seconds to decompress. Compressed size is 75.7MB, so still a net gain of 23%. Which means that using this compression level results in a faster join time for anybody with less than 1250kB/s download speed in this case (the speed it takes to download 23% more data in 18 seconds). LZMA(1) takes 22+7 seconds for the same compression level.

200kb/s? What year is this?

It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.

[bobingabout]

200kb/s? What year is this?

my god, that's lightning!

I think my upload speed is currently limited to 50Mbps, and I'm on fibre optic, but I remember back on my first ADSL, I had a whole 100kbps to work with on my upload. that's kilobits, so it's more like 10 kilobytes a second. The speed I had before upgrading to fibre was still hardcapped to 1Mbps, which more realistically was closer to 750kbps due to line quality, which is maybe 100 kilobytes a second upload on a good day. And this is STANDARD where I live, most people can't even get these kinds of high speeds. I only live just over a mile from the exchange.

Download speeds were 1Mbps when I first got ADSL, upgraded to 8Mbps, and eventually to 24Mbps when the line was upgraded to ADSL2, but line quality resulted in about 10Mbps.

so either way you look at it, 200 kbps peer to peer map transfer speed over the internet is LIGHTNING fast, no joke!

My Australian friend (Who lives out in the middle of nowhere) can only dream of the speeds I used to have! He probably gets that 200kbps DOWNLOAD on a good day.

[Zeblote]

200kb/s? What year is this?

It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.

It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...

[Koub]

200kb/s? What year is this?

It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.

It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...

I'm sure there are more people in the world with bandwidth close to eradicator's than there are with yours. Try to be comprehensive, and remember you're lucky ^^.

[eradicator]

It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...

It's also completely random. If i drive down the road a few hundred meters there's ads for FTTB. The shitty upload speeds are "traditional" here though, even the ppl with 100MB/s download only get 10 up, and the ratio gets worse for slower connections. Also remember that not everybody lives in the city center.

[bobingabout]

200kb/s? What year is this?

It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.

It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...

I don't even understand how that can be possible, where do you live?


Anyway, my input on the subject. autosaves, make 'em snappy. Manual saves, well, you've already broken from the game, put more time and effort into the compression, an extra couple of seconds processing the save shouldn't be an issue.

[olafthecat]

Sadly, my upload speed is in the Mb as well.
I use Plus Net, which is all right and have fibre optic and a decent router.
Come to think of it, HOW DID YOU GET THAT SANIC SPEED!?

I don't even understand how that can be possible, where do you live?

He/She probably either lives in Dubai or in the 22nd century
By the way, I think mine is around 30-40Mb on a good day.

[Zavian]

My upload is around 400 kilobits/s, on a good day.

[eradicator]

Anyway, my input on the subject. autosaves, make 'em snappy. Manual saves, well, you've already broken from the game, put more time and effort into the compression, an extra couple of seconds processing the save shouldn't be an issue.

I don't like the distinction between auto/manual save. I do agree with Rsed too that it's nice to have good compatability (i.e. deflate) though. Which is why i explicitly said i'd like this for mp joining only (because as far as i know the state sent to joining players is not saved to disk (permanently)?).
_______

It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...
[...]
My upload is around 400 kilobits/s, on a good day.

I never said anything about having a speed of less than 400 bits/s.
And just to clarify, i don't participate in ISP ad based number boosing, so all values i don't explicitly state as "bit" are in byte (per second).
______
And also we should stop derailing this thread into bandwidth bragging....