Better compress Saves

Post your ideas and suggestions how to improve the game.
Post Reply
Hundifan33
Manual Inserter
Manual Inserter
Posts: 1
Joined: Tue Sep 12, 2017 6:47 pm
Contact:

Better compress Saves

Post by Hundifan33 » Sat Jan 06, 2018 10:53 pm

I tried unpacking a Save and then recompressing it and noticed, that with 7zip or other compression algorithms that use lzma the file size was reduced by up to 50% (even when compressing it as a normal zip, I saved 10% in space) while this wouldn't be very important when playing alone (Drive Space is pretty much unlimited today) it would make a huge difference in Multiplayer (especially when you have a slow upstream).

golfmiketango
Filter Inserter
Filter Inserter
Posts: 524
Joined: Fri Jan 29, 2016 2:48 am
Contact:

Re: Better compress Saves

Post by golfmiketango » Sun Jan 07, 2018 12:41 am

those are very slow compression techniques (even though they are fast to decompress). The new hotness for stuff like this is zstd.

Rseding91
Factorio Staff
Factorio Staff
Posts: 8647
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Better compress Saves

Post by Rseding91 » Sun Jan 07, 2018 1:44 am

I added support to set the compression level for autosaves and for multiplayer peer joining in the config for the next version of 0.16.

The main reason we don't currently use any third party compression library is so anyone on any OS can simply open the zip and modify/see it as they please. Also better compression methods are slower (such as increasing the compression level in the zip we use now).
If you want to get ahold of me I'm almost always on Discord.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Better compress Saves

Post by eradicator » Sun Jan 07, 2018 11:55 am

As someone often plagued by slow connections i can say that in the special case of multiplayer joining even using a significantly slower compression algorithm (let's say one that takes 10 seconds to compress a 100MB map but makes it 50% smaller) would for many people still result in a significant net gain in joining speed due to the smaller download requirement.

Example:
Map Size 50MB, Download speed: 200kb/s
Without compression: 4m10s
With 50% compression and 10s de/compression overhead each: (4m10s)/2 + 20s = 2m25s ~40% reduced join duration. (Which also reduces catch up duration further)

Of course it's kinda difficult to heuristically determine if for a given join attempt strong compression would be faster. But a config (or even server hosting gui) choice for a stronger algorithm for joining would definetly be appreciated. The normal savegames could still be saved as zip files for easy opening.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

Zeblote
Filter Inserter
Filter Inserter
Posts: 972
Joined: Fri Oct 31, 2014 11:55 am
Contact:

Re: Better compress Saves

Post by Zeblote » Sun Jan 07, 2018 3:05 pm

200kb/s? What year is this?

ske
Filter Inserter
Filter Inserter
Posts: 370
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Better compress Saves

Post by ske » Sun Jan 07, 2018 6:38 pm

Zeblote wrote:200kb/s? What year is this?
Not everybody has fast internet. Sad but true.

BlakeMW
Filter Inserter
Filter Inserter
Posts: 915
Joined: Thu Jan 21, 2016 9:29 am
Contact:

Re: Better compress Saves

Post by BlakeMW » Sun Jan 07, 2018 7:37 pm

Maybe one possibility would be (a config option for) enabling lzma for zip files (I guess this would be easier if the zip library being used natively supports lzma as a compression option). Typically lzma at the lowest compression setting results in zip files about 25% smaller than deflate, and is faster than deflate at max compression and still only about 4x slower than the moderate settings. Zip files using lzma compression aren't as widely supported as normal zip files but they're still pretty normal.

Rseding91
Factorio Staff
Factorio Staff
Posts: 8647
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Better compress Saves

Post by Rseding91 » Sun Jan 07, 2018 8:28 pm

You don't get 50% better compression using higher compression settings on normal factorio saves. Which is one of the reasons we didn't do it before. It's also significantly slower than fast compression.
If you want to get ahold of me I'm almost always on Discord.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Better compress Saves

Post by eradicator » Mon Jan 08, 2018 4:10 am

Rseding91 wrote:You don't get 50% better compression using higher compression settings on normal factorio saves. Which is one of the reasons we didn't do it before. It's also significantly slower than fast compression.
I was suggesting using a different algorithm for multiplayer joining, not higher settings for deflate. I did a compression test using Xterminators 4K SPM base as a sample because it's 98.3MB size is so nicely close to 100MB. With 7z "fastest" settings (LZMA2, 64KB dictionary, word size 32, solid block 8MB, limited to 2 threads) it takes 11 seconds to compress and 7 seconds to decompress. Compressed size is 75.7MB, so still a net gain of 23%. Which means that using this compression level results in a faster join time for anybody with less than 1250kB/s download speed in this case (the speed it takes to download 23% more data in 18 seconds). LZMA(1) takes 22+7 seconds for the same compression level.
Zeblote wrote:200kb/s? What year is this?
It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

User avatar
bobingabout
Smart Inserter
Smart Inserter
Posts: 6350
Joined: Fri May 09, 2014 1:01 pm
Contact:

Re: Better compress Saves

Post by bobingabout » Mon Jan 08, 2018 10:06 am

Zeblote wrote:200kb/s? What year is this?
my god, that's lightning!

I think my upload speed is currently limited to 50Mbps, and I'm on fibre optic, but I remember back on my first ADSL, I had a whole 100kbps to work with on my upload. that's kilobits, so it's more like 10 kilobytes a second. The speed I had before upgrading to fibre was still hardcapped to 1Mbps, which more realistically was closer to 750kbps due to line quality, which is maybe 100 kilobytes a second upload on a good day. And this is STANDARD where I live, most people can't even get these kinds of high speeds. I only live just over a mile from the exchange.

Download speeds were 1Mbps when I first got ADSL, upgraded to 8Mbps, and eventually to 24Mbps when the line was upgraded to ADSL2, but line quality resulted in about 10Mbps.

so either way you look at it, 200 kbps peer to peer map transfer speed over the internet is LIGHTNING fast, no joke!

My Australian friend (Who lives out in the middle of nowhere) can only dream of the speeds I used to have! He probably gets that 200kbps DOWNLOAD on a good day.
Last edited by bobingabout on Tue Jan 09, 2018 9:00 am, edited 2 times in total.
Creator of Bob's mods. Expanding your gameplay since version 0.9.8.

Zeblote
Filter Inserter
Filter Inserter
Posts: 972
Joined: Fri Oct 31, 2014 11:55 am
Contact:

Re: Better compress Saves

Post by Zeblote » Mon Jan 08, 2018 8:03 pm

eradicator wrote:
Zeblote wrote:200kb/s? What year is this?
It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.
It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...

Koub
Global Moderator
Global Moderator
Posts: 4300
Joined: Fri May 30, 2014 8:54 am
Contact:

Re: Better compress Saves

Post by Koub » Mon Jan 08, 2018 8:06 pm

Zeblote wrote:
eradicator wrote:
Zeblote wrote:200kb/s? What year is this?
It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.
It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...
I'm sure there are more people in the world with bandwidth close to eradicator's than there are with yours. Try to be comprehensive, and remember you're lucky ^^.
Koub - Please consider English is not my native language.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Better compress Saves

Post by eradicator » Tue Jan 09, 2018 1:23 am

Zeblote wrote: It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...
It's also completely random. If i drive down the road a few hundred meters there's ads for FTTB. The shitty upload speeds are "traditional" here though, even the ppl with 100MB/s download only get 10 up, and the ratio gets worse for slower connections. Also remember that not everybody lives in the city center.
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

User avatar
bobingabout
Smart Inserter
Smart Inserter
Posts: 6350
Joined: Fri May 09, 2014 1:01 pm
Contact:

Re: Better compress Saves

Post by bobingabout » Tue Jan 09, 2018 9:01 am

Zeblote wrote:
eradicator wrote:
Zeblote wrote:200kb/s? What year is this?
It's 2018. And if my friend hosts on his home connections stuff is limited by his 50kb/s upload speed. Because ISPs here only give you the smallest possible upload they can get away with.
It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...
I don't even understand how that can be possible, where do you live?


Anyway, my input on the subject. autosaves, make 'em snappy. Manual saves, well, you've already broken from the game, put more time and effort into the compression, an extra couple of seconds processing the save shouldn't be an issue.
Creator of Bob's mods. Expanding your gameplay since version 0.9.8.

User avatar
olafthecat
Filter Inserter
Filter Inserter
Posts: 476
Joined: Fri Jul 07, 2017 4:37 pm

Re: Better compress Saves

Post by olafthecat » Tue Jan 09, 2018 1:12 pm

Sadly, my upload speed is in the Mb as well.
I use Plus Net, which is all right and have fibre optic and a decent router.
Come to think of it, HOW DID YOU GET THAT SANIC SPEED!? :o :shock: :?
I don't even understand how that can be possible, where do you live?
He/She probably either lives in Dubai or in the 22nd century :lol:
By the way, I think mine is around 30-40Mb on a good day.
Gonna start playing again with 0.16 build.
That's all.

Zavian
Smart Inserter
Smart Inserter
Posts: 1325
Joined: Thu Mar 02, 2017 2:57 am
Contact:

Re: Better compress Saves

Post by Zavian » Tue Jan 09, 2018 1:22 pm

My upload is around 400 kilobits/s, on a good day.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 2604
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Better compress Saves

Post by eradicator » Tue Jan 09, 2018 5:22 pm

bobingabout wrote: Anyway, my input on the subject. autosaves, make 'em snappy. Manual saves, well, you've already broken from the game, put more time and effort into the compression, an extra couple of seconds processing the save shouldn't be an issue.
I don't like the distinction between auto/manual save. I do agree with Rsed too that it's nice to have good compatability (i.e. deflate) though. Which is why i explicitly said i'd like this for mp joining only (because as far as i know the state sent to joining players is not saved to disk (permanently)?).
_______
Zavian wrote:It's almost like he's living on a different world... that upload is nearly 1000 times slower than what is available here...
[...]
My upload is around 400 kilobits/s, on a good day.
I never said anything about having a speed of less than 400 bits/s.
And just to clarify, i don't participate in ISP ad based number boosing, so all values i don't explicitly state as "bit" are in byte (per second).
______
And also we should stop derailing this thread into bandwidth bragging....
Author of: Hand Crank Generator, Screenshot Hotkey 2.0
Mod support languages: 日本語, Deutsch, English

Post Reply

Return to “Ideas and Suggestions”

Who is online

Users browsing this forum: Bing [Bot] and 7 guests