better compression algorithm for saves and mods

Post your ideas and suggestions how to improve the game.

Moderator: ickputzdirwech

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12406
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ssilk »

I think a better compression comes AFTER non-blocking-saving for Windows.

I play on a Mac; Believe me: it’s such a relief to be not always interrupted by game saves. You can do really, really big worlds, that take 3 minutes to save (about 100000 squaretiles). A better compression will not nearly bring that increase in gameplay, even for multiplayer and maps transfers.

That this is quite difficult to implement on Windows compared with extending the compression is a different subject.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

Rseding91
Factorio Staff
Factorio Staff
Posts: 11972
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: better compression algorithm for saves and mods

Post by Rseding91 »

Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.
If you want to get ahold of me I'm almost always on Discord.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1112
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

Jap2.0 wrote:
Sat May 09, 2020 1:32 am

It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.
i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.

catching up is when you need to process updates. map size can be independent of updates per second.

catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.

in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

Jap2.0
Smart Inserter
Smart Inserter
Posts: 2333
Joined: Tue Jun 20, 2017 12:02 am
Contact:

Re: better compression algorithm for saves and mods

Post by Jap2.0 »

ptx0 wrote:
Sat May 09, 2020 7:18 pm
stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.
Okay so that's why I asked
Jap2.0 wrote:
Sat May 09, 2020 1:32 am
So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)
I realize that it can be useful in certain cases and I'm trying to establish when you want those cases to be in light of the tradeoffs.

If it's optional all around I think it's a perfectly fine idea.
There are 10 types of people: those who get this joke and those who don't.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1112
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

Jap2.0 wrote:
Sat May 09, 2020 7:23 pm
Okay so that's why I asked
Jap2.0 wrote:
Sat May 09, 2020 1:32 am
So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)
it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

Jap2.0
Smart Inserter
Smart Inserter
Posts: 2333
Joined: Tue Jun 20, 2017 12:02 am
Contact:

Re: better compression algorithm for saves and mods

Post by Jap2.0 »

ptx0 wrote:
Sat May 09, 2020 7:32 pm
it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.
Yeah, I'm fine with that.
There are 10 types of people: those who get this joke and those who don't.

Shadow_Man
Inserter
Inserter
Posts: 42
Joined: Fri Mar 02, 2018 2:55 pm
Contact:

Add 7-zip support to game

Post by Shadow_Man »

TL;DR
Add 7-zip archiver support to game engine
What ?
7-zip is a popular free archiver with a stronger compression ratio than ZIP. Add its support to the game, along with ZIP, for mods, save files and anything else that uses ZIP packaging.
Why ?
Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.

SoShootMe
Fast Inserter
Fast Inserter
Posts: 232
Joined: Mon Aug 03, 2020 4:16 pm
Contact:

Re: Add 7-zip support to game

Post by SoShootMe »

Shadow_Man wrote:
Fri Nov 12, 2021 10:12 am
Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.
For mods, graphics dominate the size and are already compressed, so 7-zip wouldn't offer much benefit.

For saved game files, there may be significant potential to reduce size, but at the expense of increased time and RAM to create them. Unlike mods, save performance is also important so there is a trade-off.

DarkShadow44
Fast Inserter
Fast Inserter
Posts: 141
Joined: Thu Jun 01, 2017 12:05 pm
Contact:

Re: Add 7-zip support to game

Post by DarkShadow44 »

Regarding save compression we already have a suggestion.
zstd would be better than LZMA, but the advantage to zlib isn't really big.

For mods, I don't see the point either. I tried with the mods I have:

Code: Select all

for /d %X in (*) do "7z.exe" -mx=9 a "%X.7z" "%X\"
The size was 2.04 GB compared to 2.10 GB as zips. Hardly significant IMHO.

mrvn
Smart Inserter
Smart Inserter
Posts: 5073
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: Add 7-zip support to game

Post by mrvn »

I actually unpack all my mods because it makes the game start faster.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12406
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ssilk »

merged with existing topic
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

mrvn
Smart Inserter
Smart Inserter
Posts: 5073
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: better compression algorithm for saves and mods

Post by mrvn »

ptx0 wrote:
Sat May 09, 2020 7:18 pm
Jap2.0 wrote:
Sat May 09, 2020 1:32 am

It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.
i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.

catching up is when you need to process updates. map size can be independent of updates per second.

catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.

in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.
You could also have different compression for multiplayer map save on join, autosave and manual save. Only the manual saves are kept and many people change the name every save so saving space there is a long time benefit. Save on join and autosave need to be fast and save on join should compress faster than the upload speed.

DarkShadow44
Fast Inserter
Fast Inserter
Posts: 141
Joined: Thu Jun 01, 2017 12:05 pm
Contact:

Re: better compression algorithm for saves and mods

Post by DarkShadow44 »

Regarding multiplayer we have have an assortment of ideas to make it download faster here: viewtopic.php?f=6&t=100476

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1112
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions :roll:

adding a compression algorithm is much simpler and more attainable than whatever that thread wants. I can't even understand that thread, it has too much speculation and people throwing out formulas without adequate explanations. no offense, but it's something the people responding in that thread do all the time. unreadable wall of text that makes many assumptions and Wube will likely never respond to.
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

DarkShadow44
Fast Inserter
Fast Inserter
Posts: 141
Joined: Thu Jun 01, 2017 12:05 pm
Contact:

Re: better compression algorithm for saves and mods

Post by DarkShadow44 »

ptx0 wrote:
Sat Nov 20, 2021 3:14 pm
please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions :roll:
You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.
ptx0 wrote:
Sat Nov 20, 2021 3:14 pm
adding a compression algorithm is much simpler and more attainable than whatever that thread wants.
Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1112
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

DarkShadow44 wrote:
Sat Nov 20, 2021 5:12 pm
ptx0 wrote:
Sat Nov 20, 2021 3:14 pm
please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions :roll:
You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.
ptx0 wrote:
Sat Nov 20, 2021 3:14 pm
adding a compression algorithm is much simpler and more attainable than whatever that thread wants.
Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?
20% is better than 0% when your data plan has only 25GB a month available to you, so, yes.

if Wube implements ZSTD support it'll be even better, as they can create a dictionary that helps to compress their specific save data better.
My Mods - Fish Per Minute base size metric - Use your crashed spaceship as a belt balancer?
• • •
Base: Bob's @ 1 Million SPM
• • •
Linear search and overflows are indicative of sloppy coding practices.

Post Reply

Return to “Ideas and Suggestions”