better compression algorithm for saves and mods

[ssilk]

I think a better compression comes AFTER non-blocking-saving for Windows.

I play on a Mac; Believe me: it’s such a relief to be not always interrupted by game saves. You can do really, really big worlds, that take 3 minutes to save (about 100000 squaretiles). A better compression will not nearly bring that increase in gameplay, even for multiplayer and maps transfers.

That this is quite difficult to implement on Windows compared with extending the compression is a different subject.

[Rseding91]

Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.

[ptx0]

It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.

i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.

catching up is when you need to process updates. map size can be independent of updates per second.

catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.

in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.

[Jap2.0]

stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.

Okay so that's why I asked

So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)

I realize that it can be useful in certain cases and I'm trying to establish when you want those cases to be in light of the tradeoffs.

If it's optional all around I think it's a perfectly fine idea.

[ptx0]

Okay so that's why I asked

So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)

it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.

[Jap2.0]

it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.

Yeah, I'm fine with that.

[Shadow_Man]

TL;DR

Add 7-zip archiver support to game engine

What ?

7-zip is a popular free archiver with a stronger compression ratio than ZIP. Add its support to the game, along with ZIP, for mods, save files and anything else that uses ZIP packaging.

Why ?

Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.

[SoShootMe]

Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.

For mods, graphics dominate the size and are already compressed, so 7-zip wouldn't offer much benefit.

For saved game files, there may be significant potential to reduce size, but at the expense of increased time and RAM to create them. Unlike mods, save performance is also important so there is a trade-off.

[DarkShadow44]

Regarding save compression we already have a suggestion.
zstd would be better than LZMA, but the advantage to zlib isn't really big.

For mods, I don't see the point either. I tried with the mods I have:

Code: Select all

for /d %X in (*) do "7z.exe" -mx=9 a "%X.7z" "%X\"

The size was 2.04 GB compared to 2.10 GB as zips. Hardly significant IMHO.

[mrvn]

I actually unpack all my mods because it makes the game start faster.

[ssilk]

merged with existing topic

[mrvn]

It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.

i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.

catching up is when you need to process updates. map size can be independent of updates per second.

catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.

in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.

You could also have different compression for multiplayer map save on join, autosave and manual save. Only the manual saves are kept and many people change the name every save so saving space there is a long time benefit. Save on join and autosave need to be fast and save on join should compress faster than the upload speed.

[DarkShadow44]

Regarding multiplayer we have have an assortment of ideas to make it download faster here: viewtopic.php?f=6&t=100476

[ptx0]

please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions

adding a compression algorithm is much simpler and more attainable than whatever that thread wants. I can't even understand that thread, it has too much speculation and people throwing out formulas without adequate explanations. no offense, but it's something the people responding in that thread do all the time. unreadable wall of text that makes many assumptions and Wube will likely never respond to.

[DarkShadow44]

please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions

You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.

adding a compression algorithm is much simpler and more attainable than whatever that thread wants.

Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?

[ptx0]

please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions

You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.

adding a compression algorithm is much simpler and more attainable than whatever that thread wants.

Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?

20% is better than 0% when your data plan has only 25GB a month available to you, so, yes.

if Wube implements ZSTD support it'll be even better, as they can create a dictionary that helps to compress their specific save data better.