Page 2 of 2
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 9:14 am
by ssilk
I think a better compression comes AFTER non-blocking-saving for Windows.
I play on a Mac; Believe me: it’s such a relief to be not always interrupted by game saves. You can do really, really big worlds, that take 3 minutes to save (about 100000 squaretiles). A better compression will not nearly bring that increase in gameplay, even for multiplayer and maps transfers.
That this is quite difficult to implement on Windows compared with extending the compression is a different subject.
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 4:56 pm
by Rseding91
Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 7:18 pm
by ptx0
Jap2.0 wrote: ↑Sat May 09, 2020 1:32 am
It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.
i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.
catching up is when you need to process updates. map size can be independent of updates per second.
catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.
in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 7:23 pm
by Jap2.0
ptx0 wrote: ↑Sat May 09, 2020 7:18 pm
stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.
Okay so that's why I asked
Jap2.0 wrote: ↑Sat May 09, 2020 1:32 am
So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)
I realize that it can be useful in certain cases and I'm trying to establish when you want those cases to be in light of the tradeoffs.
If it's optional all around I think it's a perfectly fine idea.
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 7:32 pm
by ptx0
Jap2.0 wrote: ↑Sat May 09, 2020 7:23 pm
Okay so that's why I asked
Jap2.0 wrote: ↑Sat May 09, 2020 1:32 am
So can you tell me exactly when you want this to be default, optional, and not available? (Obviously decompression should be available everywhere.)
it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.
Re: better compression algorithm for saves and mods
Posted: Sat May 09, 2020 7:36 pm
by Jap2.0
ptx0 wrote: ↑Sat May 09, 2020 7:32 pm
it shouldn't be available if the platform doesn't support it, really - otherwise like any tunable in graphics setting (as an example) that lets you enable options you shouldn't for performance reasons, let it be turned on even if it'll slow them down too much.
Yeah, I'm fine with that.
Add 7-zip support to game
Posted: Fri Nov 12, 2021 10:12 am
by Shadow_Man
TL;DR
Add 7-zip archiver support to game engine
What ?
7-zip is a popular free archiver with a stronger compression ratio than ZIP. Add its support to the game, along with ZIP, for mods, save files and anything else that uses ZIP packaging.
Why ?
Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.
Re: Add 7-zip support to game
Posted: Fri Nov 12, 2021 10:31 am
by SoShootMe
Shadow_Man wrote: ↑Fri Nov 12, 2021 10:12 am
Using 7-zip will save storage space on servers / local drives, reduce the time for downloading mods, loading saves.
For mods, graphics dominate the size and are already compressed, so 7-zip wouldn't offer much benefit.
For saved game files, there may be significant potential to reduce size, but at the expense of increased time and RAM to create them. Unlike mods, save performance is also important so there is a trade-off.
Re: Add 7-zip support to game
Posted: Fri Nov 12, 2021 10:52 am
by DarkShadow44
Regarding save compression we already
have a suggestion.
zstd would be better than LZMA, but the advantage to zlib isn't really big.
For mods, I don't see the point either. I tried with the mods I have:
Code: Select all
for /d %X in (*) do "7z.exe" -mx=9 a "%X.7z" "%X\"
The size was 2.04 GB compared to 2.10 GB as zips. Hardly significant IMHO.
Re: Add 7-zip support to game
Posted: Sat Nov 13, 2021 1:04 am
by mrvn
I actually unpack all my mods because it makes the game start faster.
Re: better compression algorithm for saves and mods
Posted: Sat Nov 13, 2021 5:09 am
by ssilk
merged with existing topic
Re: better compression algorithm for saves and mods
Posted: Sat Nov 13, 2021 9:06 am
by mrvn
ptx0 wrote: ↑Sat May 09, 2020 7:18 pm
Jap2.0 wrote: ↑Sat May 09, 2020 1:32 am
It's tradeoffs all the way down. How much time is it worth? A minute? Five? Ten? An hour? There's no way to say that one setting will be better or worse for everyone. Keep in mind that the longer you wait to load the save, the more catch-up data that has to be sent over the network.
i'm explaining exactly when this is useful and you either don't want to or can't understand what i'm getting at. stop trying to apply it to everybody, i never asked for it to be the default option, i've said several times it can be optional.
catching up is when you need to process updates. map size can be independent of updates per second.
catching up data is no problem at 60KiB/s for a few moments but the map download will run at 5MiB/s and sustain for several minutes. hence, we really want better map compression.
in cases where several users are joining at once it will even batch the save for all of those so the server doesn't continually save for *every* single connecting player. at least, this is how i understand it to work from the 500 player game with KoS.
You could also have different compression for multiplayer map save on join, autosave and manual save. Only the manual saves are kept and many people change the name every save so saving space there is a long time benefit. Save on join and autosave need to be fast and save on join should compress faster than the upload speed.
Re: better compression algorithm for saves and mods
Posted: Sat Nov 13, 2021 2:24 pm
by DarkShadow44
Regarding multiplayer we have have an assortment of ideas to make it download faster here:
viewtopic.php?f=6&t=100476
Re: better compression algorithm for saves and mods
Posted: Sat Nov 20, 2021 3:14 pm
by ptx0
please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions
adding a compression algorithm is much simpler and more attainable than
whatever that thread wants. I can't even understand that thread, it has too much speculation and people throwing out formulas without adequate explanations. no offense, but it's something the people responding in that thread do
all the time. unreadable wall of text that makes many assumptions and Wube will likely never respond to.
Re: better compression algorithm for saves and mods
Posted: Sat Nov 20, 2021 5:12 pm
by DarkShadow44
ptx0 wrote: ↑Sat Nov 20, 2021 3:14 pm
please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions
You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.
ptx0 wrote: ↑Sat Nov 20, 2021 3:14 pm
adding a compression algorithm is much simpler and more attainable than
whatever that thread wants.
Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?
Re: better compression algorithm for saves and mods
Posted: Sat Nov 20, 2021 6:08 pm
by ptx0
DarkShadow44 wrote: ↑Sat Nov 20, 2021 5:12 pm
ptx0 wrote: ↑Sat Nov 20, 2021 3:14 pm
please don't spam this suggestion with a link to a thread that contains a lot of unrelated suggestions
You complained about big downloads for MP maps, that thread deals with that as well. Hardly unrelated.
ptx0 wrote: ↑Sat Nov 20, 2021 3:14 pm
adding a compression algorithm is much simpler and more attainable than
whatever that thread wants.
Agreed. Don't get me wrong, I'm all for adding zstd support. Although, in the best case (read: 5min compress time) I only get a save file reduction of 20% for my 270MB save. For more realistic compression times, I got reductions of < 10%.
Is there really such a big difference for you?
20% is better than 0% when your data plan has only 25GB a month available to you, so, yes.
if Wube implements ZSTD support it'll be even better, as they can create a dictionary that helps to compress their specific save data better.