Page 1 of 1

Game Save File and -Speed Related Suggestions

Posted: Sat Mar 12, 2016 1:15 pm
by ssilk
Tags: Snapshot, Compression, Serialisation


Reated Collections

viewtopic.php?f=80&t=327 World Names (Configurable additional filename chars)

Save Game Speed

Big topic, big todo. The real solution is scribbled down in some thread (TODO: find it), where the game is saved in background.
The basic idea to make the save unremarkable fast is to use mechanisms like in virtual machines.
These links probably explain, how that works:
https://pubs.vmware.com/vsphere-4-esx-v ... shots.html
https://kb.vmware.com/selfservice/micro ... Id=1015180
Image
Image

viewtopic.php?f=6&t=25640 Save in Background
viewtopic.php?f=6&t=31279 Ideas to increase multiplayer map transfer speed
viewtopic.php?f=6&t=32355 Ideas for some Loading Changes
viewtopic.php?f=6&t=32651 Multiplayer autosave experience
viewtopic.php?f=6&t=33175 Autosave in battle
viewtopic.php?f=6&t=34273 Use ZSTD for savegame compression (much faster, smaller)
viewtopic.php?f=6&t=47871 More transparent autosaving



World names and Filename restrictions

There is basically the idea to have one world and "versions" of that world. Theoretically you can use the replay save then to jump to every time in that world.

viewtopic.php?f=6&t=327 Configurable additional filename character whitelist
viewtopic.php?f=66&t=1593&p=11248&hilit ... %2A#p11245 This was discussed, when the map exchange string was introduced.
viewtopic.php?f=6&t=1712 Allow space in save file names
viewtopic.php?f=6&t=2977&hilit=world+name%2A Would be useful, for many stupid things
viewtopic.php?f=6&t=3344 Minecraft-like World Screen
viewtopic.php?f=6&t=7035 Save game names
viewtopic.php?f=6&t=7050 World Names - New worlds have a (default) name
viewtopic.php?f=6&t=26663 Organize saves into folders by game/world

Autosave / Last Saved Name

viewtopic.php?f=6&t=12733 Improved autosaving system.
viewtopic.php?f=6&t=16388 Change Default Auto Save 2min to 15min.
viewtopic.php?f=6&t=20795 Fast save via short-cut
viewtopic.php?f=6&t=28747 Last loaded file ?
viewtopic.php?f=6&t=28952 More Saveslots
viewtopic.php?f=6&t=33706 [Multiplayer] Autosave Tweaks
viewtopic.php?f=6&t=3829 Smarter autosaving (ie: Not in combat)


Save Confirmation

viewtopic.php?f=6&t=3888 Better Savegame management. (Already lost a 50h savegame)
viewtopic.php?f=6&t=8477 Confirmation dialogue for deleting saves
viewtopic.php?f=6&t=17121 Save confirmation
viewtopic.php?f=6&t=21363 Save overwriting confirmation
viewtopic.php?f=6&t=29206 Confirmation before overwriting!

Related

viewtopic.php?f=6&t=2038 Hardmode

viewtopic.php?f=80&t=21367 Game Save File ans -Speed Related Suggestions

Re: Game Save File and -Speed Related Suggestions

Posted: Sun Jun 24, 2018 10:56 am
by Dixi
Any progress on this topic?

I think save files are quite large compared to possible amount of info stored inside.
If I start exploring around and shooting manually from artillery my safe file grows to 200+Mb very rapidly. This size takes a lot of time to autosave even on fast SSD. I think biters data takes too much space in file.

Separate thread autosaving or more space optimized saves could be nice to see.

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 12:40 am
by Zavian
Well the more you build and explore the more data there is to save. (Shooting artillery off the edge of the explored region is also revealing more of the map, so it is also exploring). Every assembler, inserter or belt you build, and every map tile and biter your explorations reveal is more data that needs to be saved.

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 4:34 pm
by Dixi
Zavian wrote:Well the more you build and explore the more data there is to save.
But I'm speaking about data, probably related to biters.

Recent sample.
I have two autosaves, one 18:04 - 31Mb size, and next 18:12 22Mb size. That's extra 9Mb, or 40% more of save size. I assume bigger one has some data related to biters, they probably spawned a new base in artillery controlled range, that got shot, after that they came to laser fence (to complain, probably ;-) and got exterminated.
But those one or several groups of biters, during their existence, added about 9Mb of data to save file! That's really too much. Because if I start exploring biter occupied map a lot of chunks with biters become visible and many of biters will be "just hanging around their base" and save file will grow too fast to hardly usable size of 200+Mb.

I think that excessive save size is caused by biters. My factory, energy consumption about 1.5GW, normally fits in about 22Mb save. A save with a big factory from internet, 3.8GW, no biters, widely explored map is about 68Mb. As I said before, it took me about 2 hours of railway and turrets building, and shooting around to get to 210Mb save, that cause a lot of delays on autosaves.

Maybe it's not common for everyone, and I have some rare glitch in my game?

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 5:09 pm
by Zavian
So what are you suggesting the game do? Start deleting biters or belts or assemblers or explored chunks or something else? Because I don't see how they can make the saves smaller without saving less data. That means something would need to be deleted.

Exploring new chunks is a major cause of more data to be saved. (And I have saves a lot larger than 20MB).

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 8:51 pm
by Dixi
Zavian wrote:So what are you suggesting the game do? Start deleting biters ...
No. But I'm sure that save file data is very ineffectively optimized and a lot of unnecessary data saved. Because almost none of games has 100+MB saves.

Lets speak about same biters. Biter is a mob. X,Y, mob kind, HP, link to current strategy or group, etc., I'd say 100-200 bytes is enough to have all info about it. If we have 1000 biters that will be 200Kb or less. But megabytes per one or several groups of biters, I suppose about hundred biters total is reserved. That's probably pathfinding data, individual for each mob. It can easily be discarded and recalculated on load. This way save will be much smaller and faster, while load will take a little more time - that is less important, since we load a game not so often.

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 9:46 pm
by Zavian
So I loaded a 96 MB save, and pressed f5 to bring up the debug info 2138356 entities, with 228600 active entities, 67600 chunks (69 million tiles). I don't know any other game that can manage that many active entities and still run at a solid 60 fps/ups on my 7 year old hardware. In general factorio is well optimised. (If it wasn't I'd be lucky to get 5 fps).

You were suggesting that each biter should be less than 100-200 bytes. If each tile is 1 byte, then that leaves about 13 bytes per entity. Probably less since I don't think decoratives are counted as entities, and they are also stored in the savefile. (Those figures are for the compressed save, but even if I uncompress the save, the data level.dat is only 186MB, at 1 byte per tile that leaves each entity about 54 bytes. Again probably less after decoratives are stored).

Re: Game Save File and -Speed Related Suggestions

Posted: Mon Jun 25, 2018 10:10 pm
by Jap2.0
You also have to consider that it must be exactly the same upon save and load or else you have desyncs in multiplayer.

Re: Game Save File and -Speed Related Suggestions

Posted: Tue Mar 26, 2019 2:41 am
by mrudat
Thinking about it a little; could we stream player interaction to disk (similar to how it is streamed over the network when catching up to a multiplayer server) to fill in the gaps between autosaves? Presuming you're running the same game version (which immediately after a crash, you would be), you could catch the game up to the tick immediately before the crash happened.

Obviously this wouldn't work very well if you're up against the UPS wall, though I believe that even megabases run faster if you don't need to also run the GUI?

If you write out the tick in which the crash occurred you could run the game right up until the tick before the crash, and create a debugging save that crashes on the first tick it runs after loading.