Rapid save file size growth.

Anything that prevents you from playing the game properly. Do you have issues playing for the game, downloading it or successfully running it on your computer? Let us know here.
Post Reply
morritze
Burner Inserter
Burner Inserter
Posts: 8
Joined: Wed Nov 06, 2019 10:33 am
Contact:

Rapid save file size growth.

Post by morritze »

Hello,

I played the game on a headless server, which i keep updated to latest experimental. On 15th Nov, my savegame had ~20 megs. This is a very reasonable file size. Then, some patches happened ( not sure if they are related, but anyway.. )

On 4th Nov, the file size was 70 megs. A lot, but still playable.

Yesterday, it saved a 140 MB behemoth. I haven't been exploring map or adding entities all that much. A little, but nothing what would explain such a growth. I think the greatest "click" was a ~20k concrete paving.

At the moment, the game is slowly becoming unplayable. Joining takes ages due to save download, despite 100mbit connection. Autosaves take ages despite SSD. I would accept if I did something that caused the growth, but I would like to know what was it.

Here are the saves : http://www.xpear.net/factorio/

Please disregard the timestamp on the web server.

used mods are : Blueprint Flip & turn, Infinitezoom, max rate calculator.

Bilka
Factorio Staff
Factorio Staff
Posts: 3123
Joined: Sat Aug 13, 2016 9:20 am
Contact:

Re: Rapid save file size growth.

Post by Bilka »

Saving the biggest map shows
27.584 Info Serialiser.cpp:37: surface.commander 321407kB
Compared to the smallest save, the biggest difference is that you expanded to the "mainland" on the left, which leads to enemies trying to pathfind to you over the land bridge that you built.
I'm an admin over at https://wiki.factorio.com. Feel free to contact me if there's anything wrong (or right) with it.

morritze
Burner Inserter
Burner Inserter
Posts: 8
Joined: Wed Nov 06, 2019 10:33 am
Contact:

Re: Rapid save file size growth.

Post by morritze »

Thanks for the reply.

Pathfinding state is recorded in the save and has ~ 300 megs uncompressed ?

Does it mean, that if i get rid of the biters within the pollution cloud, it will reduce?

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Rapid save file size growth.

Post by Rseding91 »

Yes it's recorded in the save. Everything the game does is recorded in the save so you can save/load and keep playing :)

The reason it's so huge is because you've got an island with a 1 pixel wide path across to the biters. They can't path to that but they have ot try pathing around the entire lake before it will fail the search and that's just super super slow.

My recommendation is 1 of the following:

1. Disable biters if you don't want to fight them. Islands is essentially disabling them anyway - except you pay the huge performance hit of them trying to attack you when they simply can't.

2. Don't be on an island and actually fight the biters.
If you want to get ahold of me I'm almost always on Discord.

morritze
Burner Inserter
Burner Inserter
Posts: 8
Joined: Wed Nov 06, 2019 10:33 am
Contact:

Re: Rapid save file size growth.

Post by morritze »

I wouldn't expect, that the pathfinding state is on the list of things to save, since it should be somewhat deterministic and restarting the calculation after load should eventually deliver the same result as before quitting the game. The question is, of course, how long does it take. With the recent developments in pathfinding, i wouldn't expect it to be that long though..

That said, I understand this is a corner case and i ran into the problem because of my settings "abuse".

Thanks for the explanations. I'll go with option 2 :)

Zavian
Smart Inserter
Smart Inserter
Posts: 1641
Joined: Thu Mar 02, 2017 2:57 am
Contact:

Re: Rapid save file size growth.

Post by Zavian »

morritze wrote:
Thu Nov 07, 2019 8:04 am
I wouldn't expect, that the pathfinding state is on the list of things to save, since it should be somewhat deterministic and restarting the calculation after load should eventually deliver the same result as before quitting the game. The question is, of course, how long does it take. With the recent developments in pathfinding, i wouldn't expect it to be that long though..

I'm not a dev, and haven't seen the Factorio code, but the way multiplayer works requires that pathfinding is 100% reproducible and deterministic. However that doesn't mean that the path generated now is identical to the path generated 100 ticks ago, especially if players/bots are placing landfill, or destroying potential obstacles (eg rocks/trees/spawners etc). That means that if the pathfinder uses the path cache, then either the cache needs to be saved as part of the save, or it needs to be invalidated whenever anything gets built/destroyed/changed that might result in a different path on another computer that doesn't have the same set of paths cached. (eg. a multiplayer client that has just connected and downloaded the save).

Also consider the performance optimisation of allowing X path cache lookups per tick, but limiting attempts to calculate new paths to 1 per tick. Again that requires that all machines have exactly the same set of paths cached. (I don't know whether Factorio does this, but I think it likely that the devs have implemented some kind of rate limiting for path finding).

DaleStan
Filter Inserter
Filter Inserter
Posts: 368
Joined: Mon Jul 09, 2018 2:40 am
Contact:

Re: Rapid save file size growth.

Post by DaleStan »

morritze wrote:
Thu Nov 07, 2019 8:04 am
I wouldn't expect, that the pathfinding state is on the list of things to save, since it should be somewhat deterministic and restarting the calculation after load should eventually deliver the same result as before quitting the game.
"eventually" is the issue here. Even if the map doesn't change, Factorio's determinism requires that the pathfinder deliver the same result at the same time. If the pathfinding is spread across even just two ticks, the progress from the first tick must be saved so that the pathfinding completes (and the biters start moving) on the same tick for everyone, even if they happened to join the game between the first pathfinding tick and the second.

Post Reply

Return to “Technical Help”