Validate Saves

Ideas, that are too old (too much things changed) or won't be implemented cause of some reasons or if there are obvious better suggestions.

Moderator: ickputzdirwech

KeepGridButton
Burner Inserter
Burner Inserter
Posts: 11
Joined: Sat Nov 14, 2020 2:39 am
Contact:

Validate Saves

Post by KeepGridButton »

TL;DR
Consider detecting corruption by a validating a save by running it through the load routine, in order to prevent multiple saves from being corrupted..
What ?
After completing a save operation, allow gaming to continue, but in a background thread run that save through the load routine to validate there are no errors loading the save. If there is an error, warn the user the last save is potentially corrupted, and additional saves may also be corrupted. Recommend exiting the game immediately before additional saves are corrupted.

Once detected, consider performing a backup of all 3 autosaves to new filenames, so even if the user doesn't understands and keeps playing corrupting all their autosaves, they'll have the backups to go to.
Why ?
After experiencing a save corruption issue I found several other threads and an entire forum called "0 / 1" that seem to be a variety of what seems like memory corruption that resulted in a bad save. In many cases multiple saves have the exact same corruption. I assume that some memory corruption occurs, and then it is written to each save subsequently during that gaming session until all the autosaves and the player's manual save contain the same issue. If it were disk level bit rot I wouldn't expect to see that pattern.

Thus the next time the player loads the game, they find none of their saves load, and it requires someone with detailed knowledge to repair the save. So even someone who rotates through several saves might find in a single gaming session they've corrupted all of their saves unknowingly until the next time they play.

By performing a load routine validation of the save, this issue will be detected sooner(fail fast strategy) and assuming the user follows advice and exits, it prevents additional saves from being overwritten, allowing the user to go back to a previous save or autosave.

If there's something I'm misunderstanding about how this happens at a technical level, which invalidates my suggestion, then I would say in general try to determine if there's a fail fast strategy that would catch the issue sooner. Failing that, perhaps you could come up with a backup strategy for saves that makes a backup copy of a save to a subfolder when it successfully loads. This way you'd have at most one extra copy of a particular validated "good" save. Other backup strategies I considered could potentially overwrite good saves with bad ones. Overwriting the backup of the same name after a successful load means you'd have no more backups than you have saves, so it won't grow out of control or need to be managed. This would mitigate the need for intervention from technical staff to fix saves manually. Sorry if this violates the "multiple suggestions" rule.
You should be more careful, don't overwrite all your saves in one session. Make backups
I anticipate some people are going to adopt a "you should be more careful" attitude. I personally ping pong between two manual saves so that I've got an extra in case the game crashes or I get a power outage during a save operation, plus there's the autosave rotation. There's also steam cloud backups, so even if my entire computer went up in flames, I've still got access to my saves. So it feels like there's a couple levels of protection there already. I would have never thought that memory corruption would survive through multiple saves and propagate through all of them without something crashing. This isn't a criticism or complaint, I'm just saying an average user being cautious already in there save strategy probably wouldn't understand that a playthough they've put alot of time into could be wiped out in this way.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

Hm. I don’t think this a good strategy. I would use it only, if I (as dev) would search for a specific bug. Because this occurs so seldom this extra loading needs a) too much resources and b) doesn’t even guarantee that if it loads it is able to run (happened to me much more often).

Instead I would try to fix those bugs, that make this corrupt save happen and try to address the issues deeper into the architecture of the internal representation to make them more robust.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

azesmbog
Fast Inserter
Fast Inserter
Posts: 176
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Validate Saves

Post by azesmbog »

It rarely happens, or not rarely, but I have only encountered this problem twice recently. Of course, you can say that I have problems in the computer hardware, and these are my problems. And if still not ??
Saving with subsequent verification is - in my case - a very, very bad decision.
For me, saving and subsequent loading takes about 20 minutes, total 20 + 20 = 40 minutes: ((
I can do this trick every two or three days, but not every hour and a half.
Maybe all the same something to fix at the conservatory? (c)

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

It doesn’t matter what causes the error, if it is a software (Factorio inside) or a hardware or OS problem (outside Factorio). In any case if verification fails: what should be done?

As here:
If there is an error, warn the user the last save is potentially corrupted, and additional saves may also be corrupted. Recommend exiting the game immediately before additional saves are corrupted.
What if this doesn’t help? What if the error still occurs? This is a software-anti-pattern: let the user know there might be a problem and let it handle it him/herself. Gnaaaa, I learned that the hard way, that this isn’t gonna be working. It makes things worse!

What works is, if you know how to fix this problem and if you exactly know how to fix it, you can program a solution. (as already said as a way to find the bugs, this is a possibility, but then it is a clear message for the user: there was a failure, we have logged it)
loading takes about 20 minutes,
Hu? I have saves with 200+ MB and they take about two minutes for loading. And that are large worlds, where I need about 4 minutes to travel with 300km/h train from one edge to the other. How big are those saves? :shock:
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

azesmbog
Fast Inserter
Fast Inserter
Posts: 176
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Validate Saves

Post by azesmbog »

ssilk wrote:
Wed Jul 07, 2021 7:05 am
And that are large worlds, where I need about 4 minutes to travel with 300km/h train from one edge to the other. How big are those saves? :shock:
These are your small worlds :)
My save world takes ~ 3 GB, saves ~ 18 minutes on a fast SSD drive.
from the respawn point to the last point I have to go for many many hours, with 6-7 intermediate refueling with nuclear fuel - this is a little more than 2,500,000 cells on the map :) or 2,500 km?
At the same time, the UPS is still within 40.
For three years I have not had such preservation problems.
The hardware has not changed for the last year and a half, and the software has not changed or been added either, this computer is designed for only one game 24 hours a day \ 365 days a year :)

User avatar
boskid
Factorio Staff
Factorio Staff
Posts: 1565
Joined: Thu Dec 14, 2017 6:56 pm
Contact:

Re: Validate Saves

Post by boskid »

Idea of save validation by loading it directly after saving has one extreme major flaw: to verify the save is consistent, it has to be fully loaded including allocation of all the internal structures like chunks, entitites and be properly setup as some of the consistency checks work after the full load-and-setup. This basically means memory usage would increase by a factor of two. On a small save files it could be done but also smaller saves are less likely to undergo a random bitflip and the loss of such save hurts less. Its main value would be with large saves and this is exactly the case where it cannot be used due to doubling of memory requirements. I can safely throw this idea to "outdated/not implemented".

coppercoil
Filter Inserter
Filter Inserter
Posts: 348
Joined: Tue Jun 26, 2018 10:14 am
Contact:

Re: Validate Saves

Post by coppercoil »

boskid wrote:
Wed Jul 07, 2021 9:19 am
Idea of save validation by loading it directly after saving has one extreme major flaw[..]
There are players who have no memory shortage issues... Why not to add an option?

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5201
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Validate Saves

Post by eradicator »

coppercoil wrote:
Wed Jul 07, 2021 10:57 am
boskid wrote:
Wed Jul 07, 2021 9:19 am
Idea of save validation by loading it directly after saving has one extreme major flaw[..]
There are players who have no memory shortage issues... Why not to add an option?
Sounds like it would be the same as a small script that starts a headless server that loads the save and then exits?
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

KeepGridButton
Burner Inserter
Burner Inserter
Posts: 11
Joined: Sat Nov 14, 2020 2:39 am
Contact:

Re: Validate Saves

Post by KeepGridButton »

ssilk wrote:
Wed Jul 07, 2021 5:16 am
Hm. I don’t think this a good strategy. I would use it only, if I (as dev) would search for a specific bug. Because this occurs so seldom this extra loading needs a) too much resources and b) doesn’t even guarantee that if it loads it is able to run (happened to me much more often).

Instead I would try to fix those bugs, that make this corrupt save happen and try to address the issues deeper into the architecture of the internal representation to make them more robust.
Responses from staff on these issues include comments like "bit flip from cosmic rays". That sounds like they're saying its not a bug that can be fixed. I wouldn't be surprised if eventually we're going to get to a point that staff are no longer going to fix saves for us.

Doing a backup to a subdirectory after a successful load to a subdirectory should be cheap to implement and pretty cheap performance wise. You'd never have more backups than you have saves so it'd not be a big cost in space either.

Yeh you should identify and fix the root cause, but people shouldn't lose complete playthroughs everytime they encounter a bug. You're post 1.0, bugs shouldn't be this destructive IMO.

Qon
Smart Inserter
Smart Inserter
Posts: 1633
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Validate Saves

Post by Qon »

azesmbog wrote:
Wed Jul 07, 2021 8:32 am
These are your small worlds :)
My save world takes ~ 3 GB, saves ~ 18 minutes on a fast SSD drive.
from the respawn point to the last point I have to go for many many hours, with 6-7 intermediate refueling with nuclear fuel - this is a little more than 2,500,000 cells on the map :) or 2,500 km?
At the same time, the UPS is still within 40.
For three years I have not had such preservation problems.
The hardware has not changed for the last year and a half, and the software has not changed or been added either, this computer is designed for only one game 24 hours a day \ 365 days a year :)
I'm impressed. And envious, I also want a machine that just runs a save of Factorio every second for years!
Do you have a thread documenting your world? How big is it? What's your production? What does it look like and what does it do now? Purpose?

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

3 GB :shock: Wow :lol: as already said: make a video, show your creation, I’m sure many people are interested into that.

Well I have an idea, because with that size I can understand that a verify is helpful (*):
use the headless version of Factorio to load the file in background. Could also be done on another computer (loading the save over mounted network devices: check if there is a newer file than the last check, try to load, on error display alarm).

I’m sure if you ask, you find people in other boards that help to write such a script, because it’s highly dependent which platform you use and other technical dependencies.

(*) this size meets the edge of reliability of computers quite hard. Some cosmic ray can cause small bit-changes - for example. The quality/reliability of the computer hardware plays a much more important role than normally to play this without fail, because the chance that something happens rises exponentially with the size.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

azesmbog
Fast Inserter
Fast Inserter
Posts: 176
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Validate Saves

Post by azesmbog »

Qon wrote:
Wed Jul 07, 2021 9:51 pm

Do you have a thread documenting your world?
See here:
viewtopic.php?f=204&t=73328

Video would be nice, of course, all the hands would not reach, capture a video and make a tour of my sandbox.
Few probably have seen a 45 GW compact :)) nuclear power plant, and even the epic video of the explosion of this power plant is generally pleasing to the eyes) And not on every computer my base will start at all, even though it is pure vanilla.
On saves, the least developers can do is increase the maximum time between autosaves. Now, for some reason, only 95 minutes. This is not enough for me. In five hours of play, these saves will overwrite themselves, and it's good if they don't make an error.
If, for example, I had 8 hours between saves, then the rewriting would be in a day, and I would have much more time to think, check the recording and make a decision. Otherwise, I have to make a separate backup every two or three days, always with verification. Sometimes I forget to do it like the last time, and I had to go back 10 days :)

Koub
Global Moderator
Global Moderator
Posts: 6542
Joined: Fri May 30, 2014 8:54 am
Contact:

Re: Validate Saves

Post by Koub »

https://xkcd.com/378/
At some point, if you fear these kind of things, you should use ECC RAM, which will take you from a significant daily bit flip probability to 1 in a billion-ish bit flip probability.
Koub - Please consider English is not my native language.

User avatar
Nosferatu
Fast Inserter
Fast Inserter
Posts: 142
Joined: Fri Jan 20, 2017 4:48 pm
Contact:

Re: Validate Saves

Post by Nosferatu »

I was about to make a similar suggestion when I saw that op has already proposed something similar:
Failing that, perhaps you could come up with a backup strategy for saves that makes a backup copy of a save to a subfolder when it successfully loads. This way you'd have at most one extra copy of a particular validated "good" save.
This line of solutions has not been discarded yet. Correct?
My solution would have been:
Don't overwrite the last loaded save.
If the game trys to save on the file that has loaded the current game:
Back it up first. Then overwrite it.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

But should this be part of the game?

For example: on macOS I would say: you’re crazy! I have time machine backup, why should the game backup, too? Makes no sense.

A good backup means in any case, that I use a backup-strategy that fits my needs, and that is always part of my whole concept, how I make sure, that I can work, without loosing my work. It’s not part of one app to make that sure, because it makes no sense, to do that over and over for each app.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

User avatar
Nosferatu
Fast Inserter
Fast Inserter
Posts: 142
Joined: Fri Jan 20, 2017 4:48 pm
Contact:

Re: Validate Saves

Post by Nosferatu »

The average windows user has no daily backup system for his game files.
And why are there 3 autosaves? One should be enough ;)

You can see that this problem pops up repeatedly on the forums. And when it hits it usually destroyers 100s of game hours.
This is a very easy solution. So why not simply do it?

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

Nosferatu wrote:
Sat Jul 10, 2021 10:52 am
The average windows user has no daily backup system for his game files.
That’s also true for Linux users. But is that the problem of the game??
Windows has no useful backup. Users don’t want to invest in a backup medium. Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?

Sorry, I wanted to avoid ranting about missing backup, but a game cannot be misused to address that you need to have a backup that fits your needs.
And why are there 3 autosaves? One should be enough ;)
The autosave makes much sense and their target is completely different.
This is a very easy solution. So why not simply do it?
Well, there are lots of ideas about that. See

viewtopic.php?f=80&t=327 World Names (Configurable additional filename chars and Versioning)

viewtopic.php?f=80&t=21367 Game Save File and -Speed Related Suggestions

Most of them are very old.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

azesmbog
Fast Inserter
Fast Inserter
Posts: 176
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Validate Saves

Post by azesmbog »

ssilk wrote:
Sun Jul 11, 2021 8:28 am
Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?
And yet the problem is in the game.
let me be misled)
How do you suggest I make 3GB backups?
At each autosave in the background after it, overwrite the save file in a separate storage ??
The same garbage will be, without checking the subsequent download.
I do not argue at all, these are rhetorical questions, and the developers know better what and how to do.
But again, the last three autosaves are not loaded :))))))))))
Moreover, all three have different tile numbers.
Well, I had a verified save 18 hours ago. Well, I will build anew :) Zen, cho
And yes, 64GB of storage is a must.
but.
But ordinary motherboards do not support server memory with parity, and even with the declared support, I could not get it to work. These are two. Thirdly, the register memory is very slow in terms of speed characteristics. 3600 - I certainly did not see it in the price lists, but at 2400 it is a turtle. So let's not advise nonsense?))
The game is great !!
Perhaps I have already reached the limit of possibilities :) Therefore, I throw errors%))))))))

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12259
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Validate Saves

Post by ssilk »

azesmbog wrote:
Sun Jul 11, 2021 5:47 pm
ssilk wrote:
Sun Jul 11, 2021 8:28 am
Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?
I’m really sorry, that I used an example, which was too specifically pointing to you. Not good behavior.
But again, the last three autosaves are not loaded :))))))))))
Moreover, all three have different tile numbers.
Well, I had a verified save 18 hours ago. Well, I will build anew :) Zen, cho
Well, but incrementsdwith a backup, chances are very high, that there is a backup of an autosave.

Ah, and there seem to be an option to increase number of auto saves, https://www.reddit.com/r/factorio/comme ... autosaves/
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

azesmbog
Fast Inserter
Fast Inserter
Posts: 176
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: Validate Saves

Post by azesmbog »

ssilk wrote:
Sun Jul 11, 2021 11:34 pm

Ah, and there seem to be an option to increase number of auto saves, https://www.reddit.com/r/factorio/comme ... autosaves/
At first glance, it looks like an attractive solution. But..
I tried to change the time limit between autosaves.
I had 40 minutes, set 480 minutes, saved changes, loaded the game.
I went into other settings - writes "Saving - NEVER"
What to believe? this label or settings in the configuration file ??
Well, here above the good Samaritans advised to check the stability of the computer itself.
Well, now I ran a very angry and aggressive memory test.
No, this is not MemTest or TestMem. In 8 hours I will see the result.

Post Reply

Return to “Outdated/Not implemented”