Validate Saves

[KeepGridButton]

TL;DR

Consider detecting corruption by a validating a save by running it through the load routine, in order to prevent multiple saves from being corrupted..

What ?

After completing a save operation, allow gaming to continue, but in a background thread run that save through the load routine to validate there are no errors loading the save. If there is an error, warn the user the last save is potentially corrupted, and additional saves may also be corrupted. Recommend exiting the game immediately before additional saves are corrupted.

Once detected, consider performing a backup of all 3 autosaves to new filenames, so even if the user doesn't understands and keeps playing corrupting all their autosaves, they'll have the backups to go to.

Why ?

After experiencing a save corruption issue I found several other threads and an entire forum called "0 / 1" that seem to be a variety of what seems like memory corruption that resulted in a bad save. In many cases multiple saves have the exact same corruption. I assume that some memory corruption occurs, and then it is written to each save subsequently during that gaming session until all the autosaves and the player's manual save contain the same issue. If it were disk level bit rot I wouldn't expect to see that pattern.

Thus the next time the player loads the game, they find none of their saves load, and it requires someone with detailed knowledge to repair the save. So even someone who rotates through several saves might find in a single gaming session they've corrupted all of their saves unknowingly until the next time they play.

By performing a load routine validation of the save, this issue will be detected sooner(fail fast strategy) and assuming the user follows advice and exits, it prevents additional saves from being overwritten, allowing the user to go back to a previous save or autosave.

If there's something I'm misunderstanding about how this happens at a technical level, which invalidates my suggestion, then I would say in general try to determine if there's a fail fast strategy that would catch the issue sooner. Failing that, perhaps you could come up with a backup strategy for saves that makes a backup copy of a save to a subfolder when it successfully loads. This way you'd have at most one extra copy of a particular validated "good" save. Other backup strategies I considered could potentially overwrite good saves with bad ones. Overwriting the backup of the same name after a successful load means you'd have no more backups than you have saves, so it won't grow out of control or need to be managed. This would mitigate the need for intervention from technical staff to fix saves manually. Sorry if this violates the "multiple suggestions" rule.

You should be more careful, don't overwrite all your saves in one session. Make backups

I anticipate some people are going to adopt a "you should be more careful" attitude. I personally ping pong between two manual saves so that I've got an extra in case the game crashes or I get a power outage during a save operation, plus there's the autosave rotation. There's also steam cloud backups, so even if my entire computer went up in flames, I've still got access to my saves. So it feels like there's a couple levels of protection there already. I would have never thought that memory corruption would survive through multiple saves and propagate through all of them without something crashing. This isn't a criticism or complaint, I'm just saying an average user being cautious already in there save strategy probably wouldn't understand that a playthough they've put alot of time into could be wiped out in this way.

[ssilk]

Hm. I don’t think this a good strategy. I would use it only, if I (as dev) would search for a specific bug. Because this occurs so seldom this extra loading needs a) too much resources and b) doesn’t even guarantee that if it loads it is able to run (happened to me much more often).

Instead I would try to fix those bugs, that make this corrupt save happen and try to address the issues deeper into the architecture of the internal representation to make them more robust.

[azesmbog]

It rarely happens, or not rarely, but I have only encountered this problem twice recently. Of course, you can say that I have problems in the computer hardware, and these are my problems. And if still not ??
Saving with subsequent verification is - in my case - a very, very bad decision.
For me, saving and subsequent loading takes about 20 minutes, total 20 + 20 = 40 minutes: ((
I can do this trick every two or three days, but not every hour and a half.
Maybe all the same something to fix at the conservatory? (c)

[ssilk]

It doesn’t matter what causes the error, if it is a software (Factorio inside) or a hardware or OS problem (outside Factorio). In any case if verification fails: what should be done?

As here:

If there is an error, warn the user the last save is potentially corrupted, and additional saves may also be corrupted. Recommend exiting the game immediately before additional saves are corrupted.

What if this doesn’t help? What if the error still occurs? This is a software-anti-pattern: let the user know there might be a problem and let it handle it him/herself. Gnaaaa, I learned that the hard way, that this isn’t gonna be working. It makes things worse!

What works is, if you know how to fix this problem and if you exactly know how to fix it, you can program a solution. (as already said as a way to find the bugs, this is a possibility, but then it is a clear message for the user: there was a failure, we have logged it)

loading takes about 20 minutes,

Hu? I have saves with 200+ MB and they take about two minutes for loading. And that are large worlds, where I need about 4 minutes to travel with 300km/h train from one edge to the other. How big are those saves?

[azesmbog]

And that are large worlds, where I need about 4 minutes to travel with 300km/h train from one edge to the other. How big are those saves?

These are your small worlds
My save world takes ~ 3 GB, saves ~ 18 minutes on a fast SSD drive.
from the respawn point to the last point I have to go for many many hours, with 6-7 intermediate refueling with nuclear fuel - this is a little more than 2,500,000 cells on the map or 2,500 km?
At the same time, the UPS is still within 40.
For three years I have not had such preservation problems.
The hardware has not changed for the last year and a half, and the software has not changed or been added either, this computer is designed for only one game 24 hours a day \ 365 days a year

[boskid]

Idea of save validation by loading it directly after saving has one extreme major flaw: to verify the save is consistent, it has to be fully loaded including allocation of all the internal structures like chunks, entitites and be properly setup as some of the consistency checks work after the full load-and-setup. This basically means memory usage would increase by a factor of two. On a small save files it could be done but also smaller saves are less likely to undergo a random bitflip and the loss of such save hurts less. Its main value would be with large saves and this is exactly the case where it cannot be used due to doubling of memory requirements. I can safely throw this idea to "outdated/not implemented".

[coppercoil]

Idea of save validation by loading it directly after saving has one extreme major flaw[..]

There are players who have no memory shortage issues... Why not to add an option?

[eradicator]

Idea of save validation by loading it directly after saving has one extreme major flaw[..]

There are players who have no memory shortage issues... Why not to add an option?

Sounds like it would be the same as a small script that starts a headless server that loads the save and then exits?

[KeepGridButton]

Hm. I don’t think this a good strategy. I would use it only, if I (as dev) would search for a specific bug. Because this occurs so seldom this extra loading needs a) too much resources and b) doesn’t even guarantee that if it loads it is able to run (happened to me much more often).

Instead I would try to fix those bugs, that make this corrupt save happen and try to address the issues deeper into the architecture of the internal representation to make them more robust.

Responses from staff on these issues include comments like "bit flip from cosmic rays". That sounds like they're saying its not a bug that can be fixed. I wouldn't be surprised if eventually we're going to get to a point that staff are no longer going to fix saves for us.

Doing a backup to a subdirectory after a successful load to a subdirectory should be cheap to implement and pretty cheap performance wise. You'd never have more backups than you have saves so it'd not be a big cost in space either.

Yeh you should identify and fix the root cause, but people shouldn't lose complete playthroughs everytime they encounter a bug. You're post 1.0, bugs shouldn't be this destructive IMO.

[Qon]

These are your small worlds
My save world takes ~ 3 GB, saves ~ 18 minutes on a fast SSD drive.
from the respawn point to the last point I have to go for many many hours, with 6-7 intermediate refueling with nuclear fuel - this is a little more than 2,500,000 cells on the map or 2,500 km?
At the same time, the UPS is still within 40.
For three years I have not had such preservation problems.
The hardware has not changed for the last year and a half, and the software has not changed or been added either, this computer is designed for only one game 24 hours a day \ 365 days a year

I'm impressed. And envious, I also want a machine that just runs a save of Factorio every second for years!
Do you have a thread documenting your world? How big is it? What's your production? What does it look like and what does it do now? Purpose?

[ssilk]

3 GB Wow as already said: make a video, show your creation, I’m sure many people are interested into that.

Well I have an idea, because with that size I can understand that a verify is helpful (*):
use the headless version of Factorio to load the file in background. Could also be done on another computer (loading the save over mounted network devices: check if there is a newer file than the last check, try to load, on error display alarm).

I’m sure if you ask, you find people in other boards that help to write such a script, because it’s highly dependent which platform you use and other technical dependencies.

(*) this size meets the edge of reliability of computers quite hard. Some cosmic ray can cause small bit-changes - for example. The quality/reliability of the computer hardware plays a much more important role than normally to play this without fail, because the chance that something happens rises exponentially with the size.

[azesmbog]

Do you have a thread documenting your world?

See here:
viewtopic.php?f=204&t=73328

Video would be nice, of course, all the hands would not reach, capture a video and make a tour of my sandbox.
Few probably have seen a 45 GW compact ) nuclear power plant, and even the epic video of the explosion of this power plant is generally pleasing to the eyes) And not on every computer my base will start at all, even though it is pure vanilla.
On saves, the least developers can do is increase the maximum time between autosaves. Now, for some reason, only 95 minutes. This is not enough for me. In five hours of play, these saves will overwrite themselves, and it's good if they don't make an error.
If, for example, I had 8 hours between saves, then the rewriting would be in a day, and I would have much more time to think, check the recording and make a decision. Otherwise, I have to make a separate backup every two or three days, always with verification. Sometimes I forget to do it like the last time, and I had to go back 10 days

[Koub]

https://xkcd.com/378/
At some point, if you fear these kind of things, you should use ECC RAM, which will take you from a significant daily bit flip probability to 1 in a billion-ish bit flip probability.

[Nosferatu]

I was about to make a similar suggestion when I saw that op has already proposed something similar:

Failing that, perhaps you could come up with a backup strategy for saves that makes a backup copy of a save to a subfolder when it successfully loads. This way you'd have at most one extra copy of a particular validated "good" save.

This line of solutions has not been discarded yet. Correct?

My solution would have been:

Don't overwrite the last loaded save.
If the game trys to save on the file that has loaded the current game:
Back it up first. Then overwrite it.

[ssilk]

But should this be part of the game?

For example: on macOS I would say: you’re crazy! I have time machine backup, why should the game backup, too? Makes no sense.

A good backup means in any case, that I use a backup-strategy that fits my needs, and that is always part of my whole concept, how I make sure, that I can work, without loosing my work. It’s not part of one app to make that sure, because it makes no sense, to do that over and over for each app.

[Nosferatu]

The average windows user has no daily backup system for his game files.
And why are there 3 autosaves? One should be enough

You can see that this problem pops up repeatedly on the forums. And when it hits it usually destroyers 100s of game hours.
This is a very easy solution. So why not simply do it?

[ssilk]

The average windows user has no daily backup system for his game files.

That’s also true for Linux users. But is that the problem of the game??
Windows has no useful backup. Users don’t want to invest in a backup medium. Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?

Sorry, I wanted to avoid ranting about missing backup, but a game cannot be misused to address that you need to have a backup that fits your needs.

And why are there 3 autosaves? One should be enough

The autosave makes much sense and their target is completely different.

This is a very easy solution. So why not simply do it?

Well, there are lots of ideas about that. See

viewtopic.php?f=80&t=327 World Names (Configurable additional filename chars and Versioning)

viewtopic.php?f=80&t=21367 Game Save File and -Speed Related Suggestions

Most of them are very old.

[azesmbog]

Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?

And yet the problem is in the game.
let me be misled)
How do you suggest I make 3GB backups?
At each autosave in the background after it, overwrite the save file in a separate storage ??
The same garbage will be, without checking the subsequent download.
I do not argue at all, these are rhetorical questions, and the developers know better what and how to do.
But again, the last three autosaves are not loaded )))))))))
Moreover, all three have different tile numbers.
Well, I had a verified save 18 hours ago. Well, I will build anew Zen, cho
And yes, 64GB of storage is a must.
but.
But ordinary motherboards do not support server memory with parity, and even with the declared support, I could not get it to work. These are two. Thirdly, the register memory is very slow in terms of speed characteristics. 3600 - I certainly did not see it in the price lists, but at 2400 it is a turtle. So let's not advise nonsense?))
The game is great !!
Perhaps I have already reached the limit of possibilities Therefore, I throw errors%))))))))

[ssilk]

Problem of the game? No! If someone invests into a computer 64 GB Ram, he/she should be easily able to invest into a working backup, that is useful for his kind of setup. If someone invests 1000 hours into a game and builds something great and has no backup, how much is the invest worth?

I’m really sorry, that I used an example, which was too specifically pointing to you. Not good behavior.

But again, the last three autosaves are not loaded )))))))))
Moreover, all three have different tile numbers.
Well, I had a verified save 18 hours ago. Well, I will build anew Zen, cho

Well, but incrementsdwith a backup, chances are very high, that there is a backup of an autosave.

Ah, and there seem to be an option to increase number of auto saves, https://www.reddit.com/r/factorio/comme ... autosaves/

[azesmbog]

Ah, and there seem to be an option to increase number of auto saves, https://www.reddit.com/r/factorio/comme ... autosaves/

At first glance, it looks like an attractive solution. But..
I tried to change the time limit between autosaves.
I had 40 minutes, set 480 minutes, saved changes, loaded the game.
I went into other settings - writes "Saving - NEVER"
What to believe? this label or settings in the configuration file ??
Well, here above the good Samaritans advised to check the stability of the computer itself.
Well, now I ran a very angry and aggressive memory test.
No, this is not MemTest or TestMem. In 8 hours I will see the result.