Save in Background

[steinio]

Is it just me who is disturbed by the autosaving break?

I suggest to put the saving in a background task so that the game is not interupted anymore.

Greetings steinio

[Rseding91]

The game has to be paused so the current state can be exported out for saving.

[steinio]

Currently. I know pal

It's just a suggestion...

[Rseding91]

Currently. I know pal

It's just a suggestion...

Sure, but that doesn't mean it's possible "Lift a car with your left pinky" is also a suggestion but there are several reasons why that just isn't currently possible

[ssilk]

I wouldn't say, that it isn't possible, but it's not possible with the game, as it is now. And it's very expensive to change Factorio so, that it would be possible.

But it's so often requested, so there is some kind of demand.

Similar thread(s) (no deep search):

viewtopic.php?f=6&t=16388 Change Default Auto Save 2min to 15min.
viewtopic.php?f=6&t=24242 Use fork() on *nix systems for doing save game

Keywords: Autosave

[blizgerg]

I was reading this and an interesting idea came to me on how one might go about doing something like this.

Keep a journal of all events (like a playback) during the time when it is saving in the background. When save is complete, stop the journal and write it to the save file. Then when you load, you can use the journal to fix-up the inconsistencies that will exist in the save.

This might also require everything saved to also know what tick they where saved at and that info to be in the journal.

For example:

Chest A has a red circuit in it, and there is an inserter ready to move it to chest B.
Game save starts, chest A's contents are written. inserter moves red circuit, chest B's contents are written.
Save game now has double the red circuits. But the journal shows inserter moving a red circuit from chest A to chest B between those two save events. So on load the game can fix up chest A's contents.

Not knowing too much detail of how factorio actually works under the hood, this may be much more difficult than I make it sound... lol

[ssilk]

The problem is, that you cannot save while the game runs, cause you cannot know, which part of the memory contains which data of what time then.

In other words: The idea with the playback works only, if you have a clean previous state. Which needs to be essentially the whole and not parts. And to get the whole you need to save the whole.

There is the idea to save the diffs (which needs basically also a whole save of course):
viewtopic.php?f=6&t=23014 More efficient multiplayer map download
--> xdelta

[bobingabout]

I can agree from a programming perspective that a pause is necessary, I mean, they want a clean deterministic lockstep state saved, so it can be restored, and everything be as it was as if saving never even happened, and the game was just endlessly running.

To do this, you need to gather all information in the entire running game, and save it.


Now, there's 2 ways of doing this, and only 2.

The short-pause way: Pause the game, clone all instances of memory, and then resume the game while you work on saving this data.
This is actually the slower more processor and memory intensive method, because not only do you have the additional work of cloning all data, but you need the memory to store it too, and then the game might run sluggish while saving is processed while the simulation continues.

The long-pause way: Pause the game, save the state, un-pause the game. the entire save file is generated and saved while the game is paused. this is less processor intensive, because you're only doing one thing at a time, and uses less memory because you're not cloning the entire thing. You can however still create all the files that go into the zip file of the savegame, then unpause, then zip them up while the game is running again.


Now, there are other games that aren't so strict on the lockstep nature that save while the game continues to run, but obviously, certain elements of the game world are saved after others, and for the most part this is okay, because it's a slow changing world, and it doesn't really matter if one person is standing 10 paces further away, you're not going to notice, but this game is lockstep, so the entire game state has to be captured in the same tick.

I hope this explains why the game will always need to pause to save.

[ssilk]

Well, there are more ways between method 1 and 2.

The way I would say is the right is doing it with methods lent from (backup) cloning: In the moment a backup (here save) is started, any changes are logged in a way, that there are two places, where an information can be stored.


(taken from http://www.datto.com/blog/zfs-inverse-c ... me-machine )

The graphic represents the changes of a "block". With Factorio it means perhaps a chunk - or some other representation of the internal memory.
From left to right is the time.

The point "past" is the tick, where the backup starts. At this time nothing has to be done. The save can start nearly immediately (theoretical), which means, for every element in the game the serialize method is called and the result is packed and stored. That whole think can run in a second CPU.

Now with the next tick the game-update needs to write some changes to the memory.
Here things become a bit complicated:
- It needs to do copy the memory first.
- It needs to mark the old memory as outdated for the game (with the address of the new).
- Then it can make changes.
- For any other tick this repeats, but we don't need to copy the mem first (until not more than one save runs at the same time )

The backup ignores this new address; it still reads still from the "past". And when the backup is finished, the "past" snapshots can be deleted, some structures needs to be cleared etc.

This means: Things, that doesn't change while saving doesn't need to be cloned. And that is - depending on the game. But eventually only 10% of all chunks needs to be copied.

It means also: Before the game reads or writes while a backup is running it needs to check if there is a clone and change to that clone.

Some more ideas:
- In combination with xdiff (see threads above for link) it could be a very memory-save thing.
- This method would enable also some kind of faster map-transfer: "Transfer the chunks, that has been changed since".

[steinio]

Didn't know there is so much potential for discussion.

Thanks to all contributers, very interesting statements.

[zytukin]

The game already automatically stores 'replays' of your game while your playing doesn't it?

Could the game just pause the replay recording, create the autosave from that while storing new replay data in an alternate 'buffer' then 'catch up' the original replay using the buffer before resuming recording to it? Thus leaving the actual gameplay unaffected (or at least just a momentry FPS drop)

[seronis]

Factorio is a prestine example of "What to do right" when it comes to performance with a huge number of entities requiring game logic. The devs are geniuses with whatever design they used on the hard code side of the engine. That being said the only way to do a save with a given synchronized state while the game continues would be similar to what a few others above said and to how Journaling works in filesystems. Yes if a journaling system were implemented it would enable saving to not cause increased lag. But to do that the entire enterface of THE ENTIRE ENGINE would need to have two ways of accessing any variables.

variable.getRealValue(); //needed only by save routine
variable.getJournalValue(); //needed by engine while saving (actually at all times)

now the problem with this is that getJournalValue() method would have a LOT of list navigation and indirection. It doesnt ever access the value. It is instead navigating through a list of changes that potentially have 60 entries per second and totalling up all those changes before using the final computed value wherever it was needed in the code. This in itself can cause a bit of lag when you take into consideration how many 10s and 100s of 1000s of entities people have in their factory maps. And if you have that level of indirection its going to be there EVEN WHEN YOU'RE NOT SAVING. Now when you're not saving there will only be one value entry in the journal but it still means you are accessing the variable through a couple levels of indirection because of the pointers and list navigation. This means just supporting "pause free saving" will permanantly reduce the size of your factory setups.

Which is better:

1. a couple seconds of freeze time every 10-20 minutes when autosave runs
2. the entire game always only being able to have half as many objects because there is a massive overhead added to accessing every variable every time a value is needed to be checked (which is 100s of thousands of times per second)

I tried to balance the explaination somewhere in the middle of "enough info to settle dispute" and "simple enough for everyone to understand". And I suck at explaining things a bit. Hopefully i hit a good balance and successfully described why journal system for saving is a Bad Idea (tm)

[Killavirus]

meh, think your all over complicating this,

Just have 2 copies on drive :/ one live the other a backup. Most modern drives can easily keep up with it.
For slower systems or massive maps keep the pause to save as an option so it doesn't become unplayable just to fix a minor inconvenience

[Zeblote]

meh, think your all over complicating this,

nah, you are under complicating it

[ssilk]

@seronis: I think you think in the wrong direction. You just need

getValueForGame()
getValueForSave()

Where Game is the game-thread and Save the save-thread (which saves the game to disk). The Game-read needs to take care on the global state "save-in-progress" and if the value is already cloned or not. (Two "if" construct)

The Save-read is just the current getValue().

You can argue, that the Game-read is slower than now. But if programmed on a very low-level (e.g. exchanging the pointers to the method) there is also no difference - until handling cloned values. But that is only during save.

[mrudat]

This seems to be the most relevant thread to put this.

On Linux/MacOS, you can, in theory, fork off a child process, then perform the save there, and the memory manager will give you a consistent snapshot, as the entire process state is copy-on-write.

On Windows, you can't fork (easily), but you can emulate forking by using shared memory. As far as I can tell, you would need to allocate all memory holding state as shared memory, then spawn a new process that opens the shared memory segment copy-on-write, and I believe (though I'm not sure), that this gets you a snapshot of the shared memory segment similar to that you get when you fork.

Edit: fix gramma

[Darinth]

One more option that is available (or even possibly being used, I don't know how the internals of saving work) is to generate the save in-memory before writing it to disk. It's a middle-ground between saving everything to disk and cloning the entire game state. Most saves are within reason to be saved to memory, many as small as only 50 megs. Once the entire save file is generated in-memory, the game can continue while a second thread writes the save to disk. Game performance impact should be fairly low and only present for a handful of seconds every save. Best part is that it's a relatively easy method to implement. Should be able to be done by changing all of the current methods that write a file to instead write to a memory location followed by starting a thread to save and allowing the game to continue unhindered. Due to (relative) ease of implementation, it should also be reasonable to have a configuration option to disable it if for games/computers that can't spare the extra memory overhead.

Unfortunately, I don't know enough about where the save bottlenecks are for certain. I do know SSDs save faster, which indicates that at least a chunk of the bottleneck is in writing to disk (and now that I stop to think about it, indicates this technique is not used as HDD/SSD speed wouldn't effect game save if it were) and moving the save process from an SSD into ram-disk-like saving should yield at least a measurable drop in save time.

[Rseding91]

The game is saved to memory and written to disk in parallel with each other. Saving to memory is just what takes all of the time.

[E-37]

The game is saved to memory and written to disk in parallel with each other. Saving to memory is just what takes all of the time.

Which of the 2 processes does the save game bar represent? I occasionally run the game off a flash drive so that I can bring it to college and play it for 15 mins when I really need a break. I found that it saves at a normal rate but the bar stays at 100% for 10-20 seconds probably while writing to the flash drive. I guess it is an edge case but if it is just waiting to write to the flash drive it would be nice if the game could continue while writing the flash drive.

I suppose that saving in the background could run into weird cases where if it takes really long to save like if there is a HUGE map and a 30 second autosave interval that it could have multiple autosaves running at once and possibly overwriting each other. I have spent most of my time playing on a laptop that was considered a potato 3 years ago with a super slow disk and I never ran into any cases where it took longer than 10 seconds to save.

[Darinth]

The game is saved to memory and written to disk in parallel with each other. Saving to memory is just what takes all of the time.

Damn. Well there goes my biggest contribution to this conversation. I could've sworn I'd ran into threads on previous occasion that indicated SSDs saved faster, but I can't find them now so apparently I'm just getting old.

Also, given that I now know hard drive speed isn't a limiting factor... the idea of copying the entire game-state in-memory and then saving off of that is unlikely to be a viable speedup. The current method already functionally does that, except more efficiently in many ways as it only copies the data needed to save the state (by writing it to the in-memory buffer). There may be gains from a game-state journaling system (or one of several similar options that could freeze the current game state for the purpose of saves while also maintaining a separate state for the running game), but we're now getting into much harder and more time-consuming options to implement. Interestingly... it's like the factorio devs know what they're doing and already provided these optimizations where plausible.