Concurent autosave

[voddan]

Sometimes when I play Factorio the auto save dialog appears and all action freezes for several seconds

- I understand that this is the designed behavior
- I am still greatly irritated by it
- I think that it is possible to avoid stopping-the-world when autosaving

He's how concurrent autosave could be implemented:

In this post it is stated that a Factorio client can emulate a map at any moment given the original map and a stream of users' inputs.
So I propose using this mechanics to emulate a game in a parallel thread and save it instead of freezing the original game.

Step-by-step:
- introduce a buffer for user input [*]
-- it should work the same as sending user actions to a server, but storring them locally
- on autosave record 2 things:
-- a reference to a previous autosave state of the game [**]
-- a time stamp of the moment at which you want to autosave [***]
- this information together with the user input buffer (*) allows to completely restore the state of the game!
- to pack the saved state of the game in its current format, do:
-- start a separate thread with a Factorio engine
-- init it with the previous autosave (**)
-- send to it the buffered user input (*) that happened from the time of the previous autosave (**) to the time of this autosave (***)
-- save this parallel game with the standard procedure as you do now

Note:
- the original game is never stopped since all actions in the main thread take nano-seconds and several bytes of memory
- the parallel game can be run very quickly to shorten the saving procedure or very slowly to ease the load on the cpu


What do you think?

[posila]

I don't think this solution would be feasible - either your map is small and is saved within a second, or it is big and can't be simulated at fast speed - in lot of cases not even real-time speed.

On UNIX versions of 0.16 (Linux and MacOS) there is an experimental option to enable asynchronous saving, which forks the process and saves the game in fork. We tried to implement the same for Windows version using internal (undocumented) Windows functions, but it didn't work.

[sicklag]

.

[klanticus]

OpenTTD had this exact problem (freezing the game for a few seconds on every save/autosave) and they've managed to implement the asynchronous saving, so there's no pause anymore. I don't know how different are the internal architectures of both games, but they seem similar from a external high level view. maybe you could take a look at the code/talk to the devs to see how they've dealt with this issue.

[voddan]

I am glad to hear that the team is working on this.

I don't think this solution would be feasible - either your map is small and is saved within a second, or it is big and can't be simulated at fast speed - in lot of cases not even real-time speed.

Since the saving is async, it does not matter how long it takes to perform it. So even if saving takes 5 minutes, but is done in a background process, the user will never notice.

I have read somewhere on this forum that the factorio emulator is single threaded. If that is correct, a simulation of a game can be performed on a single core.
Assuming a multi-cored processor with at least 2 (4) cores, there should be no CPU problem to simulate 2 games in parallel.

The problem with my approach is in the RAM, since two identical simulations will use twice as much memory as a normal game. Still, even that simple approach should work on machines with enough RAM.

That said, I think I should underline that it is feasible to implement a fully async autosave on all platforms. Look at server-side virtual machines - they are able to save a running VM without even slowing it down, and the stop-the-word pause is in microseconds for huge servers under load. So the question of this topic is not IF it is possible, but HOW it is possible

[quyxkh]

I don't think this solution would be feasible - either your map is small and is saved within a second, or it is big and can't be simulated at fast speed - in lot of cases not even real-time speed.

On UNIX versions of 0.16 (Linux and MacOS) there is an experimental option to enable asynchronous saving, which forks the process and saves the game in fork. We tried to implement the same for Windows version using internal (undocumented) Windows functions, but it didn't work.

Please pleasepleeaseprettyplease just publish this. If copy-on-write isn't working on Windows yet, then it's not working on Windows yet. No biggie.

[orzelek]

I don't think this solution would be feasible - either your map is small and is saved within a second, or it is big and can't be simulated at fast speed - in lot of cases not even real-time speed.

On UNIX versions of 0.16 (Linux and MacOS) there is an experimental option to enable asynchronous saving, which forks the process and saves the game in fork. We tried to implement the same for Windows version using internal (undocumented) Windows functions, but it didn't work.

Please pleasepleeaseprettyplease just publish this. If copy-on-write isn't working on Windows yet, then it's not working on Windows yet. No biggie.

It's implemented and can be enabled in config from what I can see.

[Rseding91]

It's implemented and can be enabled in config from what I can see.

It is, but it's such an intense process that from a client-side experience the server lags-out for 4-5 seconds while it auto saves instead of just pausing for 1-2 seconds. (to me) it was a worse experience to lag out randomly for 4-5 seconds than to have a 1-2 second pause and knowing fully that it's just auto-save.

[quyxkh]

Oh, cool. I was looking for "auto" and "save", so i missed

Code: Select all

; non-blocking-saving=false

btw, that setting is in the default config twice with different defaults shown.

[voddan]

Cool! Gotta try it on Linux then.

it's such an intense process

Is there a way to slow the parallel simulation down? Since it is non-blocking, it could run at 1/10 speed and still get the job done, if I understood how it works.

[voddan]

Ok, so I have researched it a bit, and turns out forking threads is NOT a way to go.

The problem is that for an actively mutating application (such as Factorio), system forking is equivalent to practically stopping the application and clonning all of its memory.
This is exactly what you have experienced - main thread dramatically slowing down and CPU working very hard to copy the memory.

What is described in the topic post is different - it recreates a Factorio game from a previously saved state without attempting to clone a running one.
This method uses RAM space and disk bandwidth, but it does not uses much CPU power and its affect on the running game is close to zero.

In other words, what posila wrote about is not an equivalent implementation!

[quyxkh]

Setting

Code: Select all

non-blocking-saving=true

on my reasonably-nice-for-five-years-ago box autosaves a 130MB >1kspm base concurrently with no discernible slowdown.

It's absolutely lovely. _huge_ thank you to the devs for implementing this.