Save in Background

[wombler]

I've been playing with implementing copy-on-write to see how difficult this would be in practice.

The general idea is to install a signal handler and allocate anonymous memory. When it's time to save, mark the memory read-only from the non-save thread, launch save threads, and continue the game as normal. The handler does basic COW behavior: re-map the faulting page to a new read-write "shadow" page and resume execution. This stops the non-save threads from performing any extra saving/non-saving checks which may hamper branch prediction or complicate code. It will however slow down gameplay as pages fault.

Meanwhile, the saving threads map a second view of the anonymous memory at a different base address - this memory is guaranteed immutable, however all pointers will need to be adjusted due to them being at a different base address. The pointer lookups are extra work, and will cause a bad time if something is missed, however it's theoretically isolated to the save code paths only.

When saving is completed, the main thread copies data from the shadow pages back to the original pages, and resets the memory region as read-write.

I've pushed my PoC at https://github.com/tiedotguy/cow if there's any interest.

[ssilk]

Sounds promising... nice work...

Now we need “just” a way to transfer the changed chunks to a client that tries to catch up from a game save.

[posila]

First of all, that's impressive.

The way how to make background saving is to design your game/engine so that it allocates game state memory from some known arena and game state objects should use indices instead of raw pointers when referencing each other. Then, when saving, just mem copy the arena - sequential memory copy is fast even the slow implementations should be at least close to 10 GB/s. Then you can either just save the copy out as is, or do some compact serialization that works over the copy.

But Factorio has not been architected that way at all. The game almost never does custom memory management, uses pointers everywhere and standard library containers that do their own allocations and use pointers internally, plus some other libraries ... etc. Fork is helpful, in that it is able to make a snapshot of entire process without instantly duplicating all of its memory, also preserving virtual address space and being able to run normal saving code that already existed. Frankly, I didn't think it would work due to game having much more stuff going on than console application - like having application windows, OpenGL context, sound device, etc. But to my surprise, it did. However, we are not gonna bend and twist huge chunk of the game to make fake fork on platforms that don't support it.

When POSIX fork worked, I spent some time trying to make this implementation (https://gist.github.com/Cr4sh/126d844c28a7fbfd25c6) of Windows fork work, but couldn't. The "forked" process doesn't have properly loaded DLLs and can't do anything that would be useful to us. It is possible there is a way to make it work, but then that's putting lot of effort into something that depends on undocumented OS fuction and may just stop working in the next Windows update.

Last thing ... before coming up with possible solutions, please try to state the problem first as exactly as possible. When I see a thread saying "Game startup is slow" or "autosave takes long time" I think to myself "yeah, that thing takes longer than 16ms, so it indeed is slow" and go back to what I was doing before. If I see "Saving of my 100h map takes 5 minutes, on my i7-####K, with 16 GB RAM. Save size is 50 MB, here is the save <download>" than that's immediatelly interesting thing to look into, largely because that's not my experience with my 100h/50MB maps.

[Rseding91]

Last thing ... before coming up with possible solutions, please try to state the problem first as exactly as possible. When I see a thread saying "Game startup is slow" or "autosave takes long time" I think to myself "yeah, that thing takes longer than 16ms, so it indeed is slow" and go back to what I was doing before. If I see "Saving of my 100h map takes 5 minutes, on my i7-####K, with 16 GB RAM. Save size is 50 MB, here is the save <download>" than that's immediatelly interesting thing to look into, largely because that's not my experience with my 100h/50MB maps.

Yes! What's even more confusing is when someone uses a term we use internally but doesn't *mean* that internal thing. The most common being "memory leak" when they just mean "high memory usage".

[shakephobia]

Sorry if this is necromancy and is not allowed, but I feel like this discussion still has somewhere to go.
There is a documented feature in windows that allows taking a memory snapshot of a process:
https://docs.microsoft.com/en-us/window ... resnapshot
We use it in our project at work to take full memory dumps of our process for debugging purposes without suspending the process (very useful for data collection from client's machines for debugging without interrupting the service).
There is an important downside, and that it is not available in Windows 7 and Windows 8, only Windows 8.1 and above. I would assume that the majority of Windows factorio players would still benefit from that.
Second downside we encountered - we are in the field of security, and when cryptomining was all the rage, we tried to implement a mechanism that detects several methods. One of the ways we tried to go about it is using this feature to create a snapshot of the suspicious process and inspecting it without interrupting its work. However, we encountered several 3rd party applications in the wild that leaked handles to the snapshot process, making it unkillable. That completely killed the project for us.

From what I know about this feature, it basically does copy-on-write - at first it does nothing, but as soon as a page is referenced in the real process, its content is copied aside, the content is changed in the real page and the address to this memory section is swapped in the snapshot.

Could that help you implement this sort of this for Windows?

[ssilk]

It’s never too late to add information.

[LoneWolf_LWP]

since background saving is to much of a overhaul for factorio as stated before in this thread i would suggest this.

1. why not allow to set a save interval like in multiplayer? i only play in multiplayer just because of this option as it is now.
2. Show a warning timer (timer maybe user defined in settings? ie. 30 seconds before save starts) that a save is about to start

[Xorimuth]

since background saving is to much of a overhaul for factorio as stated before in this thread i would suggest this.

1. why not allow to set a save interval like in multiplayer? i only play in multiplayer just because of this option as it is now.
2. Show a warning timer (timer maybe user defined in settings? ie. 30 seconds before save starts) that a save is about to start

Well since this thread started, the non-blocking save was added to unix systems (via the OS fork command) so it is definitely valid.

1. You can set a save interval. It is in Settings > Other
2. I'm not sure how that would help? I personally wouldn't want to see that, it would be much more intrusive than a ~10 second (though realistically for many players <5 second) break

[LoneWolf_LWP]

since background saving is to much of a overhaul for factorio as stated before in this thread i would suggest this.

1. why not allow to set a save interval like in multiplayer? i only play in multiplayer just because of this option as it is now.
2. Show a warning timer (timer maybe user defined in settings? ie. 30 seconds before save starts) that a save is about to start

Well since this thread started, the non-blocking save was added to unix systems (via the OS fork command) so it is definitely valid.

1. You can set a save interval. It is in Settings > Other
2. I'm not sure how that would help? I personally wouldn't want to see that, it would be much more intrusive than a ~10 second (though realistically for many players <5 second) break

hmm, I thought that only applied to multiplayer, i had set that before and stil got the standerd 5min (will check it again)
And for 2 you should be able to disable it if you dont want it. it was just a thought that could help the people that get annoyed with the instant halt of the game for (depending the save file size) x amount of time.

[freeafrica]

However, we encountered several 3rd party applications in the wild that leaked handles to the snapshot process, making it unkillable. That completely killed the project for us.

I'm a bit confused about this reasoning. Why does leaking handles make a process unkillable? (Just wanna learn)

[shakephobia]

However, we encountered several 3rd party applications in the wild that leaked handles to the snapshot process, making it unkillable. That completely killed the project for us.

I'm a bit confused about this reasoning. Why does leaking handles make a process unkillable? (Just wanna learn)

A handle to a process is a handle to a kernel object called EPROCESS. Kernel objects cannot be destroyed while there are open handles to them. So a process should not be able to terminate as long as another process still has a handle open to it.
The process cannot continue to run code or do anything, it has no running threads (as snapshots are just "pictures" of memory, they have no threads), but its just... stuck.

Edit: The same goes for files. If a file is opened by two different processes (processes A,B) and A deletes the file, the file will still appear (but will be inaccessible) until process B closes the handle.