Non-blocking saves?

[JasonC]

I see an incidental reference in the 1.1.70 release notes to "non-blocking saves" (some bug was fixed related to it) with wording along the lines of "when non-blocking saves are used".

I didn't know this was a thing, and the wording implies that it's some sort of option. However, I couldn't find anything in the game settings.

Is non-blocking saves a thing and is it what I think it is? If so, how can I enable it? My saves are starting to take > 5 or 6 seconds which is fine except autosave is becoming disruptive, and it would be super cool if there was some option (even experimental) to have the saves be done in the background instead of suspending the game during the save. I couldn't find anything in Settings, though.

Thanks!

[DaveMcW]

viewtopic.php?t=57464

[quyxkh]

It's Linux-only, Windows simply can't support it. To light it up put `non-blocking-saving=true` in your config or hunt down the checkbox in the "the rest" set that shows up if you hold down ctrl+alt when you pick "settings". It's a fan-made feature, someone with source access basically offered it as a patch/pull request/whatver, it has zero official support, it's kept working because keeping it working is simple.

[Tertius]

I would kill for this feature on Windows.

[aka13]

I would kill for this feature on Windows.

Question is, who would not?

[jodokus31]

I would kill for this feature on Windows.

Question is, who would not?

Installing Linux f.e. EndavourOS or Linux Mint is an option before committing a crime
Maybe in a VMWare, but it might have some Performance drawbacks
Or install in parallel, which I have done. And now, I mainly use Linux

fun fact: I didn't even enable this function.

[JasonC]

Thanks all; and yeah, Windows, would kill. Devs please PM me your targets' names, last known locations, and photos.

[Rseding91]

Thanks all; and yeah, Windows, would kill. Devs please PM me your targets' names, last known locations, and photos.

To make it work on windows would require that the windows operating system support process level forking and copy-on-write. So you'll need to contact Microsoft about that one.

[JasonC]

Thanks all; and yeah, Windows, would kill. Devs please PM me your targets' names, last known locations, and photos.

To make it work on windows would require that the windows operating system support process level forking and copy-on-write. So you'll need to contact Microsoft about that one.

Yeah.

Crap, MS definitely already has their own merc hit squad (pretty sure they're a subgroup of their Legal department), I need to think of an offer with more value.

[pioruns]

Just a reminder, Non-blocking Saves works perfectly well in Space Age DLC! I just tested it, re-enabled it on fresh install, and works well. Very useful feature!

[Mesqalito]

I've just stumbled upon this article:
https://devblogs.microsoft.com/engineer ... ly-access/

I've no idea what they are talking about but it talks about Copy-on-Write in Windows, which was mentioned here as a requirement for non-blocking-saving to work. Any chance we're getting closer to that feature on Windows?

[Nidan]

No, that article talks about copy on write for copying files, i.e. instead of actually copying the file, both old and new are set up to share the same data on disk. What factorio needs is a way of duplicating its process without having to also duplicate all the memory associated to it.

[Tertius]

As far as I understand, Windows has a copy on write for memory as well, however you need to explicitly design and code, while with unix-like systems all complexity is completely hidden inside the fork() call. All you need to do is fork() without complex initialization, and you just need to check the return code to detect if you're the parent or the child to either continue or save and exit. The fork() is "misused" in this case to create a copy of the process to use it as snapshot of its data, then use the map info from this snapshot to create the save.

For Windows, there is https://github.com/mjsabby/CopyOnWriteDump for example, however it's not clear how practical this is for Factorio.

[eugenekay]

As far as I understand, Windows has a copy on write for memory as well, however you need to explicitly design and code, while with unix-like systems all complexity is completely hidden inside the fork() call. All you need to do is fork() without complex initialization, and you just need to check the return code to detect if you're the parent or the child to either continue or save and exit. The fork() is "misused" in this case to create a copy of the process to use it as snapshot of its data, then use the map info from this snapshot to create the save.

For Windows, there is https://github.com/mjsabby/CopyOnWriteDump for example, however it's not clear how practical this is for Factorio.

PssCaptureSnapshot is the mechanism used to implement this, but it has some serious implications for Performance and Memory Layout. Raymond Chen's Blog lays out some of the challenges. This facility is really for Debug purposes only - the Process Snapshot is given "low-priority memory configuration"; so accessing the entire buffer to read out State would likely be much slower if done on a regular basis.

[Rseding91]

PssCaptureSnapshot is the mechanism used to implement this, but it has some serious implications for Performance and Memory Layout. Raymond Chen's Blog lays out some of the challenges. This facility is really for Debug purposes only - the Process Snapshot is given "low-priority memory configuration"; so accessing the entire buffer to read out State would likely be much slower if done on a regular basis.

Interesting. But then i found this https://learn.microsoft.com/en-us/windo ... dfrom=MSDN which I wonder if that could be used to work around this? That's a fascinating prospect.

[Tertius]

Interesting. But then i found this https://learn.microsoft.com/en-us/windo ... dfrom=MSDN which I wonder if that could be used to work around this? That's a fascinating prospect.

According to Chen in the last comment to the linked Raymond Chen's blog, the "low priority memory" deals with how the memory manager pages in from disk. This is relevant, if the process uses on-demand paging and memory mapped files.
If Factorio doesn't use on-demand paging for the map data that's used to create the save file, but instead it's all actually stored in memory, there would be no speed penalty if this functionality is used for background saving. The penalty is just from low priority I/O reading data from disk.

[Rseding91]

Actually, looking at that function more it looks like it's not going to be viable. It makes a copy-on-write version of a given processes memory - however it does it as *its own process* which you can then pull bits of data from using https://learn.microsoft.com/en-us/windo ... cessmemory

That method is almost completely useless because it would require you write a completely duplicate set of save functions which operate via proxy through the process handle using offsets.

If it simply made a copy of the process memory *in your address space* then it would be as simple as getting a pointer to that memory - as the Map instance - and calling save(output).