Bikeshedding about non-blocking saving on Windows

Post your ideas and suggestions how to improve the game.

Moderator: ickputzdirwech

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Bikeshedding about non-blocking saving on Windows

Post by ptx0 »

Rseding91 wrote:
Sat May 09, 2020 4:56 pm
Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.
it has been implemented via Cygwin (and i presume subsystem for Linux but you cant guarantee its presence or featureset), and the details of those two implementations are probably pretty equally shitty: http://cygwin.com/cygwin-ug-net/highlig ... hi-process

Jap2.0
Smart Inserter
Smart Inserter
Posts: 2368
Joined: Tue Jun 20, 2017 12:02 am
Contact:

Re: better compression algorithm for saves and mods

Post by Jap2.0 »

Rseding91 wrote:
Sat May 09, 2020 4:56 pm
Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.
I'm not an expert on Windows APIs by any means, but apparently there is ZwCreateProcess which is similar. (example 1, example 2 (original) - first is GPLv2 and the second is some obscure French license so I'm not sure if you can directly use either of them, but they exist.)

Granted it's technically undocumented (except for some random book that seems to have a lot of Windows API documentation from ~2000 that I keep seeing mentioned) so that's not ideal but Windows tends to have pretty stable APIs (...for the most part, see below).

Edit: example 1 was released under MIT so that should be fine (1/2)

Edit 2: Apparently at least the first has issues on newer than Vista. But it might be possible. Maybe. Probably still not something you want to get into, but just in case there's enough interest.
There are 10 types of people: those who get this joke and those who don't.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12889
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ssilk »

Rseding91 wrote:
Sat May 09, 2020 4:56 pm
Non-blocking saving isn't possible to implement on windows in the same way it works on the other platforms. Windows does not have process forking.
I would not say impossible - just as said: much more complicated to implement, because reimplementing fork is hard. But maybe there are other ways...
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

User avatar
Optera
Smart Inserter
Smart Inserter
Posts: 2920
Joined: Sat Jun 11, 2016 6:41 am
Contact:

Re: better compression algorithm for saves and mods

Post by Optera »

Almost all non-blocking windows backups use the built in shadow copy feature.

User avatar
Oktokolo
Filter Inserter
Filter Inserter
Posts: 884
Joined: Wed Jul 12, 2017 5:45 pm
Contact:

Re: better compression algorithm for saves and mods

Post by Oktokolo »

Optera wrote:
Sun May 10, 2020 7:09 am
Almost all non-blocking windows backups use the built in shadow copy feature.
Shadow copy can create snapshots of files and volumes. To use that with a Factorio game state, it has first to become a file...

User avatar
Optera
Smart Inserter
Smart Inserter
Posts: 2920
Joined: Sat Jun 11, 2016 6:41 am
Contact:

Re: better compression algorithm for saves and mods

Post by Optera »

Oktokolo wrote:
Sun May 10, 2020 9:29 am
Optera wrote:
Sun May 10, 2020 7:09 am
Almost all non-blocking windows backups use the built in shadow copy feature.
Shadow copy can create snapshots of files and volumes. To use that with a Factorio game state, it has first to become a file...
Ah so it's writing memory to file blocking in Windows. Nevermind me then.

posila
Factorio Staff
Factorio Staff
Posts: 5264
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: better compression algorithm for saves and mods

Post by posila »

Optera wrote:
Sun May 10, 2020 10:28 am
Ah so it's writing memory to file blocking in Windows. Nevermind me then.
It is blocking, because the update thread can't be doing changes to game state while saving thread is traversing it and serializing it. On POSIX OSes, the game process is forked - the process is duplicated with almost exact state - only difference is that the original (parent) and duplicate (child) know which is which. Parent and child process share all parent's existing memory in copy on write mode, which means the OS will make new copies of memory pages for the parent and the child process as either of them starts making changes in the shared memory.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

posila wrote:
Sun May 10, 2020 1:45 pm
Optera wrote:
Sun May 10, 2020 10:28 am
Ah so it's writing memory to file blocking in Windows. Nevermind me then.
It is blocking, because the update thread can't be doing changes to game state while saving thread is traversing it and serializing it. On POSIX OSes, the game process is forked - the process is duplicated with almost exact state - only difference is that the original (parent) and duplicate (child) know which is which. Parent and child process share all parent's existing memory in copy on write mode, which means the OS will make new copies of memory pages for the parent and the child process as either of them starts making changes in the shared memory.
yeah the problem i mentioned in my original post was that fork() reimplementations on windows can not be copy-on-write, the windows kernel does not have any mechanism for this. it *must* copy the entire process memory space into a new address space and memcpy is expensive - it gets worse the larger the working set is. it's a problem that Windows must solve, not Wube.

User avatar
Oktokolo
Filter Inserter
Filter Inserter
Posts: 884
Joined: Wed Jul 12, 2017 5:45 pm
Contact:

Re: better compression algorithm for saves and mods

Post by Oktokolo »

ptx0 wrote:
Mon May 11, 2020 4:45 am
yeah the problem i mentioned in my original post was that fork() reimplementations on windows can not be copy-on-write, the windows kernel does not have any mechanism for this. it *must* copy the entire process memory space into a new address space and memcpy is expensive - it gets worse the larger the working set is. it's a problem that Windows must solve, not Wube.
The naive way to solve it, is to treat game states as immutable. The game state update code would take the old game state and create a new one including the updates.
After the update, the old state normally is discarded.
The game state save code would take an existing game state and just keep it until it is done with saving it to disk.

And if the last N game states would be kept in memory, a game state debugger could go back N game states in time. But that obviously only becomes real useful with key game states and incremental game state change overlays - or obscene amounts of RAM...

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

Oktokolo wrote:
Mon May 11, 2020 7:47 am
ptx0 wrote:
Mon May 11, 2020 4:45 am
yeah the problem i mentioned in my original post was that fork() reimplementations on windows can not be copy-on-write, the windows kernel does not have any mechanism for this. it *must* copy the entire process memory space into a new address space and memcpy is expensive - it gets worse the larger the working set is. it's a problem that Windows must solve, not Wube.
The naive way to solve it, is to treat game states as immutable. The game state update code would take the old game state and create a new one including the updates.
After the update, the old state normally is discarded.
The game state save code would take an existing game state and just keep it until it is done with saving it to disk.
that's what it means when i say it's not copy-on-write.

windows has to make a complete copy of all memory space for a process, and maps can use under vanilla gameplay about 6-8GiB memory meaning duplicating all of that every time you save the game.

it's also not a free process, copying memory is expensive. it's going to cause a pause so that the game state can be made immutable while it is copied into new address space. you can't avoid the pause, because it will corrupt the game state, without adding some stupid form of journaling into the game state that will undoubtedly impact performance for just one very small reason.

i mean, something you could argue is that you might want an option to enable saving under any exit condition to a temporary save file, like $SAVENAME"-corrupted-$(date +%s)". why do we save? to revert to autosave if the game crashes, mostly, for me - not like vanilla gameplay crashes much, but with mods, whoo boy.

sometimes it'll exit without any option to save. sometimes it allows you to save. i'm not sure why it does sometimes and doesn't others - and sometimes it allows SOME players to save the map, but not the one who experienced the error - and this isn't useful in single player.

so maybe if we can get that resolved and the game state can be saved any time the game unexpectedly exits then autosave can be simply disabled in more cases since there will be some form of obtaining the new map when Things Go Wrong.

coppercoil
Filter Inserter
Filter Inserter
Posts: 494
Joined: Tue Jun 26, 2018 10:14 am
Contact:

Re: better compression algorithm for saves and mods

Post by coppercoil »

Ok, copying RAM is expensive. Which is less expensive: full serialization and compressing like now, or copying the memory? If there’s no non-blocking save, can we have a shorter freeze?

User avatar
Oktokolo
Filter Inserter
Filter Inserter
Posts: 884
Joined: Wed Jul 12, 2017 5:45 pm
Contact:

Re: better compression algorithm for saves and mods

Post by Oktokolo »

coppercoil wrote:
Mon May 11, 2020 6:48 pm
Ok, copying RAM is expensive. Which is less expensive: full serialization and compressing like now, or copying the memory? If there’s no non-blocking save, can we have a shorter freeze?
Copying RAM is orders of magnitudes faster than storing stuff to disk.
So yes, just copying the gamne state, then continue running the game while also doing the actual saving of the copy in background is a viable strategy to reduce the freeze to less than a second.

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: better compression algorithm for saves and mods

Post by ptx0 »

Oktokolo wrote:
Mon May 11, 2020 8:48 pm
coppercoil wrote:
Mon May 11, 2020 6:48 pm
Ok, copying RAM is expensive. Which is less expensive: full serialization and compressing like now, or copying the memory? If there’s no non-blocking save, can we have a shorter freeze?
Copying RAM is orders of magnitudes faster than storing stuff to disk.
So yes, just copying the gamne state, then continue running the game while also doing the actual saving of the copy in background is a viable strategy to reduce the freeze to less than a second.
i dunno, copying once is less expensive. but i see your point. nothing's really going to get around the serialisation pause other than a full memory copy, and sometimes a memory copy pause might be preferred. of course, they tell me this is never going to happen before v1.0.

Rseding91
Factorio Staff
Factorio Staff
Posts: 13983
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: better compression algorithm for saves and mods

Post by Rseding91 »

Oktokolo wrote:
Mon May 11, 2020 8:48 pm
coppercoil wrote:
Mon May 11, 2020 6:48 pm
Ok, copying RAM is expensive. Which is less expensive: full serialization and compressing like now, or copying the memory? If there’s no non-blocking save, can we have a shorter freeze?
Copying RAM is orders of magnitudes faster than storing stuff to disk.
So yes, just copying the gamne state, then continue running the game while also doing the actual saving of the copy in background is a viable strategy to reduce the freeze to less than a second.
Currently the "write to disk" happens in parallel with copying. So it doesn't actually impact saving that much - about 15-20% savings if you just skip the write-to-disk completely. And a portion of that comes from not having to save the copied memory anywhere - meaning it would be slower still if it *did* actually have to save it fully in memory for it to finally get flushed to disk.

Reading all of those random bits of memory is that expensive (AKA: copying memory is expensive).
If you want to get ahold of me I'm almost always on Discord.

coppercoil
Filter Inserter
Filter Inserter
Posts: 494
Joined: Tue Jun 26, 2018 10:14 am
Contact:

Re: better compression algorithm for saves and mods

Post by coppercoil »

Rseding91 wrote:
Tue May 12, 2020 4:21 am
Currently the "write to disk" happens in parallel with copying.
Great, it seems it’s already optimized. Of course, I won’t stop thinking :mrgreen:
What if the copy/serialization would be performed slowly in the background without locking the game, leaving empty placeholders for some "hot" values, which would be stored separately in the final general lock? If copying is expensive, let’s copy just a fraction.
In other words, there can be frequently-changing values and rarely-changing values.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12889
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Bikeshedding about non-blocking saving on Windows

Post by ssilk »

Then you need to manage what you already have copied at which state and how actual it is.

Nothing more to say. ;)
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Bikeshedding about non-blocking saving on Windows

Post by ptx0 »

it's really not hard to run Linux, that's the real solution here. running a server on windows is a bit of a novelty since it doesn't even support basic server functionality..

User avatar
CheeseMcBurger
Long Handed Inserter
Long Handed Inserter
Posts: 75
Joined: Sun May 19, 2019 9:57 pm
Contact:

Re: Bikeshedding about non-blocking saving on Windows

Post by CheeseMcBurger »

ptx0 wrote:
Wed May 13, 2020 2:07 am
running a server on windows is a bit of a novelty since it doesn't even support basic server functionality..
What an ignorant statement.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12889
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Bikeshedding about non-blocking saving on Windows

Post by ssilk »

I think ptx0 means it in context of Factorio. Why do you mean he is wrong with it?
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

User avatar
ptx0
Smart Inserter
Smart Inserter
Posts: 1507
Joined: Wed Jan 01, 2020 7:16 pm
Contact:

Re: Bikeshedding about non-blocking saving on Windows

Post by ptx0 »

CheeseMcBurger wrote:
Wed May 20, 2020 10:02 pm
ptx0 wrote:
Wed May 13, 2020 2:07 am
running a server on windows is a bit of a novelty since it doesn't even support basic server functionality..
What an ignorant statement.
i would love for you to back that "insult" up with some evidence, as i've been trying to do all along.

if you ask someone who is used to POSIX programming standards - not having COPY ON WRITE makes Windows fundamentally broken for efficient memory management.

Post Reply

Return to “Ideas and Suggestions”