Re: Save in the background
Posted: Mon Jul 31, 2017 10:45 am
Why not?fechnert wrote:
To not talk about the implementation, but the idea about a quicksave that won't interrupt the game while playing would be very nice
www.factorio.com
https://forums.factorio.com/
Why not?fechnert wrote:
To not talk about the implementation, but the idea about a quicksave that won't interrupt the game while playing would be very nice
It'll suddenly double memory consumption as the orginal program continues updating the world, and then you have to account for all the file descriptors that are duplicated and making sure that nothign funny happens with that.mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
Well in MP server save only settings, it could be an option, when I host factorio game i have machine with 32GB RAM so uninterrupted play with save in background would be very nice even with additional RAM cost.ratchetfreak wrote:It'll suddenly double memory consumption as the orginal program continues updating the world, and then you have to account for all the file descriptors that are duplicated and making sure that nothign funny happens with that.mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
See this thread, "Use fork() on *nix systems for doing save game". The developers basically say that copying the entirety of the memory would make it very complicated to save, which I'm assuming would mean a much longer period of UPS drops instead of a very short freeze. Additionally, I assume that it would be very complicated to program.cpy wrote:Well in MP server save only settings, it could be an option, when I host factorio game i have machine with 32GB RAM so uninterrupted play with save in background would be very nice even with additional RAM cost.ratchetfreak wrote:It'll suddenly double memory consumption as the orginal program continues updating the world, and then you have to account for all the file descriptors that are duplicated and making sure that nothign funny happens with that.mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
There is a lot of misinformation in that thread.Jap2.0 wrote:See this thread, "Use fork() on *nix systems for doing save game". The developers basically say that copying the entirety of the memory would make it very complicated to save, which I'm assuming would mean a much longer period of UPS drops instead of a very short freeze. Additionally, I assume that it would be very complicated to program.cpy wrote:Well in MP server save only settings, it could be an option, when I host factorio game i have machine with 32GB RAM so uninterrupted play with save in background would be very nice even with additional RAM cost.ratchetfreak wrote:It'll suddenly double memory consumption as the orginal program continues updating the world, and then you have to account for all the file descriptors that are duplicated and making sure that nothign funny happens with that.mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
mrvn wrote:There is a lot of misinformation in that thread.Jap2.0 wrote:See this thread, "Use fork() on *nix systems for doing save game". The developers basically say that copying the entirety of the memory would make it very complicated to save, which I'm assuming would mean a much longer period of UPS drops instead of a very short freeze. Additionally, I assume that it would be very complicated to program.cpy wrote:Well in MP server save only settings, it could be an option, when I host factorio game i have machine with 32GB RAM so uninterrupted play with save in background would be very nice even with additional RAM cost.ratchetfreak wrote:It'll suddenly double memory consumption as the orginal program continues updating the world, and then you have to account for all the file descriptors that are duplicated and making sure that nothign funny happens with that.mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
All that garbage about synchronizing is exactly what fork() is solving. At the point where now the save function is called you fork(). At that point the game is in a consistent state for saving and all game data is unlocked. Then in the child you save and exit. In the parent you simply return and keep playing. The only new bit is that you need a back channel to report errors and have to reap the child. Easy enough to call waitpid once a frame while saving is ongoing. You can add a pipe to report save progress and error strings, which is also easy to do and to check once a frame. Or have a thread that does that in a blocking manner.
As for the memory overhead I can't say much there. But it only affects pages that are changed. I would assume a good portion of the memory used by factorio doesn't change every tick.
If you don't think that's correct, feel free to let him know. Teaching me won't improve the game.Rseding91 wrote:5 GB/s copying raw concurrent memory around sure. But that's not how actual programs are laid out in memory and we don't want to write out the entire contents of the processes memory to disk.SyncViews wrote:Something is wrong there, on an AWS T2 micro instance I get over 5GB/s on memcpy. There must be a lot of stuff going on around it, or operations not playing well with the cache and memory subsystem for Factorio to spend so much time "copying memory".Rseding91 wrote:
- 85% copying memory
- 10% compressing the save data
- 5% writing to disk
mrvn wrote:I'm always annoyed when trying to do something and the saving dialog pops up. And the more you play the larger the problem becomes because saving takes longer on larger factories.
Why not save in the background in a separate thread and let the game keep running?
"All" this would need to do is to make a snapshot of the game state and then save that while the game continues on the main data. Under Linux at least this is simple since you canto get a snapshot of the processes memory. This will affect performance and memory footprint somewhat during save but not stop the game. So UPS might drop while saving but the game won't stop dead.Code: Select all
fork()
Code: Select all
fork()
That is the main problem.pleegwat wrote:And even if all that is not prohibitive I believe windows doesn't have a fork() equivalent at all.
Rseding presents a false dichotomy - i.e. that the choice is either copy individual objects in the pause phase (long pause due to inefficient copy) or write the whole process memory to disk. Another option is an efficient copy in the pause phase, and then do copy #2 (serialization) in a different thread (or process) after copy #1 is done (and do compression and disk writing from that other thread as well).Jap2.0 wrote:If you don't think that's correct, feel free to let him know.Rseding91 wrote:5 GB/s copying raw concurrent memory around sure. But that's not how actual programs are laid out in memory and we don't want to write out the entire contents of the processes memory to disk.
This introduces a different problem, though, which is that if you copy the heap to a different location all the pointers in it become invalid (since they still point to the old copy, which might be changing), and the serialisation code needs to traverse the pointers to do its job. So, you can no longer write "normal" C++ code that accesses things directly, but must fix up all the addresses as you go along by the offset between the two copies of the heap - the code has to look pretty different.NotABiter wrote: Q: How do you do an efficient copy when you have all of these individual objects/allocations?
A: Any performant program that needs to quickly copy a significant portion of its sizeable state should properly manage its heaps. I.e. it doesn't matter if individual objects/allocations are in essentially random positions within a heap, because you don't copy the individual objects. Instead (right from their onset - at object creation/allocation) you put those game-state objects that will ultimately need to be copied+saved into one heap (or some known set of heaps that you keep all/most other data out of) and when it comes time to kick off a save you copy the whole heap. The heap should itself of course consist of whole pages (or "superpages" - e.g., 16KB, 32KB, or 64KB chunks), and whole pages can be copied extremely efficiently. (You also have to ensure the heap does not get too sparse over time or efficiency goes down from copying too much unused memory, but I imagine Factorio is essentially ideal in this respect since in normal play its heap requirements really only ever go one way - up.) Doing the copy this way (and then letting another thread finish the save work using that copy) should result in the pause portion of the save being *many* times faster than present - no special OS support required.
+1. I came here to suggest the exact same thing.eX_ploit wrote:Actually it's not good. Seems like you are loading all of the assets before showing main menu, while only a small minority of those assets are needed in main menu. You can just load those assets and then load everything else in background while player chooses what he's gonna play.On this new computer with normal graphics quality Factorio takes 9.84 seconds to reach the main menu. I think that's pretty good for a game these days
Don't feel like finding the exact quote, but someone said that wouldn't make sense because it takes about 2 seconds to start your game, if you have 15 seconds of loading to do left then you either have to not let the user load their game or give them a 15 second loading screen. Personally, it makes more sense to just get it over with in the loading screen.soryu2 wrote:+1. I came here to suggest the exact same thing.eX_ploit wrote:Actually it's not good. Seems like you are loading all of the assets before showing main menu, while only a small minority of those assets are needed in main menu. You can just load those assets and then load everything else in background while player chooses what he's gonna play.On this new computer with normal graphics quality Factorio takes 9.84 seconds to reach the main menu. I think that's pretty good for a game these days
Also benchmarking on the fastest computer you can buy is a nice exercise, but not practical. You probably have some stats about your user’s systems, taking a slow or average machine seems like a much better target for such optimization tests.
Anyway. I love the game and all the peeps working on it! Cheers!
I was aware of this issue but considered it not even worth mentioning in my post. It isn't really a "problem". The pointers don't become "invalid" except in some uselessly narrow idea of validity. It only takes some very simple pointer math (just a fixed/constant offset assuming you manage your memory correctly) to properly adjust the pointers for their new location, and that pointer math will have essentially zero impact on performance (which is going to be dominated by memory latency).torne wrote:This introduces a different problem, though, which is that if you copy the heap to a different location all the pointers in it become invalid
A simple offset adjustment (likely implemented as a locally-defined short-named template function to make invocations relatively painless and unobtrusive) doesn't qualify as "has to look pretty different". The code would be identical except for additions of invocations of said function.torne wrote:the code has to look pretty different.
It already works this way. As the game state is copied out it's written to disk in a different thread. By the time the copy finishes the compression and writing to disk has already finished as well.NotABiter wrote:Rseding presents a false dichotomy - i.e. that the choice is either copy individual objects in the pause phase (long pause due to inefficient copy) or write the whole process memory to disk. Another option is an efficient copy in the pause phase, and then do copy #2 (serialization) in a different thread (or process) after copy #1 is done (and do compression and disk writing from that other thread as well).Jap2.0 wrote:If you don't think that's correct, feel free to let him know.Rseding91 wrote:5 GB/s copying raw concurrent memory around sure. But that's not how actual programs are laid out in memory and we don't want to write out the entire contents of the processes memory to disk.
Because the copy finishes before the full save process finishes. There's additional things that happen after the data copy finishes to fully finish the save meanwhile the thread is able to write out the remaining data.featherwinglove wrote:I'm khan-fewzed. If Copy #2 (compression & writing to disk) needs Copy #1 (copying game state to other memory) to finish before it can begin, how can Copy #2 be done by the time Copy #1 ends?
Rseding91 wrote:It already works this way. As the game state is copied out it's written to disk in a different thread. By the time the copy finishes the compression and writing to disk has already finished as well.NotABiter wrote:Rseding presents a false dichotomy - i.e. that the choice is either copy individual objects in the pause phase (long pause due to inefficient copy) or write the whole process memory to disk. Another option is an efficient copy in the pause phase, and then do copy #2 (serialization) in a different thread (or process) after copy #1 is done (and do compression and disk writing from that other thread as well).Jap2.0 wrote:If you don't think that's correct, feel free to let him know.Rseding91 wrote:5 GB/s copying raw concurrent memory around sure. But that's not how actual programs are laid out in memory and we don't want to write out the entire contents of the processes memory to disk.
According to what you yourself have said, no, it doesn't work the way I described - Factorio doesn't do the most important part. That is, during the pause what Factorio does is a bazillion inefficient tiny (object-level) copies when what it should be doing is a small number of large (heap-level) copies. The pause time would be many times shorter if it did the large copies rather than all of those little copies.Rseding91 wrote:It already works this way.
Windows has a VirtualProtect() API that allows you to mark pages as copy on write (PAGE_WRITECOPY). I've used it before. Works nice; certainly a lot faster than fork() which has to copy the page table (vs. setting a flag on a single entry in the process's Virtual Address Descriptor tree). Mac & Linux also have an equivalent API: mmap() - and the flags are extremely similar, since virtual memory is hardware accelerated and has been for a long time. If you're lucky, there's a _mmap() API you can call in Windows as well; but I haven't looked into that.kovarex wrote:That is the main problem.pleegwat wrote: And even if all that is not prohibitive I believe windows doesn't have a fork() equivalent at all.
I suggest implementing a custom pointer wrapper template class that uses the address where the pointer is stored (or uses thread local storage) to determine what offset to add when dereferencing. The code doesn't even need to look different - just a quick change of the types of any pointers in headers. Kind of like STL's smart pointer classes. Overhead's minimal since you're going to choke on memory access latency anyway.torne wrote:This introduces a different problem, though, which is that if you copy the heap to a different location all the pointers in it become invalid (since they still point to the old copy, which might be changing), and the serialisation code needs to traverse the pointers to do its job. So, you can no longer write "normal" C++ code that accesses things directly, but must fix up all the addresses as you go along by the offset between the two copies of the heap - the code has to look pretty different.
Even if you managed to get a magically instant copy of the gamestate running in a separate thread/process which would allow the game to continue, Factorio's hunger for memory bandwidth would mean you've just created a new problem: UPS drops for a time while the save runs in the background. Since this magic free copying ability doesn't even exist, you would actually be adding more work so saving would be more painful overall.NotABiter wrote:According to what you yourself have said, no, it doesn't work the way I described - Factorio doesn't do the most important part. That is, during the pause what Factorio does is a bazillion inefficient tiny (object-level) copies when what it should be doing is a small number of large (heap-level) copies. The pause time would be many times shorter if it did the large copies rather than all of those little copies.Rseding91 wrote:It already works this way.