Friday Facts #201 - 0.15 Stable, but not really

[Jap2.0]

Summary of the last few posts:

It is technically possible to make this work. However:

• - It would push the release of 0.16 back to 2019.
  - It would be overly convoluted, difficult to implement, and even more difficult to make work across multiple operating systems.
  - It could cause a whole host of bugs regarding saving on different operating systems and make solving bugs regarding save games much more difficult.
  - It would reduce the time it takes a normal base to autosave from ~0.03 seconds to ~0.02 seconds ( in other words, it might improve it but wouldn't eliminate the very short hiccup on autosaves).
  - It would lower UPS for much longer than the current implementation.
  - It would make 2/3 of the forum stop arguing and make another 2/3 of the forum start arguing (and yes I know math).
  - It would let the developers stop teaching the forum about how saving works for a few hours, until they have to start doing it again because no one would understand it after the update either.

Anything I missed?

[featherwinglove]

Summary of the last few posts... Anything I missed?

Just about everything, but it's still funny

[kreatious]

Another idea, (instead of mmap()/VirtualProtect() shenanigans) is to implement this concurrent algorithm for saving in the background.
It's based on the concurrent mark & sweep algorithm sometimes used by the JVM garbage collector, used to minimize stop-the-world pauses.

1) Main thread decides its time to save, and starts (or reuses) a background thread.
2) Background thread marks everything as "clean".
3) Background thread serializes the game state while the game is still running. Any time the main thread changes a piece of data, it is marked "dirty". No synchronizing!
4) Background thread asks the main thread to suspend its game loop by atomically writing the address of a signal to a volatile field, and blocks on that signal.
5) Main thread suspends the game loop (because the signal was != null).
6) Main thread interrupts the user with a progress bar after 300ms (So the user isn't interrupted unless its a busy factory).
7) Main thread writes the address of its signal to a different volatile field, signals the background thread, and blocks on its signal (side effect: a memory fence is issued).
8) Background thread updates its serialization of everything marked "dirty". No synchronization is done.
9) Since we can't ask mods to be dirty, this is when the background thread saves that data, along with everything else that doesn't support being dirty.
10) Background thread signals the main thread to continue.
11) Background thread compresses & saves the game state to disk.

The save game that's created will be consistent with the state the game was in step 5 (Yay ). Most of the copying is done in the background, and there's no fancy things with virtual memory (and we don't copy memory twice, so it'll actually be faster!). This algorithm should prove to be much faster as well - a factory only covers a small fraction of the chunks that have been revealed. Because of that, I'd expect at least a huge improvement on how long users are interrupted.

"Dirty" objects could be added to a vector when the dirty flag gets set for the 1st time. Objects to be deleted can also be added to a vector and then deleted from memory after the background thread is done, or immediately if it's already done.

The biggest hurdle I see is that the game's code will have to ensure that the background thread can never possibly see an object in an invalid state. We don't want the background thread to take a branch based on one bit of memory to read some bit of memory, while the main thread flips the result of the branch and causes a crash. Things like if (ptr != null) happening while ptr is being set to null. If that's too much, then the Main thread can do all the work during its idle time while waiting for the next update (and if there is no idle time, make some!)

Thoughts?

[featherwinglove]

I'm not keen on the autosave's little progress bar and lag spike disappearing entirely in single player. I use them as prompts for writing entries in logged games (e.g. I'm still writing Muddy Mountains; now at about Day 70) and high risk activities (I tend not to save manually to enhance risk under more normal circumstances.)

9) Since we can't ask mods to be dirty, this is when the background thread saves that data, along with everything else that doesn't support being dirty.

Somehow, I don't think this would be necessary since the clean/dirty and shenanigans.suspend = true; stuff is implemented in not-mod-visible C++ or otherwise compiled code, and not in the Lua script. This is also true of most/all vanilla entities. If the Lua doesn't care about this at all, it will make absolutely no difference at all whether an entity to be marked dirty is stock or mod.

[NotABiter]

Factorio's hunger for memory bandwidth would mean you've just created a new problem

No, Factorio doesn't have a "hunger for memory bandwidth". On any reasonable modern box Factorio only uses a small fraction of the available memory bandwidth. Factorio is memory *latency* limited.
That this is the case has been measured. (He's getting only ~25 fps so he's obviously pushing Factorio hard, yet Factorio is only using ~6% of his box's memory bandwidth. Factorio is not constrained by the hardware's memory bandwidth, nor by the CPU's functional units -- the game's *achieved* memory bandwidth (what it actually manages to use) is only so low because it is *very* memory-latency bound, almost certainly due to far too many random accesses.)

Since this magic free copying ability doesn't even exist, you would actually be adding more work so saving would be more painful overall.

I never said the fast copy would be free, I said it would be many times faster than the existing copy, allowing the pause time to be reduced by that same factor, and that remains true. "Many times faster" is not all that painful.
And yes, the scheme I described adds more work, and I explicitly said so (see paragraph starting "this does require two copy operations instead of one"), and I even preemptively explained why doing the 2nd copy *doesn't matter* because I knew there would be people here who wouldn't understand that if it wasn't explained to them. Sadly, some people don't understand it even with the explanation and prefer to replace measured facts with hearsay.

Anyways, I don't really care much if the devs speed up the save or not, but I do find the spreading of falsehoods about it annoying. (Who likes being told lies? Not I.)

(and we don't copy memory twice, so it'll actually be faster!)

[jodokus31]

The main issue i've got with the autosave is, that it sometimes feels, that my GUI input isn't committed.
F.e. I try to place an inserter, point to the destination and click and go to next position and ...., but it didn't happen, because autosave interrupted it.

It could be possible to show up the autosave progressbar a bit earlier than it really interrupts the user's input. So the user is notified, can react and wait without losing input.

Or, it could also be possible to record the user's GUI input while autosave and commits it after that, without interrupting the GUI

[bobingabout]

Although it is possible to construct rail system that results in our algorithm putting two of the same colors next to each other, it is not really a problem, as it doesn't break anything, and in a normal game, the chance of it happening is close to 0.

https://us2.factorio.com/assets/img/blo ... colors.jpg
You have a yellow, overlapping a yellow, overlapping a yellow.... that's not "near 0".

They are in the same block, so it is intended they are the same color

ah, makes sense.

[chridder]

to be honest, I have no problem with the autosave at all as it works currently.

1. Yes, it brings up a little break into your work, but this is a really effect, unless you have a mega factory
2. As factorio is already more stable as many other games, I just autosave inteval (instead of every minute, for example just every 10 min.)

Let the devs focus on features and things, that are really important for us players

just my 2 cents.

[cpy]

If only saving didn't stop action ques.
But instead it is hard rubber band.

[ratchetfreak]

All the talk about putting stuff in special locations of memory and then doing a full copy misses one little detail: creating and using custom allocators in the stl is hard. It's probably not easy for the lua contexts either

It also means they need to ensure that only the custom allocator is used for game state, and ensure that the contexts for scripts and mods (the globals) is also in that custom heap. This is a recipe for disaster as small things fall through the cracks and slowly get caught 5 major versions down the line.

using custom offset pointers is also not the best idea because then you add cost to every single dereference in normal play, in a game that is already bound by memory latency you don't want to add even more time before you can start each fetch.

On this new computer with normal graphics quality Factorio takes 9.84 seconds to reach the main menu. I think that's pretty good for a game these days

Actually it's not good. Seems like you are loading all of the assets before showing main menu, while only a small minority of those assets are needed in main menu. You can just load those assets and then load everything else in background while player chooses what he's gonna play.

+1. I came here to suggest the exact same thing.

Also benchmarking on the fastest computer you can buy is a nice exercise, but not practical. You probably have some stats about your user’s systems, taking a slow or average machine seems like a much better target for such optimization tests.

Anyway. I love the game and all the peeps working on it! Cheers!

Don't feel like finding the exact quote, but someone said that wouldn't make sense because it takes about 2 seconds to start your game, if you have 15 seconds of loading to do left then you either have to not let the user load their game or give them a 15 second loading screen. Personally, it makes more sense to just get it over with in the loading screen.

Then there is the scenario of when you launch the game only to start the update process immediately. All the loading time waiting on the disk is wasted because after the update is complete the game needs to do that all over again.

[cpy]

So I guess it's nearly impossible to do.

[Light]

My big issue with the initial load times is that after updating or toggling any mod, the game has to be restarted and take its sweet time loading up again. This makes updating mods a real pain in the ass when updates are released often, when it shouldn't be that cumbersome of an experience. It's even worse when it actually takes minutes to start the client and not mere seconds.

There are some games I can think of that can seamlessly update themselves while still running. This naturally prevents a great deal of frustration when said game does have long load times or a great deal of data to be downloaded at the time.

This makes me wonder if you've looked into a way to have mod data updated while the game is still up, as mods appear to load individually and thus gives the impression you can unload and reload them individually as well. If I'm going to update (or toggle on/off) a single mod, then I'd want to see the game unload the old version of that mod and load up the new version into the client without restarting and reloading literally every other mod alongside it. Just that one mod needs to refresh, not the entire game as a whole. It would greatly save a lot of time having to avoid loading all that other stuff again since it's already in memory.

I'm unsure how viable this is with how you've built the game currently, but it's something I'd really love to see discussed and tackled at some point since that's a major detriment to dealing with a larger array of mods and mod configurations.

[IronCartographer]

On any reasonable modern box Factorio only uses a small fraction of the available memory bandwidth. Factorio is memory *latency* limited.
That this is the case has been measured.

Well sourced; I was unaware this distinction had been measured. Thanks.

[Jon8RFC]

I really enjoyed the data and explanations on the various performance optimizations.

[bobucles]

Honestly? Just keep the fork() style save, and make it a Linux server only feature. End users are not required to save their state with the server, and the main reason to run a dedicated server is to have a mega base. So keep the feature exactly where it benefits most- autosaving on a huge base- with the least amount of pain.

I do think that loading the start menu first is a good priority. Jamming the load button is an advanced player thing and their wait time is the same no matter what. New players and modders have business in the game menu, and if that can be loaded in half a second then it should be. Connecting to a server also takes up menu time so every second of menu access can be a second saved.

[bobingabout]

Honestly? Just keep the fork() style save, and make it a Linux server only feature. End users are not required to save their state with the server, and the main reason to run a dedicated server is to have a mega base. So keep the feature exactly where it benefits most- autosaving on a huge base- with the least amount of pain.

I do think that loading the start menu first is a good priority. Jamming the load button is an advanced player thing and their wait time is the same no matter what. New players and modders have business in the game menu, and if that can be loaded in half a second then it should be. Connecting to a server also takes up menu time so every second of menu access can be a second saved.

what if your server is windoze based?

[bobucles]

That's just too bad I guess. The basic impression I'm getting is that the Linux implementation is very easy while the Windows option is very hard. Take the easy fix. If server admins REALLY want it then the option is there. For most everyone else running small split second saves it doesn't really matter.

[Rseding91]

It already works this way.

According to what you yourself have said, no, it doesn't work the way I described - Factorio doesn't do the most important part. That is, during the pause what Factorio does is a bazillion inefficient tiny (object-level) copies when what it should be doing is a small number of large (heap-level) copies. The pause time would be many times shorter if it did the large copies rather than all of those little copies.

The game isn't laid out in a way to make saving efficient. It's laid out in a way to make runtime performance efficient. Things that are used *runtime* are located near each other - most of that data doesn't get saved.

[pleegwat]

Since this magic free copying ability doesn't even exist, you would actually be adding more work so saving would be more painful overall.

I never said the fast copy would be free, I said it would be many times faster than the existing copy, allowing the pause time to be reduced by that same factor, and that remains true. "Many times faster" is not all that painful.

Making up some numbers on the spot, I'd hazard factorio's save serializes 1GB of runtime state (total allocated object size) into 200MB of uncompressed savegame into 100MB of compressed savegame. The actual serialization is irrelevant because main memory is much slower than the CPU, only the data structure iteration and serialized data write matters.
Additionally, I'd hazard all 250k pages in that 1GB runtime state are modified on every game tick.

The existing code copies 200MB out of the game thread while the game thread is paused.
Your approach copies 1GB of data out of the game thread while the game thread is running. Since the game thread does the modifications, the game thread incurs COW penalties. And since memory is the bottleneck this will still cause game slowdown.

I wouldn't be surprised if running gzip compression in the game thread instead of in a background thread would actually be faster.

[ohmusama]

Despite all the arguing about what is clearly and X Y problem, As a large map player, thank you for any small improvements in save & load times.