Friday Facts #201 - 0.15 Stable, but not really

[NotABiter]

The game isn't laid out in a way to make saving efficient. It's laid out in a way to make runtime performance efficient.

I really don't think you're going to convince me that's not a false dichotomy, especially when Factorio's memory bandwidth numbers show that it is not "laid out in a way to make runtime performance efficient". (Laid out in a way that improved efficiency relative to some earlier version of Factorio? Sure. Efficient enough? Perhaps. All the way to an objective measure of "efficient"? Nope - not when 90% of the memory bandwidth and most the CPU cores are still sitting on the floor.)
Anyways, I also don't believe it's worth my time, nor yours, to engage in such convincing attempts (and what I can say to support my case is ultimately limited by the fact that I can't divulge "proprietary information" on how the game engine I'm working on works), so I'm not going to pursue this topic further.

Your approach copies 1GB of data out of the game thread while the game thread is running. Since the game thread does the modifications, the game thread incurs COW penalties.

No, that is incorrect. I explicitly argued against using COW and my scheme doesn't require it. Your understanding of memory performance is also too naive to make a sound performance analysis - go read and understand What Every Programmer Should Know About Memory and then maybe you'll get why your "memory is the bottleneck" statement is too imprecise to either usefully describe the problem or determine what solutions would be effective.

The main issue i've got with the autosave is, that it sometimes feels, that my GUI input isn't committed.

I agree - that has caused me much more annoyance (though I guess I'd still only rate it "mild level annoyance" and it is only an issue at all occasionally, not every save) than the save times and it should be so much easier to improve. (I suppose save times annoyance could increase due to 0.15.x encouraging larger bases - I'm only now heading towards "megabase" territory for the first time ever because in prior versions there seemed no point to it, so I may see longer save times than ever before and reach "new levels of annoyance". But then the save-time improvements Rseding already did should at the very least offset that to some degree once 0.16.x is out.) To expand on the pause UI issue a bit, one thing that makes it more annoying is that you simply don't know if what you did "took" or not, so the first thing you have to do after the save finishes is inspect what you were doing to determine its post-save state to then determine whether or not you have to redo what you just did. (In general doing the redo blindly doesn't work because in a lot of cases a 2nd click could actually undo whatever was done, or open some unwanted window, or something else annoying.) That then leads to a more "intent" waiting (an additional psychological factor) because you're waiting not just to continue playing but to see something in particular (something that is often hidden by... the pause pop-up).

I think the "game-save early warning" might (I'm not sure - it's possible I might actively dislike it) also be nice for having some chance to get a character away from danger - e.g. if navigating train tracks. There can sometimes be some "clumsiness" during the recovery period after a save completes, and in that case it's best not to be too close to a 40 ton vehicle moving at 270 kph.

[kreatious]

Short of optimizing the save process by concurrently* serializing the non-changing stuff before pausing the game and bringing it into a consistent state, it would be nice if the game buffered the player's inputs while the game is paused, and replayed them after its unpaused.


* or doing it during the time the main thread's waiting for the next tick; a previous post of mine outlined a simple algorithm on how to do it.

[featherwinglove]

In the early days computers were much simpler. The var-
ious components of a system, such as the CPU, memory,
mass storage, and network interfaces, were developed to-
gether and, as a result, were quite balanced in their per-
formance. ...
This situation changed once the basic structure of com-
puters stabilized and hardware developers concentrated
on optimizing individual subsystems.

Bullshit historical contexts as fictional as Star Wars opening crawls preface quite a few decent technical papers, but I am still not impressed. First, define "early days". The 1990s? '80s? '70s? '60s? '50s? (Oh, the mix of hardware nonsense in the post-WWII era is thoroughly hilarious!) The 2nd century BC during which the Antikythera Mechanism was developed? Creation week? (About the only possibility where this summary is accurate!)

I'd like to elaborate a bit on NotABiter's third paragraph: What really annoys me is that there is one frame prior to the autosave where my inputs are not accepted, but the game is still updating and save bar has not yet appeared. It's the lack of consistency that pops my bubble: the game isn't quite stopped for the autosave, but is half stopped in such a way that is very confusing and causes errors. A lot of this could be solved by fully executing that last tick and all user inputs made during it to the game state and autosave.

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

[Rseding91]

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

Something like:

"Auto-save will happen in 5 seconds, press <TAB> to continue"

Then 1 second later

"4 seconds to autosave <TAB> to continue"
...
3
...
2
...
1
...
"Autosave starting <TAB> to continue"

?

[kreatious]

Personally, I'd prefer interrupting the user. Buffering user input & stalling on a single tick for a half second would be less noticeable than the current state of affairs.

Can't the game serialize the non-changing stuff (dirty flags) while the main thread's waiting for the next update tick?

[bobucles]

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

...
"Autosave starting <TAB> to continue"

?

Good work! Players need time to read the warning, so pausing the game lets them be fully aware of what's happening.

The entire process of splitting the game data and saving the frozen state without pausing the main game represents a considerable performance issue. The feature isn't even very meaningful until a base gets exceptionally large, at which point you're talking about a rather large performance kick for a feature that amounts to "a nice thing". If a computer isn't good enough it may simply chug along as it attempts both actions at once. There is also a considerable spike in RAM usage as well.The game could lag. The RAM usage could dip over capacity and into HD cache. In a poor circumstance it may even cause a crash. All of these issues point to a harsh truth that it can't exist as an ordinary option.

If the devs choose to try it out don't be afraid to keep it super experimental or only existing on the versions that can be easily implemented. It's not a core feature and the current system isn't that bad for ordinary use. If forked saving works out, great. If not, you didn't lose much sleep over it.

[5thHorseman]

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

Something like:

"Auto-save will happen in 5 seconds, press <TAB> to continue"

While funny, i do not think that's the idea featherwinglove had. Just a simple "Autosave in 3, 2, 1" for 3 seconds, without interrupting what you are doing, would be a boon for me when I'm working on something. Getting interrupted by autosave is a pain, as this thread has shown. Simply being aware of it's coming may be all many of us need.

Similar to yellow lights at intersections. They allow us to speed up and get through quickly slow down and pause comfortably, so we are not stopped in a more sudden and violent manner by the sudden change from green to red.

[dee-]

Would it help to reduce the memory latency by converting some/most of the pointer indirections between items and objects in memory to a more array-like structure, over which the logic and the memory access can just iterate instead of jump around?

And once you have a, say, 1024-objects-in-a-array slab and need more, you can link them together to a list in case of more needed entries - or even reserve new memory with n*2 and copy over the n items from the smaller array into it (see: Java Vector).

Combine this with a "deleted" flag to reduce copy shuffle in case object are not modified but even removed (see: "deleted" flags on database entries) and simply skip them when iterating the array in search of new things to do.

Depending how many moving or sorting needs to be done on the objects, it's also quite easy to move the items one slot up or down with a simple memcopy, moving a bulk of objects in one go. Or mark moved items as deleted and append them/insert them where needed. Then do a defragmentation step every one a while (w.g. just before writing to disc) to reclaim memory and purge deleted items.

Serializing these blocks to disk is also very cheap as it's just writing a single-piece chunk of memory. Not using pointers also removes the need to handle dangling pointers on (de)serialization.

(0,00000002 €)

[ratchetfreak]

Would it help to reduce the memory latency by converting some/most of the pointer indirections between items and objects in memory to a more array-like structure, over which the logic and the memory access can just iterate instead of jump around?

And once you have a, say, 1024-objects-in-a-array slab and need more, you can link them together to a list in case of more needed entries - or even reserve new memory with n*2 and copy over the n items from the smaller array into it (see: Java Vector).

Combine this with a "deleted" flag to reduce copy shuffle in case object are not modified but even removed (see: "deleted" flags on database entries) and simply skip them when iterating the array in search of new things to do.

Depending how many moving or sorting needs to be done on the objects, it's also quite easy to move the items one slot up or down with a simple memcopy, moving a bulk of objects in one go. Or mark moved items as deleted and append them/insert them where needed. Then do a defragmentation step every one a while (w.g. just before writing to disc) to reclaim memory and purge deleted items.

Serializing these blocks to disk is also very cheap as it's just writing a single-piece chunk of memory. Not using pointers also removes the need to handle dangling pointers on (de)serialization.

(0,00000002 €)

Putting things in arrays assumes that the things are accessed in a consistent order every time for all time. This is not true. After a while update/access order of entities tend towards indistinguishable from random as they sleep for some time until an inserter gets to put in the resources in response to robot dropping them off/belt arriving, output being unblocked.

[featherwinglove]

Personally, I'd prefer interrupting the user.

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

Something like:

"Auto-save will happen in 5 seconds, press <TAB> to continue"

...
1
...
"Autosave starting <TAB> to continue"

?

You guys are nuts

I've never had EvoGUI make me press Tab. I hope you got that.

Buffering user input & stalling on a single tick for a half second would be less noticeable than the current state of affairs.

Can't the game serialize the non-changing stuff (dirty flags) while the main thread's waiting for the next update tick?

Um... isn't the dirty flag for the changing stuff?

[kovarex]

Autosave related: Shouldn't the default autosave interval be just increased, as dying doesn't mean end of the game now?

[Sigma1]

Autosave related: Shouldn't the default autosave interval be just increased, as dying doesn't mean end of the game now?

I wouldn't mind that, the constant autosaves can get a bit annoying if you forget to set the interval. Especially now that you can respawn.

[looney]

While we're on the subject of stealing focus from user events..... The same happens in MP with jitter / packet loss. Why can I only edit the chat box text when the server is responding? It's rather annoying to lose a few characters of typing due to lag.

[Slada]

Can't you just load main menu first while loading the rest of game in the background? Navigation in the main menu takes time, why not to use this time to load the game?

[Sigma1]

It's rather annoying to lose a few characters of typing due to lag.

This is incredibly annoying, especially when typing a longer message or command.

[bobucles]

Autosave related: Shouldn't the default autosave interval be just increased, as dying doesn't mean end of the game now?

Awwwwww, but that makes sense!

Yes, the main reason for autosave was to reduce the punishment of setting up stuff and then running into a surprise train. But if you respawn it's no big deal, unless your main base is 10km from spawn. Then it's a bit of a jog.

Incidentally should a player be able to change their spawn location?

[Nosferatu]

Autosave related: Shouldn't the default autosave interval be just increased, as dying doesn't mean end of the game now?

Strange for some reason I never realized that you can use the Continue Button after the "Game finished - You have been defeated" Screen.
Maybe rename the whole Game finished dialog - it also felt like "This is the end" for me when I launched my first rocket.
There should also be some indication that you can go back to your corpse to get your inventory back.

[jodokus31]

As for "game save early warning", I run EvoGUI. I think the best implementation would be a litte countdown timer to the next autosave on a display similar to that in EvoGUI.

Something like:

"Auto-save will happen in 5 seconds, press <TAB> to continue"

While funny, i do not think that's the idea featherwinglove had. Just a simple "Autosave in 3, 2, 1" for 3 seconds, without interrupting what you are doing, would be a boon for me when I'm working on something. Getting interrupted by autosave is a pain, as this thread has shown. Simply being aware of it's coming may be all many of us need.

Similar to yellow lights at intersections. They allow us to speed up and get through quickly slow down and pause comfortably, so we are not stopped in a more sudden and violent manner by the sudden change from green to red.

+1

Would like it, too. Maybe toggable in options, because it's needed more, when the map got bigger or some might dislike it.

[Sigma1]

some might dislike it.

Just make it as a small notification in the corner and it shouldn't annoy anyone.

[Kevin Ar18]

Regarding the rail block visualization... While it is nice to have, it is really not helpful in understanding Factorio's block system. What I desperately need is a button that shows what blocks are being reserved by trains as they are moving, like this:


from: http://forum.simutrans.com/index.php?topic=17060.0

Why?
Even if you understand most of Factorio's track signaling, there are MANY edge cases in Factorio that are hard to understand. The block visualization does NOTHING to help me figure out those edge cases, since I can already see block numbers by just clicking on the track. What I really need is a way to see what blocks are being reserved by trains as they move, so I can tell how the system REALLY works in use.

What about the arrows that show up when you hover over a train? Sadly, that doesn't help much. The problem is the arrows only show the path of the train. I need to see the ENTIRE block sections being reserved if I am to figure out all the edge cases.