Friday Facts #201 - 0.15 Stable, but not really

[IronCartographer]

Even if you understand most of Factorio's track signaling, there are MANY edge cases in Factorio that are hard to understand. The block visualization does NOTHING to help me figure out those edge cases, since I can already see block numbers by just clicking on the track. What I really need is a way to see what blocks are being reserved by trains as they move, so I can tell how the system REALLY works in use.

What about the arrows that show up when you hover over a train? Sadly, that doesn't help much. The problem is the arrows only show the path of the train. I need to see the ENTIRE block sections being reserved if I am to figure out all the edge cases.

This was the basis for my suggestion here: viewtopic.php?p=295065#p295065

[NotABiter]

Autosave related: Shouldn't the default autosave interval be just increased, as dying doesn't mean end of the game now?

I've been playing since 0.11.x and I thought your default auto-save setting was excessive even back then with death = death, and upping that interval is always one of the things I do right away after unzipping a new release. Every 2 minutes is really too frequent, commonly eliciting a "Wait, didn't it *JUST* do a save???" reaction. Note, though, that autosave isn't just for character death, it's also for game crashes, for horrible irreversible mistakes (e.g. accidentally deconstructing your base in map view when you thought you were deconstructing an outpost, accidentally setting off a nuke in your base, having your base get completely overrun due to an "oopsie" with the power, etc.), recovering from a less then stable computer crashing or locking up (which my computer was doing with increasing frequency until I finally cleaned the dust out of it), and for recovering from real-world power outages. That said, losing 5 or 10 minutes max isn't all that painful (and should happen rather rarely for the typical player now that you've got "continue") so I would suggest upping the default interval to at least 5 minutes, maybe 10.

[mrvn]

I won't claim to know much about fork(), so I might be misunderstanding it, and I won't claim to know how much of that thread is misinformation, but I would consider Rseding91 a reliable source and the following quote appears to be relevant:

• 85% copying memory
• 10% compressing the save data
• 5% writing to disk

Something is wrong there, on an AWS T2 micro instance I get over 5GB/s on memcpy. There must be a lot of stuff going on around it, or operations not playing well with the cache and memory subsystem for Factorio to spend so much time "copying memory".

5 GB/s copying raw concurrent memory around sure. But that's not how actual programs are laid out in memory and we don't want to write out the entire contents of the processes memory to disk.

If you don't think that's correct, feel free to let him know. Teaching me won't improve the game.

Those 85% listed as copying memory are actually the serialization with millions of function calls. It "copies" bits of data from all over the place in little bits and pieces. That's why you get nowhere near 5GB/s on that operation. With fork() on the other hand you just copy the page tables to mark memory copy-on-write. Page tables are roughly 0.2% of your memory used so they are comparatively small. Then, while the game runs, any page that is written to for the first time is copied. The whole page is copied with a highly efficient memory copy operation. That's where you get 5GB/s speeds. So it would take way less time than those 85%.

But lets say copying takes 85% of the time. That would still mean it runs in parallel with the game. You probably have to copy most of the belts in the first tick (less with the new belt optimization) so that would suck. But a lot of the map looks static on a tick by tick basis and might not have to be copied at all since it doesn't get written to during the save.

[cpy]

Not dropping user inputs during save could be a place to start. But if i had to choose, i'd choose main loop update threading anytime.

[NotABiter]

With fork() on the other hand you just copy the page tables to mark memory copy-on-write.

Clarification: At a hardware level, the pages are not marked copy-on-write -- they are marked read-only. The OS remembers internally (purely in software) that these pages are to be treated as copy-on-write.

That would still mean it runs in parallel with the game.

No, that is incorrect. While the serialization work would execute in parallel with the game thread, the copy(-on-write) operations do not run in parallel with the game thread. The sequence that actually happens is:
1. fork() marks all the pages as read-only
...
2. the game thread tries to update (write to) some memory (in a page it hasn't written since step #1)
3. that write operation page-faults (fails)
4. the hardware suspends the game thread and invokes the OS page fault handler (interrupt 14 - #PF)
5. the OS page fault handler then examines the page fault information provided to it by the hardware and determines that the fault is due to a write to a copy-on-write page
(The details the hardware provides include that this was a write, the address of the instruction that was trying to do the write, and the data address that faulted which for an unaligned write could be slightly different than the address that was written, along with various other bits and bobs. The hardware does not provide the OS information about what value was being written.)
6. the OS copies the page and adjusts the game thread's page table to have a read-write page table entry pointing to the new copy
(It could in theory change everyone else to point to the new copy instead, but that would in general be more costly.)
7. the OS returns execution back to the game thread (which has been suspended for the entire copy operation)
8. the game thread reissues the write that previously failed, and this time it works

As you can see, the entire copy-on-write operation is serialized with the game thread, not in parallel. fork() is a low-performance solution (relative to the scheme I suggested) because:
1. For every single page copied you have to pay the price of a page fault, including two context switches (which trashes your cache), not to mention time spent actually executing the OS code and updating the page table. (There are no page faults in my scheme.)
2. That copying is single threaded. That's bad for performance because it generally takes more than one thread to fully saturate the memory system of a decent gaming box, so the copy will be much slower than necessary. (My scheme allows for a multithreaded copy operation.)
3. Data that doesn't need to be copied will be copied. E.g. when the game goes to do graphics and sound work that will involve writing to memory which will then trigger copy-on-write. That data doesn't need to be copied though because it's not used by the save-game thread. (My scheme allows in-process separation of such data so it doesn't get copied. To do the same with fork you'd have to move all such data to yet another process which would then likely have a significant, possibly crippling, performance impact on normal tick updates.)
4. It only copies single pages at a time, and single page copies are slightly slower than copying superpages due to page-bank misses in the DRAM. (My scheme allows copying superpages. Yes, this one is minor, but I hate wasting performance.)

Oh, and fork() isn't available on Windows. My scheme works under all OSes. Why is anyone still talking about fork?

(The devs aren't going to do any of this so the discussion is not useful for that. But I was once again "triggered" by people posting *false* information.)

[Tony88]

Why is everybody talking about speeding up the autosave? The real problem is that user input is trashed during the save. Even worse, the autosave can also have nasty side effects. For example, if I am drawing a deconstruction area when the save hits, the save totally cancels the rectangle I was drawing rather than restoring it after the save. All keyboard and mouse input should be saved and executed after the save completes.

Somebody else said the same thing as my earlier post in this thread (sorry I didn't track it down to quote it):

All this talk about optimizing startup brings this back to mind: Why not check for and install updates BEFORE spending time loading all that stuff, since it will all need to be loaded again after installing an update??? This reloading makes updating seem slower than it really is.

They also pointed out that the game restarts whenever a mod is updated or deleted, not just when the game is updated.
I suspect the reason the update check happens after all that loading is to allow using the main game's user interface for the update check. Can't the user interface be started in time for the update check, then load all that stuff after determining that no update is to be done? Or, write a small program to check and execute program updates, before starting the game?

[Sigma1]

On the topic of loading, as a mod developer it would be nice to be able to trigger some kind of reload in-game.

[Toawa]

Hmm.. A bit curious about one thing; you say you have ~84m calls to incremental save functions; are those straight-up calls (each with a stack wind/unwind) or are you using loops, TCO/TCR, etc?

[5thHorseman]

Why is everybody talking about speeding up the autosave? The real problem is that user input is trashed during the save. Even worse, the autosave can also have nasty side effects. For example, if I am drawing a deconstruction area when the save hits, the save totally cancels the rectangle I was drawing rather than restoring it after the save. All keyboard and mouse input should be saved and executed after the save completes.

Oh jeez I forgot about that. You picked a fairly innocuous example though. What about during editing of a blueprint you just made? Highlight the area, click on all the parts you don't want in the blueprint, pick a name, decide on 4 icons, set up 3 of them (each time clicking the icon box, clicking the tab, finding the picture, clicking it) and on the 4th one, autosave. EVERYTHING VANISHES. Start over. Do not pass go. Do not collect $200.

[NotABiter]

A bit curious about one thing

why you should not be curious about that

Even if you care about save-game performance, those things you bring up won't even budge the needle. My best guess is Factorio's save-game time (the serialization part which is when the game is paused) is something like 90% memory latency, and a good chunk (maybe half) of the rest is branch misprediction costs. The specific low-level mechanism used for method invocation is in the "< 1% performance impact" category.

Modern x86 accurately predicts "non-degenerate" 'ret's (via a special return stack buffer), so switching from indirect call (call/ret) to indirect jump (tail-call) won't do squat to reduce mispredicts. Modern x86 also has very tricky handling of the stack pointer so it can process multiple stack adjusts without having to serialize such operations the way it does with dependent chains of operations on regular data registers. The "Stack Engine" Intel uses has such crazy performance that you might not believe it if you're not familiar with the details. E.g., from here:

The stack engine can handle three additions per clock cycle. Consequently, no instruction will have to wait for the updated value of the stack pointer after a stack operation.

The typical way to reduce branch mispredicts when dealing with virtual functions (which is a typical way to do serialization functions - some sort of indirect branch to select the appropriate serialization function) is to group the objects by type (thereby grouping them by identical serialization function) which then helps out the branch predictor. (Another similar approach is dedicated loops, one per type/serialization-function, each loop using direct call, possibly inlined.) Maybe you would want to ask them if they do that, as that should at least have some small yet measurable impact on performance.

What about during editing of a blueprint you just made? Highlight the area, click on all the parts you don't want in the blueprint, pick a name, decide on 4 icons, set up 3 of them (each time clicking the icon box, clicking the tab, finding the picture, clicking it) and on the 4th one, autosave. EVERYTHING VANISHES. Start over. Do not pass go. Do not collect $200.

Yeah, but if we speed up the game save time, then it would be better because it would delete your stuff many times faster!
(related: I wish I could redo a blueprint *without* losing the name and the icons. I'm always forgetting something and end up doing the same blueprint 3 or 4 times before getting it right. Though neither of those issues is a big deal compared to the horribleness that is the current blueprint book. The old one was better. Having to do "linear search" over the blueprints in a book to use one without pulling it out of the book is insane, and not being able to organize blueprints in a 2D fashion like the old book supported is also a major step backwards.)

[Toawa]

A bit curious about one thing

Even if you care about save-game performance, those things you bring up won't even budge the needle. My best guess is Factorio's save-game time (the serialization part which is when the game is paused) is something like 90% memory latency, and a good chunk (maybe half) of the rest is branch misprediction costs. The specific low-level mechanism used for method invocation is in the "< 1% performance impact" category.

Modern x86 accurately predicts "non-degenerate" 'ret's (via a special return stack buffer), so switching from indirect call (call/ret) to indirect jump (tail-call) won't do squat to reduce mispredicts. Modern x86 also has very tricky handling of the stack pointer so it can process multiple stack adjusts without having to serialize such operations the way it does with dependent chains of operations on regular data registers. The "Stack Engine" Intel uses has such crazy performance that you might not believe it if you're not familiar with the details. E.g., from here:

The stack engine can handle three additions per clock cycle. Consequently, no instruction will have to wait for the updated value of the stack pointer after a stack operation.

The typical way to reduce branch mispredicts when dealing with virtual functions (which is a typical way to do serialization functions - some sort of indirect branch to select the appropriate serialization function) is to group the objects by type (thereby grouping them by identical serialization function) which then helps out the branch predictor. (Another similar approach is dedicated loops, one per type/serialization-function, each loop using direct call, possibly inlined.) Maybe you would want to ask them if they do that, as that should at least have some small yet measurable impact on performance.

I'll admit, after I posted that I thought to myself, "that was a bit silly; if they're getting 300k calls per ms, then that's probably not an issue". I was still curious, though; the language I do most of my work in doesn't give that fine a control on the instructions generated, so I'm always kinda wondering. But bear in mind, the last time I really studied architecture was in college in the early 2000's, and at that time, it was a big deal.

That being said, there's no need to be snide about it.

[NotABiter]

That being said, there's no need to be snide about it.

I provide a sincere and informative response and in return you accuse me of having behaved badly somehow. I'm pretty sure that was not necessary.

[Aprillion]

How does the rail segments example RECOLOR after placing the signal? Does the incremental recolor mean it will add a 4th color here or it will change 1 of the blues to yellow?

[abordoli]

I can only imagine that if the devs are/were experiencing development delays/stalls/wasted-time-waiting due to "the state of affairs of the way things operate", then they MUST be Saints because I'm at my wit's end with just my experiences being on the user end.

I love the game. It is a masterpiece in concept. Here's hoping that the eloquence and ease of design and usability (for both devs and users) catches up...

Cheers!
~B

[featherwinglove]

Why is everybody talking about speeding up the autosave? The real problem is that user input is trashed during the save. Even worse, the autosave can also have nasty side effects. For example, if I am drawing a deconstruction area when the save hits, the save totally cancels the rectangle I was drawing rather than restoring it after the save. All keyboard and mouse input should be saved and executed after the save completes.

...Um, this.

Though neither of those issues is a big deal compared to the horribleness that is the current blueprint book. The old one was better. Having to do "linear search" over the blueprints in a book to use one without pulling it out of the book is insane, and not being able to organize blueprints in a 2D fashion like the old book supported is also a major step backwards.)

And this.

[mrvn]

With fork() on the other hand you just copy the page tables to mark memory copy-on-write.

Clarification: At a hardware level, the pages are not marked copy-on-write -- they are marked read-only. The OS remembers internally (purely in software) that these pages are to be treated as copy-on-write.

That would still mean it runs in parallel with the game.

No, that is incorrect. While the serialization work would execute in parallel with the game thread, the copy(-on-write) operations do not run in parallel with the game thread. The sequence that actually happens is:

The *it* above refers to the saving operation and it's 84 million function calls.

I've recently been playing with AAI mods and after marking a large zone (800k tiles) the saving takes about 10 minutes. For all that time except, maybe the first 10 seconds, I could haven been playing in parallel to the autosave.

As the devs said the copying is not the part that takes time. It's the 84 million serialization calls. So I'm not worried about the copying running in sequenze with the game thread. So everything you wrote after misunderstanding what *it* refers to, while true, becomes irrelevant.

[ratchetfreak]

As the devs said the copying is not the part that takes time. It's the 84 million serialization calls. So I'm not worried about the copying running in sequenze with the game thread. So everything you wrote after misunderstanding what *it* refers to, while true, becomes irrelevant.

except the amount of memory copied when using fork is a order of magnitude or two greater than the copies done by the api calls. So it probably will end up taking significant time.

[bobucles]

except the amount of memory copied when using fork is a order of magnitude or two greater than the copies done by the api calls. So it probably will end up taking significant time.

That will spike system demands for sure. In the end we're talking about saving a few seconds of game time every few minutes. It's a nice thing, but the huge system demand and marginal benefit just don't make sense in the regular scheme of things. It's a niche advanced option at best.

[abordoli]

I'm auto-saving every 30 minutes and, tbh, I don't even know why I'd save that frequently. I DO gather that the longer "in between" saves, the longer the save operation will be. I'm quite betwixt as I'd rather saves not bother me at all, but then again..."a stich in time...saves nine" keeps nagging at me.
~B

[mrvn]

except the amount of memory copied when using fork is a order of magnitude or two greater than the copies done by the api calls. So it probably will end up taking significant time.

That will spike system demands for sure. In the end we're talking about saving a few seconds of game time every few minutes. It's a nice thing, but the huge system demand and marginal benefit just don't make sense in the regular scheme of things. It's a niche advanced option at best.

Think minutes, not seconds. Obviously for small games where saving takes just a second the whole thing is irrelevant. I think I mentioned I had saves take 10+ minutes. Arguably that is an extrem. But saving taking 20-60 seconds is not uncommon.

And remember that copying a page of memory is also very fast. I don't remember the exact number but something silly like 5 random memory accesses (different cache lines) in a page take as long as copying it.