factorio optimization thoughts - is it even deterministic?

[craktorio]

playing factorio and hearing how well it is allegedly optimized, had me wonder many times about whether one huge potential for optimization has been used or not, now hear me out, this might be a silly idea given that I am not a programmer.

my assumption is that every game has to be calculated, and since it is all calculated, there is only one outcome, and given no user input, the game should always arrive at the same outcome given its inputs, I call this deterministic calculation, it's what I generally synonymize with computing.

but designing space ships and watching frames drops from all the asteroids made me wonder about one thing: does the engine of factorio actually calculate space ships and asteroids, or only when I am looking? how could the game get away by not calculating space ships when not looking? well, when not looking, there is no user input, so no live input variables which make time sensitive calculations necessary. furthermore a spaceship, like any subfactory, is a set of given variables to the engine and thus the cpu it runs on, which means why not skip the whole calculation, and merely guess the outcome given the known input variables?

now what do I mean by guessing in comparison to actual computation? well you know a set of fixed variables, like how many asteroids will come per timeframe, how many rocket towers you have on your space ship, you merely calculate the probability of anything hitting your ship, and if you defenses are good, and the probability is 0, why calculate asteroids at all, especially if you also know how many grabbing arms you put on the space ship, so you can guess how many chunks it will collect on average etc etc.

I call this probabilistic computing.

so here we have two terms, one is deterministic computing, the other is probabilistic computing. I guess most games have to run actual computing (deterministic ones) especially fps shooters as live input is crucial, and the engine has to directly respond to live user input. it can never guess what you will do next, it will have to update the whole engine and all of its variables every tick.

but factorio? especially space ships? hell no they don't have to be calculated deterministically, right? I guess one could save 99% of cpu, if not 99.9% or even 99.99% given this approach. now the big question: does factorio already do this? does factorio already not process factories or space ships, when I am not looking or when I am not there with the engineer to potentially mess things up?

what about biters? the game knows how they work, how they attack, their properties etc etc. if the game knows all this, and I am not looking, maybe working on a base on fulgora, does the game actually calculate biter attacks which to me have no random outcomes anyways, but expectable ones, and if so, why calculate them in detail, why not calculate them probabilistically?

does the game engine really update everything on every surface every tick? or does factorio skip the calculating for things that do not need to be calculated, but can rather be (close to) accurately guessed given the non-changing nature of a set of already known variables, i.e. guessing based on already known probabilities?

is the factorio engine actually already guessing or not? if not, what a huge performance improvement it could gain from stopping to calculate things which can be approximated instead (with the same outcome of course, in the long run)

what do I mean by "with the same outcome in the long run"?

well does it really matter if you calculate every variable of a system (like a spaceship or a subfactory) every 60th of a second, and you end up with x amount of legendary plates, or you guess the same outcome instead? you will end up with the same outcome, except in one scenario the CPU knows cause it computed this outcome and tracked its progress for every tick in the game, and in the other scenario it just gives you the legendary plates cause it knows "I would have gotten 256 leg plates if I had done the work given the variables in front of me". how does it know? it looks at the variables, checks inputs and outputs, knows this setup will yield x amount of leg plates, and then just updates the user account with x amount of legendary plates instead of actually "producing" them in a factory, which takes way more cpu.

idk, am I tripping balls here, am I missing something basic in computing? are my questions silly or stupid? and if not, is factorio already doing exactly this, or something very similar?

it all boils down to

TLDR: does factorio's engine, as part of extensive optimizations, sometimes or most of the time skip calculating the whole multi-planet world, and go over to guessing outcomes/outputs, instead of actually calculating every tick of its production lines?

this question came up way too many times while playing factorio for me not to finally mention it on here.
what this means if I am on to something here: 99.9% performance improvements for megabases. if the game does not need to calculate bases anymore, cause it knows all the values beforehand, it can merely simulate it. the only surface it would actually need to calculate, would be the one you are currently on, and you could be sitting on an empty spaceship while doing everything in remote view, or doing nothing at all.

to me the idea is fascinating. belts would not have to be calculated anymore, neither inserters. you just check the probabilistic algo for how much stuff goes on this belt on average, how much will the miners produce on average, how much input diminishes on average cause an ore patch gets depleted, how fast does it get depleted on average, greate functions and variables about this autimatically, use them to approximate the outcome 99.99% accurately compared to the computed version of game reality.

I'd take 99.99% accuracy with this probabilistic method over 100% accuracy but with 99% worse processed performance any day. what about you? if your calculated game gives you 1000 plates for running your perfectly designed factory, but in my game, you only get 999plates, cause the guesstimation algo is not 100% accurate, but you can build your base 100x bigger, say 100 million spm megabase with 60 ups on average cpu easy.

what would you pick? I'd pick the 999 plates every day of the week.

do you need your game to actually be deterministic or probabilistic?

am I talking nonsense? looking forward for a good discussion. would also me nice if the game makers could chime in on this, or at least point to some FFF about this, it is has been addressed before. I a way I can't believe a heavily optimized game which mostly runs without immediate user input (every surface you are not watching or moving on) is actually updated every tick of a second, instead of summarized to a known set of non-changing variables, and this formula being updated until the user might change something about this forumla (i.e. live user input) at which point the guessing needs to stop and the calculating needs to kick in again.

is this switching even possible? if so, it would be so nice to know it is happening in the background. if not, we'll see 100m+ spm megabases if it gets implemented. basically you calculate what you see, and what you might change within about a second as a user, and all of the rest you estimate or guess, and the guessing will be good enough so nobody would notice.

[Tertius]

The game computes an exact complete new game state every tick in a deterministic way. If you don't look, it's not being rendered, but it's still being computed. The headless server does this: full game state calculation but no rendering.

If you want to switch to probabilistic average calculation, you have to ask: how long should be the time frame to collect statistics so it's reasonably stable to extrapolate production? If you look at the production graphs, you need hours of continuous undisturbed production to get a really accurate prediction.

And often, you will see waves even for stable production: you cannot exactly say why, but you will see some output raises, then declines, then raises again, with no apparent reason because the input is always the same. But if you look deeper, it's because of the dynamics of interaction between inserters, belt and belt speed, stack size of input slots and machine speed. You will even out such waves, and this will make the factory different to what it produces when you're looking and when not.

The fascination of Factorio is the exact and full simulation, down to (for example) picking each single item from a belt. Did you notice picking up items from a belt depends on the inserter arm position, belt position of the item, belt speed and arm speed? You will take away the soul of Factorio if you start averaging stuff like the pick up process. And because of the super exact simulation, any averaging will be simply different to what you get if you let the complete simulation run.

This exact simulation gets surprising results often, that cannot be represented by averaging. This is what happens with real engineering as well. Engineering deals with the real world, not with an average of all factors. By averaging, you will be able to build production lines that works well while you don't look and get vastly different output while you do look. You can also cheat this way: make the engine believe some output is vastly higher on average than it actually is for low throughput items if you exploit statistical variance.

[craktorio]

thanks, good and clear answer. given the potential performance gains of something like averaging, especially for space platforms, I was 50/50 about expecting some kind of averaging having already been implemented. take my promethium ships for example, they do their stuff, nothing ever goes wrong, I completely can disregard them once designed well, they are also fully autonomous, it would be trivial to no compute them at all, at least I thought so.

again, the potential performance boost - at least in my mind - would warrant for a good look into this pseudo-computing tactic 100%. it's nice that modern cpus are so powerful they can run a megabase with 60ups ez pc, macs and linux.

from a computer-engineering point it would also be a nice challenge, like merging the two worlds without noticable lags or hiccups whenever the user switches his remote view to some distant space platform, at which point it has to be calculated again, not just averaged.

the required timeframes you mentioned are trivial, too. as long as belts and inserters - even though they produce inexplicable waves in megabase production - are after all deterministic, one should be able to model what they do, even in waves. I mean are they deterministic or is there some spaghetti code which makes sure they are essentially unpredictable when applied in mass?

not saying every part of the game should be attempted to be averaged, but cmon something like promethium collector ships, one could really boil them down to a formula and then just use the outcome of the formula over time, instead of running full calculation of every swing of every inserter on this ship.

btw is averaging even a thing in computing? what is the technical term for this? is it modelling? extrapolations?

think about asteroid defense on the promethium ship: say you built your defenses slightly lacking, so you get hit by a random asteroid once in a while. why not use spare cpu cycles to extrapolate the design of your slightly lacking ship, let it determine (calculate) that on average it will get hit by an asteroid every 5 minutes, and then just notify that it got hit, on average every 5 minutes, without even knowing it was, when you switch views, it quickly merges averaged world with calculated world, connects both and shows you one of the many possible aftermaths of an asteroid impact on your particular ship.

is this too much effort for the aforementioned performance gains?

you said something about the soul of factorio. well, hard to argue along those lines. I like what it does, or how it does what it does, on the other hand I would not mind 10x or 100x performance gains though, as I like megabases, too. cpus will get faster, but not 100x faster before 2040 or something. I can't wait this long. not saying I need this, just saying it could be nice.

[mmmPI]

there is this mod, https://mods.factorio.com/mod/ups_saving_quality_ships, from what i understand if your platform hub is of higher quality, the platform act as if it was several platform. You just take the result of one platform, and after all, you could have 2 or 3 doing the same thing, so you get 2 or 3 times the result everytime that one reference platform is calculated.

Obviously if you had 2 distinct platform , you could have them scheduled with opposite phase, so that one would be unloading science, while the other is chasing promethium, and if you are willing to extrapolate further you could imagine similar mechanism to have several copies but it's not the same as having several actual distinct platforms. Like if you have some refuel interrupts, with different platforms it would be naturally staggered throughout time, whereas if you extrapolate an average value and multiply by the amount of extrapolated platform, then when that one reference platform is going to refuel, it will also refuel as if it was "many" platform.

[h.q.droid]

"Where you look" does make some difference. It's confirmed in 2.1 and I strongly suspected it did in 2.0. Factorio is deterministic but view panning is significant.

2.1 added a feature that makes enemies move and fishes swim inside your remote view. Most likely, that means you can activate an otherwise sleeping chunk by looking at it. In the extreme case, from the same state you can trigger or avoid a combat by choosing to look at different chunks in a specific order.

Also, when reverse engineering the quality RNG, I found particle / glow animations (like turbine steam or heating tower fire) to be significant game-play-wise by sharing RNG state with quality and maybe turret aiming. If they were disabled for invisible chunks (I didn't check this part), you can change quality / combat outcomes by looking at different animated things. I remembered my promethium platform getting destroyed when I idled on Nauvis, but surviving just fine when I loaded autosave and remote-viewed to check exactly what hit it.

[Panzerknacker]

I think what OP is trying to do is not possible. Since basically all the factories on the different planets are connected in a certain way, they depend on timings of certain inputs and outputs. For example, a factory produces a science pack and the moment it is finished and shipped off to another factory is determined by the layout of that first factory.

Factories can be built in many ways and many things can go wrong which affect the timing of the inputs and outputs. You cannot just have a 'average output per minute' without doing the complete simulation because it would simply be incorrect.

And at the same time that is the whole charm of the game, knowing that everything is actually running in real time and that this iron ore you accidently drop on that copper ore belt is eventually going to clog the system and completely shutdown the entire factory.