Optimization idea: abstraction

[ssilk]

I wonder how the game should work without them. Factorio’s magic happens exactly there, when things are linked together.

[KoblerMan]

The whole abstraction thing is a neat idea on paper, but I really don't see it as being practical, and it seems out of scope for a game like Factorio anyway. That's even assuming the whole idea is compatible with the game engine, which hasn't been built for it. IMO it would create more problems than solutions.

I think Wube has done a fantastic job of optimizing their game as it is, in reality I would argue abstraction on this scale is unnecessary.

[Optymistyk]

Tbh I was sure this is already in place cuz the game is just so well-optimised and it makes so much sense.

Every production line works in a deterministic manner so it should be possible to express everything as a "lazy" function - only calculated the moment a player checks it

But it needs a lot of preconditions to make sure, the simulation is always exact:
- all input belts are always compressed
- enough free space on output belts to put items on
- inserter are fast enough to feed/empty the smelters
- there is no complex belt stuff, like when it is possible that two different items could be on one lane
- same is with mixed input lanes. As soon as another item on a lane is imputed it needs to switch back to simulation.
- there are no circuits involved
- there is always 100% power available (well I think this can be simulated up to some degree)
- pipes are never filled below some level X
- player is not changing anything inside that factory street are
- the theoretical input/output is equal to the current input/output
- and of course some more preconditions.

None of those are in fact necessary.
- If the belt is not compressed you still know the rate at which items are placed on the belt by the inserters
- If the rate at which items are placed on the belt exceed the belt throughput then it is limited to the belt throughput
- If the inserters are not fast enough then the smelters produce proportionally less items
- It can still be expressed as a mathematical function
- It can still be expressed as a function
- Circuits are literalry just functions
- If there's less than 100% power everything slows down proportionally
- If the liquid flows at a rate lower than the required rate X then the production slows down proportionally
- When the player is nearby the abstraction switches to simulation, after the player makes changes the function can be updated
- Not necessary

It would be incredibly complicated to implement and any bugs could be completely gamebreaking but it would defo improve performance (which I think is fine tbh)

[ssilk]

Every production line works in a deterministic manner so it should be possible to express everything as a "lazy" function - only calculated the moment a player checks it.

Ahem, this is not correct. It works only deterministic under very specific conditions.

None of those are in fact necessary.

That depends on what you want to achieve!

If you want to have a correct simulation it’s wrong. If you want to make an abstraction, that is sometimes correct: maybe.

- If the belt is not compressed you still know the rate at which items are placed on the belt by the inserters

But only after some processing time.

- It can still be expressed as a mathematical function

It’s not deterministic under conditions outside of the function scope.

- Circuits are literalry just functions
- If there's less than 100% power everything slows down proportionally
- If the liquid flows at a rate lower than the required rate X then the production slows down proportionally

Combinators slows down, when they have not enough power. The result is not longer deterministic (in most cases). And the same is it with inserters and fluids, they don’t slow down proportionally and you forget the time it takes for an item to run through the process.

It would be incredibly complicated to implement and any bugs could be completely gamebreaking but it would defo improve performance (which I think is fine tbh)

Well, as said, only under special conditions this behaves deterministic. It’s not super complicated, but I admit it’s complicated and the result might be questionable, because it could be very complicated for a player to achieve the conditions under which that works deterministic.

[Optymistyk]

- If the belt is not compressed you still know the rate at which items are placed on the belt by the inserters

But only after some processing time.

- It can still be expressed as a mathematical function

It’s not deterministic under conditions outside of the function scope.

Well now you're making me do maths

Imagine a somewhat simplified example: There's a straight (infinite) line of yellow belt. A yellow inserter(no upgrades,) is placing items on the right side of the belt. You can express what the inserter is doing as a periodic function. If you want to calculate at what times t an item is placed:
cos(t+φ)=0, t∈[0,+infinity)

Each point where the graph crosses the t axis on the right side is the moment in time when one iron plate is placed on the belt. The function does so at any t∈(1/2π + φ, 3/2π + φ, 5/2π + φ,7/2π + φ...). 1π is the period T, or the time between the placement of each iron on the belt.

Since we know that a yellow inserter works at a rate of 0.83 items/second, π can be expressed as 1/(0.83 items/second * Power/100%) where P is constant Power; π =~ 1.2 seconds with full power. So if π = 1.2s then t∈(0.6s + φ, 1.8s + φ, 3s + φ, 4.2s + φ...). If Power levels are not constant then you have to track the history of power in the system, which Factorio is already doing cuz you can display a nice Power graph by clicking any power pole. Because most things work at a rate proportional to their power level, it might even be enough to just use an average instead.

φ is the Phase of the function, or the initial delay before the first placement. For now let's just say φ=0. Then t(0.6s, 1.8s, 3s, 4.2s...). Let's calculate s - the distance each plate has travelled from it's initial point of placement. s = v*(t0 - t), where v - the speed of the yellow belt belt(1.875 tiles/second), t0 - the time passed since the inserter started placing items. So let's say it's been 10 seconds since the inserter started placing items, then s∈(~17.6,~15.4,~13.1,~11...) tiles.

Okay, but the way it is now we have to keep track of every point of placement for every inserter and it's not too convenient. Let's instead count from the beginning of the belt for each inserter. For that let's reintroduce φ. If the initial point of placement of an inserter is where the belt starts, then φ = 0. However if the inserter is further up the belt then it's as if the items it places had a head start compared to the inserters that are closer to the beginning. Hence it has a negative Phase delay φ = -x/v, where x - distance from the beginning of the belt, v - the speed of the belt(1.875 tiles/second). For an inserter that's 19 tiles up the belt at 10 seconds in, s∈(~27.6,25.4,23.1,22...) tiles from the beginning of the belt.

So if we take it all together we can express how far up the belt each item an inserter places is t0 seconds since the inserter started working:
f(t0) = 1.875[t0-n*1.2s * P/100%] -x/v, where n = the number of the item, n∈(1,2,3,4,5...). If there's several inserters placing the same kind of items we can just sum this: f(t0) = Σ(1.875[t0-n*1.2s*P'/100%] -x'/v), where x' ∈ (x1,x2,x3,x4...),P'∈(P1,P2,P3,P4...). If there's another set of inserters placing another kind of item on the same lane then they can be tracked separately using the same function or combined using Fourier Transform. If the inserters are taking items from a smelter that produces items with a period T then f(t0) = Σ(1.875[t0-n*max(T,1.2s)*P'/100%] -x'/v) This is still somewhat simplified as it does not take into account belt backing up or inserters trying to place an item where there already is one, but I'd be here all day if I were to write and prove a complete function.

So as you can see it can be expressed as a function and it does not require that the belt is compressed or that power is 100% or that there's only 1 type of item on a lane or that the smelter is producing items fast enough and I promise you every other precondition you mentioned also does not apply because the system is 100% understood and deterministic

- Circuits are literalry just functions
- If there's less than 100% power everything slows down proportionally
- If the liquid flows at a rate lower than the required rate X then the production slows down proportionally

Combinators slows down, when they have not enough power. The result is not longer deterministic (in most cases). And the same is it with inserters and fluids, they don’t slow down proportionally and you forget the time it takes for an item to run through the process.

Well, granted circuits sometimes change the way they work at low power. Still, they always work in a deterministic manner, there's no randomness involved here. As far as I'm aware inserters work at a rate proportional to their power level cuz every rotation consumes a fixed ammount of power. I don't know that much about liquids but I'd assume they also work in a deterministic fashion

[ssilk]

Hm. Interesting. And yes, everything in Factorio is deterministic. I meant it differently. It’s the base of the game, that it is deterministic. But deterministic means not, that it can be reduced to a simple formula.

Just two examples, why that can’t be expressed as mathematical expression in some special cases.

1. What happens, when an inserter cannot place an item on a belt? It waits a bit, until the belt is free again. I think that’s what you mean with shifting
This shifts the item on the belt so, that it comes out a bit later.

Now what if this shift happens not every time? I mean: there is another assembler before that and he doesn’t get regularly items, because he is the last on the supply-belt.

When you look now on the output belt you see eventually a pattern of items. So it is still mathematical expressable.

But now we add production modules. This randomizes the pattern so strong, that the mathematical formula gets so complicated, that it is easier to run the simulation instead.

2. I turn power on and off, just depending on, if the inserter for the one boiler gets a piece coal. And it gets a piece of coal only, if the smelters don’t need coal.

I know, that is a stupid setup, but what I want to show is, that you now have an extremely fractal design, and even if you are able to express that in a formula a very simple change can change the whole calculations.


That are the reasons why I say it works only, if the belts are fully compressed: the output pattern cannot change anymore. And if there is full power.
And if

[blazespinnaker]

+1

Factorrismo like addition would allow for much larger scale factories in vanilla. By allowing a player to identify inputs and outputs, a factory could be promotable to abstractable after it had proved itself over some period of time, perhaps time proportional to the complexity of the factory itself.

Perhaps too much volatility in production curves would result in a factory unable to be promoted to abstractable. This would create a very unique challenge in abstracted factory design.

The potential problem at that point would be that if these factories became too easy to put down, you could quickly launch a rocket within very little time.

I think it'd be very reasonable that abstraction shouldn't have any impact on resource cost or time / complexity to place. It would be no different than placing a blueprint.

In fact, you could even punish the player by not allowing a previously abstracted factory to be used even a little bit until it was fully placed. This is a huge penalty and in this way, abstracted factories would actually be negative in time to launch a rocket.

In addition, you could require that the player first terraform any land under the factory such that it has a proper factory floor. Also, walls, to provide a barrier against biters. Other things, such as roboport coverage with mininum # of robots could be required as well.

Any biter getting through a wall would just halt the factory. It would be tempting to take factories out of simulation mode for the player and updating them, but the impact on UPS could be frightening in some cases. Better would be to just let the biters run rampant through a bunch of unpowered entities.

The end benefit would be truly massive. They would allow for much greater scale factories via infinitely more efficient UPS usage, and maybe provide a minor added benefit for newer players in terms of managing complexity.

Allowing trains in factories would be cool Something may be required here to do faster large scale terraforming tho.

Of course, what would we do with all that scale? I think adding a bunch of very expensive infinite research would be a terrific addition to go along with this feature. See this thread here for more on that - viewtopic.php?f=5&t=93333&p=528820&hili ... ch#p528820

edit to add: I spent some time on this and I was able to enhance the factorissmo2 mod to do much of the above. https://mods.factorio.com/mod/Factoriss ... c7db4f5997

It's actually a pretty cool mod. Can't wait to use it in more megabase mass scale.

[ssilk]

What I find currently much more appealing than this, is the idea, that you create many of equal “elements” and you calculate only one.
Because of the deterministic nature of Factorio all others must behave the same, if they have the same input.

See this thread: viewtopic.php?f=6&t=91587 Nuclear Reactors BluePrint Cloning

[blazespinnaker]

What I find currently much more appealing than this, is the idea, that you create many of equal “elements” and you calculate only one.
Because of the deterministic nature of Factorio all others must behave the same, if they have the same input.

See this thread: viewtopic.php?f=6&t=91587 Nuclear Reactors BluePrint Cloning

Yes, this is what I've done, basically.

I'm not entirely clear on what you think the difference is, but perhaps it is this improvement -

I do leave the initial factory running (there is no API that can put entities to sleep that I've seen) and continually let it update a dictionary of its inputs/outputs. In this way the accuracy is more long term and continual.

I can't think of a way to do all of this deterministically, if that is what you were referring to. Hopefully there is a way someone can propose, but I'm not particularly optimistic. Or very concerned, really. When you get to megabase scale these sort of marginal issues are very tiny relative to everything else.

UPS becomes the overwhelming roadblock as you can't expand without torturing your factory designs, and then only in a limited way. I wouldn't use this mod obviously if UPS wasn't an issue, there would be little point. And I agree, deterministic production is a neat aspect of factorio, but afaik impossible to achieve at the sort of scale I envision without making some very creative assumptions.

In terms of deterministically working for multiplayer, I imagine that's fairly easy to do. Factorio is not that rigorously deterministic as there are non deterministic inputs, such as player actions.

But thank you for the nuclear power link. For some reason power didn't occur to me, but now that you shared it, I am so absolutely going to do that and should be fairly straightforward. I detest solar panels and accumulators

I'm probably going to make a new mod rather than using forked version of factorissimo. It's fairly limited in factory size and I need something much much larger. It really also has much different goals other than UPS optimization. Finally, I think I will not use pipes, belts, or chests but rather just cargo trains and fluid wagons.

The only downside is I don't really have the art to back this all up with.

[ptx0]

I do leave the initial factory running (there is no API that can put entities to sleep that I've seen) and continually let it update a dictionary of its inputs/outputs. In this way the accuracy is more long term and continual.

well if the assets exist on any surface, and even if you find entities on the factory and put them to sleep when the user leaves the building, inactive entities still contribute to more than 4 update loops in the game engine. The "entity" update loop goes to sleep, but the "ElectricNetwork" and other usages all still must update.

basically, you've done nothing to fix UPS other than build an expensive, Lua-scripted, infinity chest.

[blazespinnaker]

I do leave the initial factory running (there is no API that can put entities to sleep that I've seen) and continually let it update a dictionary of its inputs/outputs. In this way the accuracy is more long term and continual.

well if the assets exist on any surface, and even if you find entities on the factory and put them to sleep when the user leaves the building, inactive entities still contribute to more than 4 update loops in the game engine. The "entity" update loop goes to sleep, but the "ElectricNetwork" and other usages all still must update.

basically, you've done nothing to fix UPS other than build an expensive, Lua-scripted, infinity chest.

ahh, no, that's not true at all.

However, I'm going to resist getting into a discussion with you as it has been a net negative to these forums in a couple of places now, one of which has resulted in mod intervention. Enjoy your week, sir.

[ptx0]

I do leave the initial factory running (there is no API that can put entities to sleep that I've seen) and continually let it update a dictionary of its inputs/outputs. In this way the accuracy is more long term and continual.

well if the assets exist on any surface, and even if you find entities on the factory and put them to sleep when the user leaves the building, inactive entities still contribute to more than 4 update loops in the game engine. The "entity" update loop goes to sleep, but the "ElectricNetwork" and other usages all still must update.

basically, you've done nothing to fix UPS other than build an expensive, Lua-scripted, infinity chest.

ahh, no, that's not true at all.

i'm sorry, which part was untrue?

[blazespinnaker]

But I think it's a bad idea attempting to do it as a mod for Factorio. You'd be attempting to circumvent the game simulation in a game that's built around everything (that is built by player) being simulated and accounted for a all times. Making a new game in UE4/Unity seems easier than trying to bend the game this way through its modding API.

Unfortunately, it's either this or trash the megabase and start anew, unless you're aware of a third choice?

I prefer this option as it frees up UPS for more mining operations and much more vast and complex train network, while still allowing for work on optimal factory designs (which aren't redundantly being processed)

The idea on linking factories together for input/output is an interesting one for deterministic simulation, but it seems a bit too unrealistic and awkward (I imagine factories would constantly be falling out of sync), even though it would be more accurate..

[Eketek]

I have an idea for a mechanism which could feasibly be used to make parts of the factory virtualizeable. Also seems like it would fit into the game reasonably well and has additional uses in general gameplay.

Concept: Synchronized Factory Blocks

A synchronized factory block would be an arbitrary collection of entities which are all either concurrently active or concurrently suspended.

While suspended, every belt/transport line would stop, assembler progress would cease, inserter arms would lock in position, and [depending on game/mod settings] fluids and heat would stop flowing. And, when resumed, everything in the block would re-activate and proceed exactly as if it had continued operating.

Player would designate a factory block and assign the conditions which would activate or de-activate the block. Presumably, these would be manual toggle, a circuit signal, mod behaviour, and [most importantly] the combined state of the block's inputs and outputs.

Auto-toggling a factory block based on the combined state of the block's inputs and outputs (all inputs packed and all outputs open) would enforce the conditions needed for deeper optimizations (and thus would make such optimizations a lot more accessible in casual-ish gameplay).

[adam_bise]

Suppose I build up my smelting arrays to the point where UPS is affected, but then fully compress their inputs to achieve abstraction, raising the UPS back up. Then I do the same for wires, circuits, and so on.. At some point I overlook a shortage in my power generation, and at once nothing is abstracted. I might not be able to recover this situation due to extremely low UPS. Kind of like painting yourself into a corner there.

This problem would not happen if belts required power and their speed was linear to power input I think.

[blazespinnaker]

What I find currently much more appealing than this, is the idea, that you create many of equal “elements” and you calculate only one.
Because of the deterministic nature of Factorio all others must behave the same, if they have the same input.

See this thread: viewtopic.php?f=6&t=91587 Nuclear Reactors BluePrint Cloning

This is as good a thread as any for this topic.

. Adding a new child would mean then just to pause all child’s, until the new child is in synchronized state.

The only way that statement makes sense is if you don't believe a child factory needs to follow the complete production path of the parent. In that case, it is pretty much the same as simply estimating the input/output, but you have placed some concrete, fixed rules around it. I fail to see much benefit over estimation, except maybe it can work better in multiplayer. It certainly wouldn't be a more accurate simulation and some of the constraints around it seem rather awkward for the player to manage for no particular advantage.

More accurate would be what I originally thought you were suggesting, which was synchronizing a child so that it must follow the entire production path (inputs and outputs) of the parent exactly.

A simplified way to think about a production path is this:
Starting from zero state
- at time 1 (after a wipe), you begin with Input at time 1, this creates intermediates plus any outputs at time 1.
- at time 2, you use the intermediates from time 1 + Input at time 2, to create intermediates, plus any outputs at time 2
- at time 3, you use the intermediates from time 2 + Input at time 3, to create intermediates, plus any outputs at time 3
- at time N, you use the intermediates from time N-1 + input at time N, to create intermediates, plus any outputs at time N

We are given the inputs and outputs via tracking the parent factory so do not need to calculate or know the intermediates for a virtual child factory. If a virtual child factory had timely inputs matching the above inputs at times 1 through N (ie, synchronized), then it would be reasonable and accurate to assume that the outputs would be the same as the ones above - outputs 1 through N.

However, total times 1 through N will be arbitrarily, infinitely long unless you periodically reset it to zero state. All of the inputs and outputs at time t will be dependent on the child factory first following the entire series of inputs and outputs from time 1 to t-1

In this case, there is no such practical thing as "waiting until the new child is in synchronized state" if the series length can be arbitrarily long. It could be hours, maybe, perhaps dozens of hours that you would be waiting, depending on how long things had been working.

[ssilk]

I don’t understand what you mean, but I mean you think too complicated.

What I meant with “waiting until synchronized“ is that robots actively bring a sub-factory into synchronized state by adding/removing items (to assemblies, on belts, in inserters etc.), changing modules, “magically” changing production-cycle etc. Then, when everything is exactly as in the other child’s, the production can continue. Yet I haven’t thought deeper than that, because it is clear that this needs some additions to Factorio’s state-engine.

And I don’t want to continue discussion in this thread, it’s already messy. Please wait, for coming thread about this subject (I think today, but can’t promise).

[blazespinnaker]

Sure, but I suggest before you post you think through the implications a bit more as I think you're missing some rather critical subtleties. Determinism is a nice benefit, but making awkward leaps and compromises just to get there doesn't add that much unless you end up tracking the base game closely.

That said, I enjoyed the base idea quite a bit and found it very creative, intriguing and inspiring.

Too bad there doesn't seem to be much interest beyond a very niche crowd

[BicycleEater]

I do love the idea, but a mod could never do it that well.
I don't know, but I'm pretty sure the UPS hit from doing the lua scripting required would out do any benefit. When it comes to performance, an interpreted scripting language replacing hard-coded c++ is not a good move.
The problem with determinism is more than just idealogical, it is fundamental to the multi-player aspects of the game: The game has to be able to ensure that its state is the same in every client, as otherwise they will diverge. If this isn't guaranteed, then the clients and server have to communicate every tiny aspect of the operation of the game.
I do not think that at this stage of development, the developers would add something like this - in my opinion Factorio is mostly focussed on the early to mid game, where most players spend their time, and not the super-late game, where UPS becomes a considerable concern, meaning that these changes would be difficult, bug-inducing, hard to use, and rarely used.
If I were to guess, the best thing might be in train to train areas of the factory, where the inputs and outputs are most distinct, and isolated from the rest of the world, and the change would include a full simulation of the inside, but not in separate entities - the biggest hit to UPS will not be positions on belts, but entity-to-entity interactions, at least from a basic guess.
This would be very hard to make mod compatible - it would have to know the recipes and how they interact.
The hardest thing to simulate would be the odd conditions which limit throughput - e.g. an inserter waiting while the input is full.

Other problems also arise: if the game moves back to reality whenever something breaks the abstraction, if you ran out of power, the game might crash, as it tries to replace everything with the entity version.

[blazespinnaker]

Yeah, scripting can be a hit, but if you use chest like objects rather than belts, the hit isn't bad from what I've seen so far. It'd be nice if we were given some UPS efficient transfer apis tho.

For MP, there are global variables according to the tutorial and that these are synced. There is in fact very little data that needs to be synced up here, so if that works, determinism shouldn't be an issue w/ regards to MP.

Train to train is exactly how i'd like to do it, if we're thinking the same thing. Train -> Simulated Factory -> Train

I haven't had much time so I'm just using the chest connections right now in f2 - https://github.com/MagmaMcFry/Factoriss ... st.lua#L94 but will switch to cargo/fluid wagons when I do get some time.

I dunno if mod compatibility is an issue, tbh. The mod is agnostic when it comes to what it's dealing with in terms of input and output, but it's an interesting point.

Simulated impaired input is the challenge for sure. Not sure that's doable, tbh.