Optimization idea: abstraction

[Zanthra]

So to be clear, your abstraction would not 100% match the factorio simulation since walking away and returning after some time, objects on belts would not be in the same location as if you stayed in visual range of them the whole time?

[Nemo4809]

So to be clear, your abstraction would not 100% match the factorio simulation since walking away and returning after some time, objects on belts would not be in the same location as if you stayed in visual range of them the whole time?

IMHO a 100% match isn’t necessary. A less efficient throughput replacement is acceptable.

This way players can’t use it to cheat. When they choose to use the abstraction, they are choosing to sacrifice throughput for UPS efficiency.

[Zanthra]

I didn't say it was necessary, I just want to be clear on what we are talking about here. Sacrificing accuracy for speed is an interesting trade-off, but a completely different discussion than the challenges of how to make an accurate abstraction without actually running the whole simulation.

[StephenLynx]

So to be clear, your abstraction would not 100% match the factorio simulation since walking away and returning after some time, objects on belts would not be in the same location as if you stayed in visual range of them the whole time?

If you are taking a more loose approach, yes. Otherwise you could only simulate when all the inputs are 100% saturated and everything is already as buffered as possible. That way you always know where things would be when you stop simulating. But I think anything that wouldn't make a difference if all the inputs are fully saturated for a while after the simulation is stopped is acceptable.

[Deadlock989]

Goalposts successfully moved. Suddenly we're talking about factories getting a bit more shit if they move off screen. Brilliant.

So we've now successfully transitioned from "You can magically do everything, literally everything, much cheaper at 100% verisimilitude for absolutely zero extra processing cost and exactly the same end results" to "You can maybe do some very specific things cheaper sometimes but only if they involve fully compressed belts in absolutely ideal situations, and only if you literally avert your eyes from it, and only if you accept a massive loss of accuracy".

I mean, Wube could actually turn it into a selling point. "Factorio - the Night At The Museum Edition. Build your factory in Schrödingeresque hellscape where everything behaves differently when no-one is looking. Featuring deep strategy: choose exactly which belt to stand over because you want it to work better."

[Zanthra]

So to be clear, your abstraction would not 100% match the factorio simulation since walking away and returning after some time, objects on belts would not be in the same location as if you stayed in visual range of them the whole time?

If you are taking a more loose approach, yes. Otherwise you could only simulate when all the inputs are 100% saturated and everything is already as buffered as possible. That way you always know where things would be when you stop simulating. But I think anything that wouldn't make a difference if all the inputs are fully saturated for a while after the simulation is stopped is acceptable.

Even with compressed belts it is challenging to know what state the system will be in some number of ticks in the future. This is why in relation to the red science assembler setup: When and how is the exact position and number of iron gears sitting on the intervening belt calculated? For a perfect abstraction, one that can substitute for the actual simulation with no noticeable effects, it has to be 100% identical for any tick in the future you want to deabstract it at.

[boskid]

I mean, Wube could actually turn it into a selling point. "Factorio - the Night At The Museum Edition. Build your factory in Schrödingeresque hellscape where everything behaves differently when no-one is looking. Featuring deep strategy: choose exactly which belt to stand over because you want it to work better."

and "due to latency (in MP), game state does not yet know you are watching here, but since you are watching, we will produce some broken graphics of assemblers maybe working but not really, so please choose if you want latency hiding or nice graphics in all cases"

[eradicator]

a perfect abstraction, one that can substitute for the actual simulation with no noticeable effects, it has to be 100% identical for any tick in the future you want to deabstract it at.

Oh yea. You just need to know the exact starting state and then you can tell the abstraction to calculate the exact state it will have 1 tick later! It wouldn't even be slower than what we have now!

___
On a much much smaller scale, theorecically. If an assembler had an energy buffer of n>1 ticks. Would there be any relevant benefit in incrementing the crafting process only every n ticks instead of every tick? Sort of a "working sleep mode".

[Klonan]

On a much much smaller scale, theorecically. If an assembler had an energy buffer of n>1 ticks. Would there be any relevant benefit in incrementing the crafting process only every n ticks instead of every tick? Sort of a "working sleep mode".

Sure, it would work, but the assembling machine update is pretty much just 'increment the recipe progress', so it already doesn't take very much time to update at all.

The things that are worth optimizing are the complex things that interact with other entities,
Inserters, Units, Pipes, etc.

And those can't be optimized by a simple `update every n ticks` system

[Oktokolo]

Not even beacons are easy to abstract, it seems...

[eradicator]

On a much much smaller scale, theorecically. If an assembler had an energy buffer of n>1 ticks. Would there be any relevant benefit in incrementing the crafting process only every n ticks instead of every tick? Sort of a "working sleep mode".

Sure, it would work, but the assembling machine update is pretty much just 'increment the recipe progress', so it already doesn't take very much time to update at all.

Yea. I was just pondering the "Schrödingeresque" idea by @deadlock. Wondering if it's possible to completely remove the recipie incrementation when nobody is looking. Which depending on the recipe duration and scale of the factory migth have been interesting (beacon-less assemblers wouldn't be more expensive than speed-beaconed ones anymore?), but i guess when inserters start adding/removing stuff to the inventory the assembler has to wake up anyway?

The things that are worth optimizing are the complex things that interact with other entities,
Inserters, Units, Pipes, etc.

And those can't be optimized by a simple `update every n ticks` system

Yes. Perfectly clear. I specifically asked for the assembler because it's a perfectly closed system during crafting. Other such systems might be inserters while they're "in swing" (but that would be visible). Assembler and mining drills seem to be the only machines where you wouldn't see the resulting "jumpiness" of the internal progress without opening the gui.

Anyway thanks for confirming that the benefit is too small ;).

[Sopel]

Factorio doesn't update assembler progress every tick. Just when product is made or something else happens. At least that what one of the devs told me on reddit when I brought up the topic.

As for the "abstraction".
To even make it feasible I think it would require the user to specify an area and the graphics to be changed to not reflect the internal state, like a warehouse or something.
The big problem though is how to produce stuff. For a simple case it could calculate max rate and output items based on proportion of input items. Though in general it's problematic for many reasons:
- You CANNOT simulate the build to see whether it will work forever and never stop. If belts are turing complete (i think there was a post on reddit where someone made logic gates with belts and splitters) you cannot do it in general in any way.
- The build may have different parts that produce different items based on how many items you supply. For example having an overflow when there's >80% of some fluid going into something else.
- The general performance depends on small stuff, like for example on which side of the belt items are being input.
- Circuit networks in general are impossible to abstract because circuits are turing complete.

The only solution I see is having predefined entities that skip steps, but it sounds like a pain to balance and may impact immersion. Mods are for that.
One could also make something more dynamic based on heuristics like from Max Rate Calculator but it would be broken by design.

edit.
Having recipes which take more and produce more in one iteration would already help with UPS a bit. See https://mods.factorio.com/mod/SchallRecipeScaling But it has to be opt-in as it increases latency

[Rseding91]

Factorio doesn't update assembler progress every tick. Just when product is made or something else happens. At least that what one of the devs told me on reddit when I brought up the topic.

Yes it does and always has.

[Eketek]

Seems to me that the there are a few potential optimizations closely related to the topic which might be a bit more realistic for common cases:

There may be performance to gain simply by identifying factory modules which are isolated from the rest of the factory and processing them separately from the chunks they occupy.

If a factory module is exhibiting highly periodic behavior (every entity in the module performs the exact same sequence of actions on a recurring basis), then updates can be performed by capturing a complete cycle (or computing it on another thread) and re-playing that until something interferes.

[Frosti85]

I had the same idea, but then I realized how pointless this would be.

Imagine now you have this abstracted green chip factory, which makes like 1k green chips per minute, whatever.

Now you can place many ob these blackbox factories.

But in the end, you will be playing the same game, just that you have multiplied all the numbers by a factor of 1000. It would be totally pointless.

You could just change all the items, so that 1 green chip would become 1000 green chips.

That would be the end result of this.

You already have the blackbox green chip factory. It's called the assembler. It makes 1 green chip out of some ingredients, in a black box way.

The end result of the abstraction you suggested, would be to just increase the numbers.

This would only work for very basic factories without any circuit logic or more complex stuff. And these basic factories are basically assemblers in the end.

It would just increase the numbers.

[Hannu]

I do not agree. What is fine thing in Factiorio is that there are belts transport and inserters move every item exactly and factory seem to be working real thing. Practical limits of Factorio are huge. Far more then can be shown of seen. I do not see any reason to go for larger number of entities. And I am especially against replacing simulated function with boring black boxes just to be able to show larger numbers on production stats.

Maybe Factorissimo like mod, which handle factories as simple black boxes, would be good if there is much demand for this kind of thing.

[ssilk]

For me this is not really needed, because - I already talked about it - there are some mods, that will make the need for speed of this suggestion more ore less obsolete.

That are these points:
- boxing mod (deadlocks stacking inserters and extensions are able to assemble stacked items with stacked items. That means to produce 200 (a stack) of green circuits you need only one update of the assemblies, instead of 200 and you need to belt transport 1 item instead of many.

- Klonans Mining drones. That mod reduces also the needed CPU power.

- transport drones (also Klonan). This mod in conjunction with the boxing is sooo cool. No big bus. Instead streets with a lot of traffic.

All in all this brings a lot of CPU power back. I’m yet not sure how much. But it seems to be 3 times faster with the same amount of produced items.

What I mean: this suggestion/solution is in my eyes too obvious. Just make everything faster, bigger, more. And if it the simulation is not really exact anymore: who cares? (I know what I said some month ago, but I changed my opinion).

Instead of that in my eyes it’s about making things more clever, neater, smaller. Which the above mentioned combination of mods does.

[Qon]

I've been thinking of making a mod for this (or something similar at least, to make it possible). I'm almost far enough with my specification to start coding it. But I'm too busy at the moment to do it. But it might be my next big mod project.

This post is going to be a bit of a raw brain dump that might not make sense if you haven't read the topic, or maybe even then.

Goals

My goal is to maintain 60 UPS pretty much regardless of factory size and having production/minute be as close to vanilla as possible but not really paying attention to /tick production stats. It wouldn't be exactly as the devs intended it with each item actually being accounted for precisely. And I do love that about Factorio. But the game is great without it also and it doesn't hurt the game if it's implemented as a mod. If players can continue to play and expand their factories instead of abandoning them because they start dropping UPS then at that point the tradeoff seems to be pretty ok.

Features and requirements spec:

I absolutely need active = false to mean that the entities don't use CPU cycles. This makes it "simple" and fast to enable/disable abstraction quickly. And belts and bots etc will stay in position carrying material as if it was always running. With the assumption that the running factory looks "the same" all the time it should be OK to just pause the belts and bots and then unpause when a player comes to inspect. If materials have dried up at inputs inputs then that's just some buffered items that can be accounted for and won't really affect actual production but just when those things get used.

• It should just be plug-and-play, install the mod and after a while (if it's added to an already running big base) UPS should increase so that you can play without slowdowns any more.
• No input output marking
• to greatests extent should work with any building style
• no new entities
• This of course excludes Factorissimo style buildings and building with blurry glass roofs etc. The actual entities represent themselves, just active=false'd.
• This also means the player have to manage the factory just the same as if there really was proper simulation of all the items. No boxing of items that reduce belts, all the same space requirements, no difference to blueprints or calculations need to be made etc. Should "feel" like the real thing, except better UPS.

techical features:

• should be compatible with most mods
• should be able to handle belts, pipes, bots and trains in local networks. Maybe even new logistics methods like Klonans transport drones could be handled, maybe even without specific code to handle those if they are used within local production cells, but that might be strecting it.
• shouldn't be need recomputation for non-affecting edits
• should enable and disable parts as seamlessly as possible, instantly. After initial measurements the goal should be that only what you are currently working on and looking at is working. As soon as you look away it becomes abstracted and the new thing you look at is running instead.
• Less power means slower production, abstraction just adjusts proportionally (depends on power abstraction precision).

Limitations of requirements:

For this to be achievable I'm fine with minor losses of accuracy. A 20.00k SPM factory should still produce 20.00k SPM, losing or gaining less than 1% is not of great concern if I can get everything else good enough. But it should be as close as reasonably possible for number of items consumed and produced. The speed of production depends on power (might be less precise) and isn't as critical to get close to perfect imo, as long as it is within a few %. If things run 3% faster or slower, who cares if it's almost count perfect? The inputs and outputs might be constrained by some other non-abstracted factory anyways so in that case the abstraction could run it 100 times faster and you would just get "spikier" production but not actually any faster production. Sources of inaccuracy would be productivity modules. 2 things in might lead to 2 things out, but 3 things in could give 4 things out. The productivity bars could be read or the output calculated, but then you are talking about possible doing abstracted output higher than what you have measured so it just seems like a lot of work that generates more work for minor accuracy increase but at the risk of accuracy decrease if done with wrong assumptions and more CPU time by the mod. So not worth it I think, just measure for a bit longer for more accurate results.

The mod API doesn't give view position, but character position and player selected gives enough that it is almost achievable to know where the players are looking. So for that reason I'm fine with players seeing frozen machines if they use map view and accidentally/deliberately don't mouse over any entities.

Power draw might be tricky to get right, both to estimate, it's probably ok to approximate that with slightly less precision (so within a few %) instead of aiming to get it almost perfect like production stats. People overbuild on solar and nuclear on scales like 25% power excess or sometimes much more.

I think it's ok to only start measuring (abstraction preparation) when input buffers are filled and power is always at 100%. Should be reasonable to maintain for a few minutes.

Measurement really only needs to check that power is consistent every frame. Inputs and outputs are buffered and only need to be checked sometimes. So measurement is almost free, the heavier calculations only happen once or twice for the system at start and end of measurement. Finding and maintaining areas to abstract is another part that could require a fair ammount of work. But this isn't time critical and can be spread out over many tick, over minutes or hours. Doing it spread out over more time just delays the time it takes for a completed factory unit to be abstracted for the first time after being completed/modified and then up and running. For big factories the cell units that have not been modified by the player in the last 10 minutes is probably 90% or more, typically.

I'm not planning on making it completly impossible to "abuse" into behaving differently when abstracted from what it otherwise would do. It aims to do "the same thing as previous minute, repeated over the next hundred minutes". If a factory cell produces one thing at odd hours and other things at even hours then it should be accurate in it's production count, it just wouldn't switch to the other behaviour if there's circuits that route things differently certain times, or factory cells with circuits wouldn't be abstracted maybe.

Buffered items should be counted or assembler types should have crafts counted. Or buffered items should be detected and not be part of the measured and cached output for repetition. So you don't get free outputs because buffers were emptying while measuring. But otherwise it doesn't need to be impossible to abuse with all mod configurations etc. It just shouldn't accidentally give the wrong output when abstracted. If people want to abuse or trick the abstraction then they might as well just install Creative Mod or use infinity chests so it's not like real cheats aren't available if my mod isn't perfect...

The parts that will have items drawn from and outputted to will have to be chests. There items can be created or destroyed faster by Lua than Factorio by itself since items are not inserted individually but filled/drained completely instead with an API call, and big chains of interactions of entities can be abstracted away.

Ores on ground and labs/research also need to be handled. Especially ore deposits since most of material flow happens with mining ores and then smelting so it would be a great boost if even just that was sped up.

Difficulties

Most of the hard things to do will be to maintain data structures over a dynamic factory, find input outputs, not spend too much time on checking if things interfere with earlier abstractions, finding connections, maintaining hierachies and overlapping abstractions, simulating power draw etc, splitting work over several ticks efficiently in Lua...

But a more limited form that skips cells with logistics chests in it and doesn't pay as much attention to accuracy of the speed, power draw and splits the world into simpler disjointed cells etc might be possible to prototype up a bit quicker and then I'll see if the project is worth improving up to full spec.

[eradicator]

Features and requirements spec:

I absolutely need active = false to mean that the entities don't use CPU cycles. This makes it "simple" and fast to enable/disable abstraction quickly. And belts and bots etc will stay in position carrying material as if it was always running.

Belts don't care about "active".

[Qon]

That's not great. They can be paused with circuits but not by mods?
So do belts require CPU cycles if some items are on them but nothing is taken on or off them? (Inserters active false).

I'm don't care about circular belts, so the items will be stationary at the ends after a short while. It would look kind of bad. Maybe i need to add circuits to shut the belts off? If that makes them inactive.