Optimization idea: abstraction

Post your ideas and suggestions how to improve the game.
Post Reply
StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Optimization idea: abstraction

Post by StephenLynx »

From reddit:
https://www.reddit.com/r/factorio/comme ... ssibility/
Recently a different post about abstraction made me think. What if the game could take isolated groups of assemblers and internally, abstract it? Only calculate what's coming in the area and how much it should come out.

Take this red circuit production area of my factory as an example: https://i.imgur.com/PjIDmPs.jpg

Plastic, copper plates and green circuits come in, red circuits come out in a steady pace. As far as production goes, it isn't related to anything else in between the train stations, aside from power supply. So couldn't the individual assemblers, inserters and belts be abstracted and just calculated what's in, what's out and the leftovers for the next cycle?

Now, I know this present some issues. What about when the player comes to look at it and expects to see individual wires going around and other stuff? And I agree, that do pose a challenge. I'm not talking about making the game behave any different as far as the player knows and sees. What's I'm talking about here is not about a hard suggestion, just a venue of thought that might produce some fruits if the developers think about it.

But if this is possible, think about how much UPS it would save. How larger factories could be or how better lower end machines could run it. I dare to say it might be the biggest optimization for a modular factories this game have ever seen.

User avatar
5thHorseman
Smart Inserter
Smart Inserter
Posts: 1138
Joined: Fri Jun 10, 2016 11:21 pm
Contact:

Re: Optimization idea: abstraction

Post by 5thHorseman »

It would be crazy. Imagine an entire smelting array abstracted to just a few variables, like how many plates can be created per second (or tick or whatever), how long it takes a chunk of ore to turn into a plate, and how much power per plate is used. Then a train comes and drops off ore, which goes into the abstraction zone and disappears for x seconds, using y power, and a plate comes out of the zone on another belt.

I don't know how the game could properly do that on its own, but imagine something like Factorissimo where the player sets them up and then throws a black box over them intentionally.
Image Image

Koub
Global Moderator
Global Moderator
Posts: 6029
Joined: Fri May 30, 2014 8:54 am
Contact:

Re: Optimization idea: abstraction

Post by Koub »

The first thing that came to my mind reading OP's post is https://mods.factorio.com/mod/Factorissimo2 too :)
Koub - Please consider English is not my native language.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4360
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

StephenLynx wrote:
Sat Mar 07, 2020 1:08 am
What if the game could take isolated groups of assemblers
The impossibilty of this is the whole reason why factorio is in large parts not multi-threaded yet. As has been stated in various FFFs over the years. Everything interacts with everything. And imho one of the major selling points of Factorio is that it precisely simulates everything. If you put 20million ore in you get 20million plates out, and not "approximately 19.5million plates". So even if you could abstract it would be ugly.
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11421
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

TL;DR
Parts of a factory can be simulated, if the conditions that influence that part have very specific values. E.g. input belts needs to be fully compressed, and many more. In that case it is safe to switch to a simulation.

Long read:
Well, I think this is a way we should think about deeper.

What I’m thinking here is also Factorissimo, but instead of just one building it should have the “real size”.

So what I see here are some possible solutions, which would simulate a realistic production process for very large quantities but switch back to the realistic simulation for low quantities.

How could that be done?

First I think about the concept of factory-streets. Like a gigantic smelting array. This can be simulated very exactly. See for example the max rate calculator https://mods.factorio.com/mod/MaxRateCalculator

But it needs a lot of preconditions to make sure, the simulation is always exact:
- all input belts are always compressed
- enough free space on output belts to put items on
- inserter are fast enough to feed/empty the smelters
- there is no complex belt stuff, like when it is possible that two different items could be on one lane
- same is with mixed input lanes. As soon as another item on a lane is imputed it needs to switch back to simulation.
- there are no circuits involved
- there is always 100% power available (well I think this can be simulated up to some degree)
- pipes are never filled below some level X
- player is not changing anything inside that factory street are
- the theoretical input/output is equal to the current input/output
- and of course some more preconditions.

You see it’s difficult to achieve those conditions so I would like to introduce a “factory-street-area”. Normally this area is hidden with some kind of industrial roof (game doesn’t need to simulate the inside), but when the player enters, the roof becomes transparent. When you want to built inside such an area the factory street switches to the normal mode and everything behaves as it should. You can then built inside the factory street, but you cannot built some constructions. For example you accidentally side-insert a different item into a belt-lane. You cannot do circuits, or you forgot to power a device.

But when you go outside and the above conditions are meet, then it switches to simulation mode after a while. After a while means: the theoretical input/output needs to be more or less equal to the current measured input/output.

And as soon as one condition isn’t meet anymore the factory street switches back to normal mode. That is likely, if the output is blocked or not enough input comes in, or if power fails. But for those cases that input is missing or output is blocked it’s already so, that the inserters and entities go to sleep. So not too much CPU used. The factory street covers only the case, when it is running with full speed.

So what does this bring?

- useless if you only have one factory street, because if you want to change the layout everything is as before. You need to built many streets.
- useless if you are not capable to supply a FS with a constant stream of material. This is an interesting task.
- And this is also a moment, where a factory street could be an advantage for the player: lets say the FS works and it is able to switch to simulation mode. What if the player then is able to increase the speed or productivity etc with modules? The modules work only in the simulated mode. ... not thought to the end but that could be the moment when you really want to build this.
- also useless, if you don’t need so much output...
- ... which is a quite interesting logistic task to build that so, that the factory street is either working in simulated mode or is turned off, because that takes much control from outside. This makes it also much easier to switch off the power, if nothing is produced anymore.


But for the case that the preconditions are meet, and you have many factory streets working in simulation, this should be a real bringer! And it would enable much, much bigger factories than now.

If this is also combined with some boxing mod, like deadlocks https://mods.factorio.com/mod/deadlock- ... es-loaders which enables to produce with boxed items, then I think factor 10-100 bigger factories should be possible. :)

I think for many players that would be a really good argument. :D
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11421
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

And one word more to how this is looking. Above I suggested a factory street looks like a big industrial hall. As player you see just roofs.

I would turn that into: it is looking like a glasshouse. You see the devices working and so on, and as long as you are not touching anything, you can walk in and out.

For the simulation-mode I suggest this: it takes some seconds to measure, if the theoretical input/output meets current input/output. In that time the game creates an animation of (lets say) 15 seconds. Then it plays this animation instead. Because you see that through the glass of the glasshouse it is a bit blurred, so you might not recognize the repeats. It repeats the animation over and over, until switching back to normal mode.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

About requiring fully compressed belts, I don't know. Is that level of precision really required? Here's how I see it: 10 copper enter the abstracted zone. So we know there can't be more than 20 copper wires in the system if that's what we are producing. Now we switch back to non-abstracted mode and all we know is that were are 10 copper in the system. We turn those into copper wires, place then around. Even if these wires end up in places where they wouldn't be if nothing was abstracted, how relevant is this difference? We didn't rush anything because of how fast copper enters the area, get it? We didn't abstract anything that would make materials go faster through the system as a whole than they can actually go, because we didn't abstract the part where the inserters take copper from the train and place them on the belt. If you are in the ballpark of producing hundreds of SPM, possibly thousands, I don't see small imprecisions like this making a difference.

And about how hard it could be to figure what's actually isolated and isn't, I feel like there are some solid cases. Let's say the game thinks like this: this belt can only take things from this train station, and everything that comes in through here ends up on a different train station. There are no signals, combiners, logistic chests involved in any of this. So I'll start abstracting after all inserters took the items from the station until the final products start being put on another train station. Even if in the end everything can be connected to everything, you can still see parts that are linear and those are much easier to abstract. I understand that there might be situations where throughput is not the same, let's say you have too many assemblers on one side of the splitter and too few on the other. So the game could run an internal benchmark to see how fast the whole area actually is and use that as the result of the abstraction.

And about insufficient power, it could simply do what assemblers do. Slow down proportionally.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11421
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

StephenLynx wrote:
Sat Mar 07, 2020 2:09 pm
About requiring fully compressed belts, I don't know. Is that level of precision really required? Here's how I see it: 10 copper enter the abstracted zone. So we know there can't be more than 20 copper wires in the system if that's what we are producing. Now we switch back to non-abstracted mode and all we know is that were are 10 copper in the system. We turn those into copper wires, place then around. Even if these wires end up in places where they wouldn't be if nothing was abstracted, how relevant is this difference?
A good question!

You can see that from two positions: Relative and absolute. You see it realtive. 10 items more from 10000 is 1 ‰. Doesn't matter.

My opinion is, that you need to see it absolute. If there is a lack in production and the biters come and you need a weapon upgrade to fight them, then 10 items more ore less can be in the end the difference between win and loss. You can argument: How can you know it? It doesn't matter. My answer: I know that it might be a difference. I can see, that the items don't come out in the pattern as they should. The simulation is tricking me out and takes me away the need to optimize difficult building situations.
We didn't rush anything because of how fast copper enters the area, get it? We didn't abstract anything that would make materials go faster through the system as a whole than they can actually go, because we didn't abstract the part where the inserters take copper from the train and place them on the belt. If you are in the ballpark of producing hundreds of SPM, possibly thousands, I don't see small imprecisions like this making a difference.
I can construct you several situations, where this is heavily important. ;)

Lets take for example a mixed production of rails and poles. Both need iron sticks. So you put in iron, steel, copper and stone and out comes medium and big pole and rails. Obviously the iron sticks are made inside the "simulation".

Yes, this works perfect as long as you take only poles. But at some point you need rails AND poles. The orginal built has then the problem, that there are not enough sticks and it produces only rails, no poles anymore! Because the inserters for the rail assembly takes the sticks away before the poles -inserters can grab it.

But the simulation will produces poles and rails in equal amounts!

This is totally wrong. It kills the whole game by doing that.
And about how hard it could be to figure what's actually isolated and isn't, I feel like there are some solid cases. Let's say the game thinks like this: this belt can only take things from this train station, and everything that comes in through here ends up on a different train station. There are no signals, combiners, logistic chests involved in any of this.
Not that I insist on that, but from my above post you see, that this is out of scope for simulation, because there are too much side-effects, like there is suddenly a wagon, which can be more or less full.
So I'll start abstracting after all inserters took the items from the station until the final products start being put on another train station.
Ahhh, do you mean, that Factorio should find areas that can be simulated by itself? Interesting idea.
Even if in the end everything can be connected to everything, you can still see parts that are linear and those are much easier to abstract. I understand that there might be situations where throughput is not the same, let's say you have too many assemblers on one side of the splitter and too few on the other. So the game could run an internal benchmark to see how fast the whole area actually is and use that as the result of the abstraction.
How do you simulate, that an inserter might be too slow to grep items from belt? Big difference in output.
And about insufficient power, it could simply do what assemblers do. Slow down proportionally.
... which will fill suddenly belts, that are normally not filled, because not so much is produced, but the input keeps at the same level (for a while).

I can construct doozens of such situations and they will all the basics, the Factorio-essence, the many small details, that makes the fun.

As said: It is possible to simulate (very) big parts of Factorio, if the surrounding circumstances are perfect. I don't see that as a problem: Instead of maximum throughput you need to optimize for maximum stable conditions. Big difference in build style! New tasks, new fun.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

Nemo4809
Long Handed Inserter
Long Handed Inserter
Posts: 94
Joined: Thu Jan 16, 2020 10:49 am
Contact:

Re: Optimization idea: abstraction

Post by Nemo4809 »

What happens if you accidentally feed iron plates into your abstract wire factory?

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

ssilk wrote:
Mon Mar 09, 2020 10:59 pm
But the simulation will produces poles and rails in equal amounts!
That should absolutely not happen. I don't think a simulation that produces results wildly differently should happen. The engine could do one of 2 things in this kind of situation, IMO: 1: only abstract parts that produce 1 final product. 2: benchmark to see what should be the production ratio. Also, I don't think an area under attack should be simulated at all. Or simply put: the simulation is redone after anything that could affect production changes.
ssilk wrote:
Mon Mar 09, 2020 10:59 pm
Ahhh, do you mean, that Factorio should find areas that can be simulated by itself? Interesting idea.
Exactly. I never proposed a feature, something the player would actively do. Just shortcuts the game itself would take when possible.
ssilk wrote:
Mon Mar 09, 2020 10:59 pm
How do you simulate, that an inserter might be too slow to grep items from belt? Big difference in output.
Benchmarks. The game tests a possible simulation and see how efficient it actually can be.
ssilk wrote:
Mon Mar 09, 2020 10:59 pm
... which will fill suddenly belts, that are normally not filled, because not so much is produced, but the input keeps at the same level (for a while).
So just reduce the input proportionally and let the belts before the simulation fill. You are just filling belts sooner than later. Or I dunno, just stop simulating at all on low power scenarios. Factories run at full power 99% of the time anyway.
ssilk wrote:
Mon Mar 09, 2020 10:59 pm
As said: It is possible to simulate (very) big parts of Factorio, if the surrounding circumstances are perfect.
Exactly. When you have an optimization that would reduce processing power required to less than 1%, you don't have to apply it every where. Just do whenever possible and dedicated players will just start to design around these possibilities. Casual players wouldn't see a different, most of the time.

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

Nemo4809 wrote:
Tue Mar 10, 2020 12:04 am
What happens if you accidentally feed iron plates into your abstract wire factory?
Update the simulation, stop simulating, just don't let the iron enter the simulation and clog the belt. Any of these would work.

User avatar
Oktokolo
Filter Inserter
Filter Inserter
Posts: 874
Joined: Wed Jul 12, 2017 5:45 pm
Contact:

Re: Optimization idea: abstraction

Post by Oktokolo »

I would not expect this to be ever implemented.

But it can be approximated by modding in a huge assembler or furnace and corresponding recipes for batch-processing N times the ingredients for N times the result.
Combine that with loaders (vanilla or Miniloader) for input and output to get most of the blackbox optimization.

User avatar
5thHorseman
Smart Inserter
Smart Inserter
Posts: 1138
Joined: Fri Jun 10, 2016 11:21 pm
Contact:

Re: Optimization idea: abstraction

Post by 5thHorseman »

StephenLynx wrote:
Tue Mar 10, 2020 1:56 am
Also, I don't think an area under attack should be simulated at all.
So you're simulating your circuits and a biter comes in and starts attacking it. How do you un-simulate that block in order to know where every plate, wire, and circuit is at the moment the biter attacks? How about what percentage the assemblers are done with their products, and what each inserter is doing?

Considering the devs stated views on these sorts of things, assume they will not accept anything that isn't actually what the simulated box would be if it had never been simulated in the first place.
Image Image

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

5thHorseman wrote:
Tue Mar 10, 2020 2:58 am
So you're simulating your circuits and a biter comes in and starts attacking it. How do you un-simulate that block in order to know where every plate, wire, and circuit is at the moment the biter attacks? How about what percentage the assemblers are done with their products, and what each inserter is doing?
The same way it would be unsimulated if the player comes near it. Approximate things. See how many buffered items are, throw them around the best way possible.

And what exactly did they say? I don't read FFF religiously anymore. Not to mention that games are not exactly set in stone. As a software developer myself, I know that people sometimes make certain concessions given new information.

Nemo4809
Long Handed Inserter
Long Handed Inserter
Posts: 94
Joined: Thu Jan 16, 2020 10:49 am
Contact:

Re: Optimization idea: abstraction

Post by Nemo4809 »

StephenLynx wrote:
Tue Mar 10, 2020 1:58 am
Nemo4809 wrote:
Tue Mar 10, 2020 12:04 am
What happens if you accidentally feed iron plates into your abstract wire factory?
Update the simulation, stop simulating, just don't let the iron enter the simulation and clog the belt. Any of these would work.
What exactly do you mean by "update the simulation"? Update the equation? How much time is that going to take? Do you switch equation every time input conditions change?

Say you take the naive solution and just filter all the inputs. There are still going to be variation in the rate of input and that will affect congestion within the abstract factory. That's going to be hella hard to model and compress down into a single equation without the equation being as difficult to evaluate as just simulating it.

Heck, even something as simple as an inserter putting items on a belt is going to be problematic since there are all sorts of edge cases - e.g. inserter can only put items on the belt if there is free space, and to know how much free space is under the inserter hand you basically have to simulate the belt ...

That said, if we standardize the input - like a box of iron plates (1000 plates inside) - it might work. With all necessary inputs present, simulate the process from start to finish - with no other inputs until completion - to get its characteristics. From then on, the abstract factory will only process one set of input at a time, disallowing further input until the process "completes", and the output and remainder materials is ejected from the abstract factory.

User avatar
5thHorseman
Smart Inserter
Smart Inserter
Posts: 1138
Joined: Fri Jun 10, 2016 11:21 pm
Contact:

Re: Optimization idea: abstraction

Post by 5thHorseman »

StephenLynx wrote:
Tue Mar 10, 2020 3:30 am
And what exactly did they say? I don't read FFF religiously anymore. Not to mention that games are not exactly set in stone. As a software developer myself, I know that people sometimes make certain concessions given new information.
I can't point to a quote, but there is an underlying "this is EXACTLY predictable and can be recreated with no guesswork so long as you know the rules" to everything. This permeates through the game all the way down to the ground scatter.

One thing I recall - again about ground scatter that seems to be sticking in my head for some reason - was that in one FFF comments section someone asked why they bother saving all the ground scatter in the save file and why don't they just load it randomly. The answer was one word: "Determinism."

Imagine suggesting to that person just tossing plates on belts randomly.
Image Image

Nemo4809
Long Handed Inserter
Long Handed Inserter
Posts: 94
Joined: Thu Jan 16, 2020 10:49 am
Contact:

Re: Optimization idea: abstraction

Post by Nemo4809 »

OK, I believe this could totally work.

Let there be special input and output chests for creating abstract factories.

You place 2 input chests and filter them to have 100 iron plates and 150 copper plates respectively. You then place inserters to extract the plates, place belts and assemblers to make the green circuits along with belt and inserters to drop the the produced green circuits into the output chest.

You highlight the whole setup like you would a BP - you must include at least one input and output chest in the selection. The game will then run a simulation where 100 iron plates and 150 copper plates are pulled out by the inserters you placed and sent off to do whatever your setup does. The simulation runs until everything "goes to sleep" and there is no more activity. The game then checks to see what is produced in the output box and how long the simulation took - how long it would take to process all the iron and copper as if you just let the setup run.

Using the above information, it will create your abstract factory. When you funnel exactly 100 iron plates and 150 copper plates into the abstract factory, that's when it will start up, "process" the materials for the amount of time based on the simulation, and spit out the green circuits (again as determined by the simulation). The factory then "resets" and all remaining material left in the factory - except the material in the input/output chests - are destroyed. Then the next 100/150 plates is processed when it arrives.

Completely deterministic. Throughput might end up lower but it would be very cheap UPS-wise since the simulation need only be done once.

xfir01
Long Handed Inserter
Long Handed Inserter
Posts: 93
Joined: Fri Feb 10, 2017 5:08 pm
Contact:

Re: Optimization idea: abstraction

Post by xfir01 »

Sounds kind of like Whistle Stop Factories.
https://mods.factorio.com/mod/WhistleStopFactories

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

5thHorseman wrote:
Tue Mar 10, 2020 10:53 am
One thing I recall - again about ground scatter that seems to be sticking in my head for some reason - was that in one FFF comments section someone asked why they bother saving all the ground scatter in the save file and why don't they just load it randomly. The answer was one word: "Determinism."

Imagine suggesting to that person just tossing plates on belts randomly.
It can be deterministic. I was thinking about one specific way: you take the buffered input and place them the closest to the start of the simulation. If you produce subproducts, like wires for red circuits, you start also placing the wires after a certain point, since you placed plates in places that would be taken to produce wires.

StephenLynx
Inserter
Inserter
Posts: 31
Joined: Thu May 04, 2017 10:55 pm
Contact:

Re: Optimization idea: abstraction

Post by StephenLynx »

Nemo4809 wrote:
Tue Mar 10, 2020 11:10 am
OK, I believe this could totally work.

Let there be special input and output chests for creating abstract factories.
I don't think it would need special chests. If you know the chest is receiving items from a rail track or having it's contents placed into a rail track, you could use these chests as end points and simulate everything in between.

Post Reply

Return to “Ideas and Suggestions”

Who is online

Users browsing this forum: posila