Optimization idea: abstraction

[ssilk]

And one word more to how this is looking. Above I suggested a factory street looks like a big industrial hall. As player you see just roofs.

I would turn that into: it is looking like a glasshouse. You see the devices working and so on, and as long as you are not touching anything, you can walk in and out.

For the simulation-mode I suggest this: it takes some seconds to measure, if the theoretical input/output meets current input/output. In that time the game creates an animation of (lets say) 15 seconds. Then it plays this animation instead. Because you see that through the glass of the glasshouse it is a bit blurred, so you might not recognize the repeats. It repeats the animation over and over, until switching back to normal mode.

[StephenLynx]

About requiring fully compressed belts, I don't know. Is that level of precision really required? Here's how I see it: 10 copper enter the abstracted zone. So we know there can't be more than 20 copper wires in the system if that's what we are producing. Now we switch back to non-abstracted mode and all we know is that were are 10 copper in the system. We turn those into copper wires, place then around. Even if these wires end up in places where they wouldn't be if nothing was abstracted, how relevant is this difference? We didn't rush anything because of how fast copper enters the area, get it? We didn't abstract anything that would make materials go faster through the system as a whole than they can actually go, because we didn't abstract the part where the inserters take copper from the train and place them on the belt. If you are in the ballpark of producing hundreds of SPM, possibly thousands, I don't see small imprecisions like this making a difference.

And about how hard it could be to figure what's actually isolated and isn't, I feel like there are some solid cases. Let's say the game thinks like this: this belt can only take things from this train station, and everything that comes in through here ends up on a different train station. There are no signals, combiners, logistic chests involved in any of this. So I'll start abstracting after all inserters took the items from the station until the final products start being put on another train station. Even if in the end everything can be connected to everything, you can still see parts that are linear and those are much easier to abstract. I understand that there might be situations where throughput is not the same, let's say you have too many assemblers on one side of the splitter and too few on the other. So the game could run an internal benchmark to see how fast the whole area actually is and use that as the result of the abstraction.

And about insufficient power, it could simply do what assemblers do. Slow down proportionally.

[ssilk]

About requiring fully compressed belts, I don't know. Is that level of precision really required? Here's how I see it: 10 copper enter the abstracted zone. So we know there can't be more than 20 copper wires in the system if that's what we are producing. Now we switch back to non-abstracted mode and all we know is that were are 10 copper in the system. We turn those into copper wires, place then around. Even if these wires end up in places where they wouldn't be if nothing was abstracted, how relevant is this difference?

A good question!

You can see that from two positions: Relative and absolute. You see it realtive. 10 items more from 10000 is 1 ‰. Doesn't matter.

My opinion is, that you need to see it absolute. If there is a lack in production and the biters come and you need a weapon upgrade to fight them, then 10 items more ore less can be in the end the difference between win and loss. You can argument: How can you know it? It doesn't matter. My answer: I know that it might be a difference. I can see, that the items don't come out in the pattern as they should. The simulation is tricking me out and takes me away the need to optimize difficult building situations.

We didn't rush anything because of how fast copper enters the area, get it? We didn't abstract anything that would make materials go faster through the system as a whole than they can actually go, because we didn't abstract the part where the inserters take copper from the train and place them on the belt. If you are in the ballpark of producing hundreds of SPM, possibly thousands, I don't see small imprecisions like this making a difference.

I can construct you several situations, where this is heavily important.

Lets take for example a mixed production of rails and poles. Both need iron sticks. So you put in iron, steel, copper and stone and out comes medium and big pole and rails. Obviously the iron sticks are made inside the "simulation".

Yes, this works perfect as long as you take only poles. But at some point you need rails AND poles. The orginal built has then the problem, that there are not enough sticks and it produces only rails, no poles anymore! Because the inserters for the rail assembly takes the sticks away before the poles -inserters can grab it.

But the simulation will produces poles and rails in equal amounts!

This is totally wrong. It kills the whole game by doing that.

And about how hard it could be to figure what's actually isolated and isn't, I feel like there are some solid cases. Let's say the game thinks like this: this belt can only take things from this train station, and everything that comes in through here ends up on a different train station. There are no signals, combiners, logistic chests involved in any of this.

Not that I insist on that, but from my above post you see, that this is out of scope for simulation, because there are too much side-effects, like there is suddenly a wagon, which can be more or less full.

So I'll start abstracting after all inserters took the items from the station until the final products start being put on another train station.

Ahhh, do you mean, that Factorio should find areas that can be simulated by itself? Interesting idea.

Even if in the end everything can be connected to everything, you can still see parts that are linear and those are much easier to abstract. I understand that there might be situations where throughput is not the same, let's say you have too many assemblers on one side of the splitter and too few on the other. So the game could run an internal benchmark to see how fast the whole area actually is and use that as the result of the abstraction.

How do you simulate, that an inserter might be too slow to grep items from belt? Big difference in output.

And about insufficient power, it could simply do what assemblers do. Slow down proportionally.

... which will fill suddenly belts, that are normally not filled, because not so much is produced, but the input keeps at the same level (for a while).

I can construct doozens of such situations and they will all the basics, the Factorio-essence, the many small details, that makes the fun.

As said: It is possible to simulate (very) big parts of Factorio, if the surrounding circumstances are perfect. I don't see that as a problem: Instead of maximum throughput you need to optimize for maximum stable conditions. Big difference in build style! New tasks, new fun.

[Nemo4809]

What happens if you accidentally feed iron plates into your abstract wire factory?

[StephenLynx]

But the simulation will produces poles and rails in equal amounts!

That should absolutely not happen. I don't think a simulation that produces results wildly differently should happen. The engine could do one of 2 things in this kind of situation, IMO: 1: only abstract parts that produce 1 final product. 2: benchmark to see what should be the production ratio. Also, I don't think an area under attack should be simulated at all. Or simply put: the simulation is redone after anything that could affect production changes.

Ahhh, do you mean, that Factorio should find areas that can be simulated by itself? Interesting idea.

Exactly. I never proposed a feature, something the player would actively do. Just shortcuts the game itself would take when possible.

How do you simulate, that an inserter might be too slow to grep items from belt? Big difference in output.

Benchmarks. The game tests a possible simulation and see how efficient it actually can be.

... which will fill suddenly belts, that are normally not filled, because not so much is produced, but the input keeps at the same level (for a while).

So just reduce the input proportionally and let the belts before the simulation fill. You are just filling belts sooner than later. Or I dunno, just stop simulating at all on low power scenarios. Factories run at full power 99% of the time anyway.

As said: It is possible to simulate (very) big parts of Factorio, if the surrounding circumstances are perfect.

Exactly. When you have an optimization that would reduce processing power required to less than 1%, you don't have to apply it every where. Just do whenever possible and dedicated players will just start to design around these possibilities. Casual players wouldn't see a different, most of the time.

[StephenLynx]

What happens if you accidentally feed iron plates into your abstract wire factory?

Update the simulation, stop simulating, just don't let the iron enter the simulation and clog the belt. Any of these would work.

[Oktokolo]

I would not expect this to be ever implemented.

But it can be approximated by modding in a huge assembler or furnace and corresponding recipes for batch-processing N times the ingredients for N times the result.
Combine that with loaders (vanilla or Miniloader) for input and output to get most of the blackbox optimization.

[5thHorseman]

Also, I don't think an area under attack should be simulated at all.

So you're simulating your circuits and a biter comes in and starts attacking it. How do you un-simulate that block in order to know where every plate, wire, and circuit is at the moment the biter attacks? How about what percentage the assemblers are done with their products, and what each inserter is doing?

Considering the devs stated views on these sorts of things, assume they will not accept anything that isn't actually what the simulated box would be if it had never been simulated in the first place.

[StephenLynx]

So you're simulating your circuits and a biter comes in and starts attacking it. How do you un-simulate that block in order to know where every plate, wire, and circuit is at the moment the biter attacks? How about what percentage the assemblers are done with their products, and what each inserter is doing?

The same way it would be unsimulated if the player comes near it. Approximate things. See how many buffered items are, throw them around the best way possible.

And what exactly did they say? I don't read FFF religiously anymore. Not to mention that games are not exactly set in stone. As a software developer myself, I know that people sometimes make certain concessions given new information.

[Nemo4809]

What happens if you accidentally feed iron plates into your abstract wire factory?

Update the simulation, stop simulating, just don't let the iron enter the simulation and clog the belt. Any of these would work.

What exactly do you mean by "update the simulation"? Update the equation? How much time is that going to take? Do you switch equation every time input conditions change?

Say you take the naive solution and just filter all the inputs. There are still going to be variation in the rate of input and that will affect congestion within the abstract factory. That's going to be hella hard to model and compress down into a single equation without the equation being as difficult to evaluate as just simulating it.

Heck, even something as simple as an inserter putting items on a belt is going to be problematic since there are all sorts of edge cases - e.g. inserter can only put items on the belt if there is free space, and to know how much free space is under the inserter hand you basically have to simulate the belt ...

That said, if we standardize the input - like a box of iron plates (1000 plates inside) - it might work. With all necessary inputs present, simulate the process from start to finish - with no other inputs until completion - to get its characteristics. From then on, the abstract factory will only process one set of input at a time, disallowing further input until the process "completes", and the output and remainder materials is ejected from the abstract factory.

[5thHorseman]

And what exactly did they say? I don't read FFF religiously anymore. Not to mention that games are not exactly set in stone. As a software developer myself, I know that people sometimes make certain concessions given new information.

I can't point to a quote, but there is an underlying "this is EXACTLY predictable and can be recreated with no guesswork so long as you know the rules" to everything. This permeates through the game all the way down to the ground scatter.

One thing I recall - again about ground scatter that seems to be sticking in my head for some reason - was that in one FFF comments section someone asked why they bother saving all the ground scatter in the save file and why don't they just load it randomly. The answer was one word: "Determinism."

Imagine suggesting to that person just tossing plates on belts randomly.

[Nemo4809]

OK, I believe this could totally work.

Let there be special input and output chests for creating abstract factories.

You place 2 input chests and filter them to have 100 iron plates and 150 copper plates respectively. You then place inserters to extract the plates, place belts and assemblers to make the green circuits along with belt and inserters to drop the the produced green circuits into the output chest.

You highlight the whole setup like you would a BP - you must include at least one input and output chest in the selection. The game will then run a simulation where 100 iron plates and 150 copper plates are pulled out by the inserters you placed and sent off to do whatever your setup does. The simulation runs until everything "goes to sleep" and there is no more activity. The game then checks to see what is produced in the output box and how long the simulation took - how long it would take to process all the iron and copper as if you just let the setup run.

Using the above information, it will create your abstract factory. When you funnel exactly 100 iron plates and 150 copper plates into the abstract factory, that's when it will start up, "process" the materials for the amount of time based on the simulation, and spit out the green circuits (again as determined by the simulation). The factory then "resets" and all remaining material left in the factory - except the material in the input/output chests - are destroyed. Then the next 100/150 plates is processed when it arrives.

Completely deterministic. Throughput might end up lower but it would be very cheap UPS-wise since the simulation need only be done once.

[xfir01]

Sounds kind of like Whistle Stop Factories.
https://mods.factorio.com/mod/WhistleStopFactories

[StephenLynx]

One thing I recall - again about ground scatter that seems to be sticking in my head for some reason - was that in one FFF comments section someone asked why they bother saving all the ground scatter in the save file and why don't they just load it randomly. The answer was one word: "Determinism."

Imagine suggesting to that person just tossing plates on belts randomly.

It can be deterministic. I was thinking about one specific way: you take the buffered input and place them the closest to the start of the simulation. If you produce subproducts, like wires for red circuits, you start also placing the wires after a certain point, since you placed plates in places that would be taken to produce wires.

[StephenLynx]

OK, I believe this could totally work.

Let there be special input and output chests for creating abstract factories.

I don't think it would need special chests. If you know the chest is receiving items from a rail track or having it's contents placed into a rail track, you could use these chests as end points and simulate everything in between.

[Nemo4809]

OK, I believe this could totally work.

Let there be special input and output chests for creating abstract factories.

I don't think it would need special chests. If you know the chest is receiving items from a rail track or having it's contents placed into a rail track, you could use these chests as end points and simulate everything in between.

It necessary so the game know where the input and output points of the abstract factory are and how much material to wait for before starting.

[ssilk]

I read all through and currently I think this could be Implemented as a mod.

To explain that a little bit more:
- works only with factory parts, that are connected by belts only! (No logistic network, no trains). The reason for that is, that nothing else can transport such a constant stream of items.

- needs clear input and output belt lanes, input must be compressed, output needs to be clean. This is because of above: you need to simulate a constant stream.

- if the mod finds such “points” (see below), it needs to measure the real stream for some seconds, which means to measure the input items, the output items and calculate how many items are inside. It needs also to measure the needed amount of power. If that is all more or less constant the simulation can take over. It removes then the complete simulated part (it is stored somewhere) and replaces everything with graphics, that looks like the original and Is animated somehow.

- in the moment when this determinism is broken (input goes down for example), it restores everything. This means also, that - for example - circuit network is not allowed. Electric switch can be used, if the same electric network covers only, completely the simulated part.

- because I think it is very difficult to find such input and output points automatically (that takes a lot of CPU which is the basic reason for this idea, because there is never too much CPU-time left) I think that the player needs to mark input and output belts him/herself. I think to some kind of special belts. That are also the measure points. When done, mod can look then, if this is an allowed point. Alternatively I think to making that with a rectangle area.

- anything that breaks this very fragile deterministic behavior isn’t either allowed or it will fallback to the normal mode.

Even with that it is difficult and results in non-realistic behavior. Some example: blue modules production, green and red circuits in, blue module out. Because it takes so long to produce one module it can happen, that the output isn’t a constant stream of items. Instead it is a peak of items for some seconds and there is some more times when the output is empty.

I don’t know what to do with that, because that cannot be simulated (makes no sense). I think about to limit that to situations, where both - input and output - needs to be (nearly) full.

[darkfrei]

It can be done as another one Factorissimo, where you need two chests, one as input, one for output, one electric energy interface. If the internal factory is done without gaps and bottlenecks, then the production can be simulated just as infinity chest.

[eradicator]

@ssilk:

I'm sorry, but as a modder that is all laughable to be polite. No personal offense meant, i realize you're not a modder and just enthusiastic.

1) Constantly scanning the surface (and all fucking belts?!) to try and automatically detect factory blocks? Insane. If someone managed to do it anyway it would eat all your UPS just for scanning.

> Solution: Let the player mark the area. (Reading further. Oh...you realized that this is bullshit. Good. Because it is :D. But belts are not enough. The player would have to mark a whole area because otherwise you're back at trying to magically auto-detect input-output pairs.)

2) Replacing stuff with fake graphics... how would that even work. Trying to emulate the graphic effect of a belt moving items is impossible to mod in a way faster than just letting real belts do their job. Not even speaking of trying to fake animated inserters.

> Solution: None. Nope. Nada. Can't be done. You'd have to use some "blurry" animated blob that doesn't show any actual content (because suprise: we're simulating the whole thing so we don't need to know the precise content).

3) "Restoring when determinism breaks." Apart from the misuse of the word "determinisim" here (if that really broke you'd just desync/crash), what do you expect to happen? Instantly spawning in large amounts of infrastructure would create lag spikes on the order of *minutes*. Unplayable.

> Solution: None. The whole area would likely need to be invulnerable and only pop back into existance when the player explicitly commands it. Maybe pretend there's a huge wall around it or something.


There's no theoretical problem with any of the in/out rates though. You just measure the input (after all the script has to consume it) and then produce the output. For simplicity each area would have to be limited to one type of output and/or 2 or 3 types of recipe so that there is a clear correlation between input amount and output amount. No need for bursts, just output average over time.

Conclusion: As darkfrei says: the most realistic solution is a factorissimo style blackbox.

[leadraven]

It all sounds like a generic compound crafting machine. Simply make mega-assembler. But for what?
And I don't see any reason to implement it except for the entertainment of developers, but I'm sure they have other thigs to do.