Cloning as massive UPS improvement (Factory-Streets)

[ssilk]

TL;DR

Factorio's fully deterministic game engine enables, that if we have two completly identical areas, and both have the same input and output (of items, power, other events...) that both behave the same.
That can be used vice versa: Instead of calculating both areas, we need to calculate only one of them and use the results to render both. Which can be used to set up very big areas of cloned production chains to create crazy big factories.

What?

The idea was born from that threads:
viewtopic.php?f=6&t=82060 Optimization idea: abstraction
viewtopic.php?f=6&t=91587 Nuclear Reactors BluePrint Cloning
Also
https://mods.factorio.com/mod/Factorissimo2

Edit:
https://mods.factorio.com/mod/dupebox comes in in mid January 2021


This shows basically the idea

A number of cloned basic-circuit production. Every production-site is surrounded by a fence (here a wall) to prevent accidental changes. The underground belts are the INPUT and OUTPUT. The lights at the side symbolize the OPERATION-MODE. All other clones need to be in the same electric network. You see this example is not perfectly synchronized: The items on the belts are in different positions. Not all lamps are on. This cannot happen, when fully synchronized.

Screenshot 2021-01-06 at 17.16.32.png (808.76 KiB) Viewed 6086 times

Abstraction is an idea, which can be used only either by near estimation of what comes out from a production or under very distinctive circumstances only. The cloning could solve all of that issues (but brings of course some others ).
It's very difficult to explain that idea, so first some

Glossar

container: one production-site, a sub-factory, a plant, a child.
cluster: A factory-street, many cloned production-sites. See picture above.
master: origin, there is - like in Factorissimo - an extra Factorio-surface (see wiki), which contains the original production-site. When in operation-mode (green lamp), that surface cannot be changed.
(virtual) clone: A child. This is - basically! - the copied rendered display of the master surface in the operation-mode green. There are no real entities and so on. The technical difference to a container is, that you will not always see the clone; when editing or so you are operating on a copy of the master.
synchronization: This is a process, that brings all clones to the exactly same state as the master (or all other clones). There are different methods to achieve that. The simplest is to "reset" all entities (just the same state as if the entitiy is just places, without any items)
state: Everything that can change in a container.
operation-mode: There are different operation-modes.
Green: Synchronized, working, input is full and output is clear (and so on, see below).
Yellow: The player is editing. All changes are made to all containers.
Red: Unsynchronized, after editing, placing etc. Something prevents the automatic synchronization to work. For example the container is not in the same electric network.
Blue: Synchronizing. See below.

Details

How to build a cluster?
1. We build a production-unit in a way, that has distincted input- and outputs. Nothing really complicated or different than yet.
2. Containerizing it. We have a tool similar to a blueprint to draw the borders of the container. This automatically will add the fence, the special underground-plates, the input- and output-gates, power connection (only one is allowed), and the operation-state-lights.
3. When we have done that, we cannot simply change the container anymore. We need to enter the container for that, which turns automatically the container (the whole cluster) off.
4. We can just copy paste that container: Click on the container (selects the whole container), copy it, then paste it like a blueprint. Or mark an area around the container, which copies the container plus the marked area.
5. Copying a container automatically links the clone to the master. To see, which are the same clone all linked containers are marked, when hovering over one.

How to edit/change a container in a cluster?
By defition a container is a copy of the master. Which means: If you edit one container, you edit the master and with that all other containers.
This has the advantage, that after editing all containers have the same state, no synchronization is needed. But the disadvantage is, that the containers change "magically", when you edit one.
Many things can be done, to remove that kind of magic, but I prefer this behavior, because it is just fast and straight-forward.

When is synchronization needed?
Only when you place a new container. For that case it needs to be synchronized with the existing. When you edit a container, the master is edited and with it all cloned containers.

How does synchronization work?
There are many ways to achieve that. I describe two:
Force reset
This means, that all containers are reset to the state, as if freshly blueprinted. No items, all entities are reset to their initial state.
The question is, what to do with the items, that had been cleared from belts, assemblies etc. I leave that question to the implementer.

Soft sync
This means, that the cluster needs to be stopped (all containers stopped) and are brought into the same state. Items are requested to be placed by bots on belts and into assemblies (like with modules). As long this takes, the containers are in blue-operation mode.

How to keep the cluster synchronized?
The synchronized cluster (all green) works only, if
- there is enough items/fluids on every input of every container.
- there is enough space left for the produced items/fluids on every output of every container.
- if all containers are at the same electric network.
- if all signal-cables (in- and outgoing) are on the same common circuit-network.
- any edit of a container changes the master and so it changes all containers.
This is of course much more complex to implement and needs some changes in the Factorio engine. And I think this is not really needed, if editing a container is equal to edit all containers.

What happens if not enough input or output is full?
This is simple: The whole cluster stops working until all inputs are filled/all outputs are cleared. The lamp symbolized symbolize that (eg. green blinking). The container, which has not enough input/too much output shows extra lights (yellow blinking?).

To avoid too much waiting time I can think of small buffers in the inputs/outputs.

Circuit network?
You can use input- and output-circuit networks, but like the electric network all inputs/outputs needs to be on the same network.

The fence takes too much space. Why I'm not allowed to change a container directly?
I didn't say that the fence needs an extra tile more space. This is implementation detail.

The fence is to prevent from accidential changes.
Technically the player operates then on a different Factorio-surface.
Think also to multiplayer: There can be multiple players in different containers, editing the same master at the same time. Dunno, if that is a good idea.

How can this be implemented as mod?
Take Factorissimo, implement the cloning/linking, a hard-reset-sync to all clones, and the rules that keep the cluster synchronized (see above). Then we only need to have the logic to pick up the items from all inputs and multiply the production for all outputs. In truth only one surface is calculating. It might depend on the size of the internal structure (bigger = better), if we will really see big performance gains.

Can I turn off single containers?
Makes no sense, because afterwards the cluster needs to be re-synchronized.

Can I turn off the cluster?
Maybe. I think to a similar mechanism as the power switch: Every input/output has a logic like the power-switch. If turned off, the input/output is closed. Output still works. That enables to "clear" a container from as much items as possible and when all items are used (nothing is produced anymore) an power-switch inside the container can turn the power off for all.

Why?

- Incredible performance/UPS gain!! New targets for mega/giga/penta-factories.
- A completly new way to build interesting factories. The main target is here to serve enough items to all containers and transport the produced items away. You cannot simply rely on belt queues.
- New things that can go wrong: If you have too much containers this works extremly ineffective. That is a completly new game principle.
- Super nice looking. See the above picture. Everything is synchronized. It will look like a ballet.
- Thinking not in pure item-production, but in balanced production-chains. It will make much more sense to have one chain for all red-, green-, purple-, and so on -bottles.

[NotRexButCaesar]

Is it really possible for something this big to be implemented so late?

[ssilk]

Why do you think this is big? The basic elements are already here. What needs to be implemented into vanilla is basically
- display a surface on another surface (many times), could be used for many other things.
- handling of the player that he changes that surface
- implement all the input/output exchange points (linked chests are already implemented)
- implement the operation-modes, the logic, when the cluster stops.
- implement synchronization ( nice to have: soft sync)

Of course many small things more, like the fences graphics, etc. But I think this can all be estimated yet, and there is no big unknown thing, that could prevent implementation. This suggestion has a very high gameplay-value, while its costs are only medium high. (Or I forgot something important )

[blazespinnaker]

Is it really possible for something this big to be implemented so late?

Another possibility is that wube enables some LUA apis which make it easier to do this (or something similar) in a highly playable and highly efficient UPS manner with a mod. Adding some fun way of making duped factories more UPS efficient adds a huge amount of playability to the game, especially by enabling larger and more complex train networks. It'd be fun dealing with biters late game, but they're kind of eliminated. For the worse for the game, sadly.

Maybe make it an extremely costly infinite tech or something similar. That plus a few more infinite techs would be quite enjoyable.

That all said, I posted here and on reddit about supporting UPS efficiency with a late game mod. Very little traffic or engagement. I'm skeptical there is much demand here in numbers except from an overly dedicated niche few.

Megabases where UPS is an issue take some time and dedication to build, probably not many folks have that kind of time to spare to a video game. Or they get bored of factorio first.

Sometimes when you're a super fan getting tunnel vision about what matters to others can be too easy.

Hopefully we'll see better engagement on this thread. I'm not hyper optimistic tho. (Prove me wrong, please! =)

[blazespinnaker]

Why do you think this is big? The basic elements are already here. What needs to be implemented into vanilla is basically
- display a surface on another surface (many times), could be used for many other things.
- handling of the player that he changes that surface
- implement all the input/output exchange points (linked chests are already implemented)
- implement the operation-modes, the logic, when the cluster stops.
- implement synchronization ( nice to have: soft sync)

Of course many small things more, like the fences graphics, etc. But I think this can all be estimated yet, and there is no big unknown thing, that could prevent implementation. This suggestion has a very high gameplay-value, while its costs are only medium high. (Or I forgot something important )

One thing to keep in mind, is that there are two ways of looking at this.

If wube enables it, versus doing it with a mod.

If wube enables it, than I would probably reduce the suggestion to a few sentences. Prescribing how it should be done it really isn't necessary or efficient. There are an infinite number of different ways to do something like this when you have core code access, and without access to that core code, skill sets of developers, time available, goals of the business, and deep understanding of the entire userbase (plus concrete statistics on how they play), it's pretty much impossible to design something as optimal as if you do have all those things.

Biasing them on an approach runs the risk of neglecting coming up with a better way. Better would be for them to propose something using their expertise, perhaps with alternatitives, and for the community to provide feedback.

I will say, if I had core code access, synchronization would not be how I would do it. That seems like artificial game play (factories don't need to be synchronized in the RW) and really my interest in it is only a side effect of the fact that it is doable way of achieving accurate factory simulation with a mod.

If it's done as a mod, than I think there are a really a large number issues with the above, but I don't think you had doing it with a mod in mind so I'll avoid commenting from that perspective. That said, I think doing it as a mod is probably the most likely way this will be achieved. We might be able to convince wube to add some apis to help achieve this.

[ssilk]

Blazespinnaker: can you please avoid double-postings? And please keep a bit more on subject: it’s not needed to write two posts about the same subject in two different threads, that you think, nobody is interested in more UPS.

I had of course mods in my mind as you can read in OP. That’s the reason I put so much details in that post.

[blazespinnaker]

Blazespinnaker: can you please avoid double-postings? And please keep a bit more on subject: it’s not needed to write two posts about the same subject in two different threads, that you think, nobody is interested in more UPS.

I had of course mods in my mind as you can read in OP. That’s the reason I put so much details in that post.

Hmm does reply notify properly if I do everything in one post? I find people reply to my posts but I miss them because I don't get notified.

Which particular comment is not about this topic?

Where did I say nobody is interested in more UPS? That is a very unfair and rather hostile misreading, if I may so, of what I said. I said it is a limited and niche audience. This is an important point because there are financial motivations behind whether or not this make sense as a game suggestion, or even if it makes sense discussing it as one as AmericanPatriot pointed out. Better would be to discuss it as APIs you'd like to see to enable it, IMHO.

I'm not sure why you think this would work as a mod. You'll want to think through the implementation a bit more carefully. I've commented elsewhere regarding that, so I'll let it stand. I do however see that you've integrated some of my recommendations, though without giving me credit. A tad anti social, tbh, but I get it, I was like that once too.

If it is a mod proposal though, you probably want to post in the mods section and think through the apis that you will use and how they work. If you are unable to do that, there is no sin in asking for help.

Mods versus game suggestions really are two different kettle of fish.

There really is too much artificial / forced game activity here to make it worth considering as a game suggestion, especially around synchronization. I don't mean to sound critical, I obviously found the idea intriguing, creative and inspiring, but rather I'm just trying to short circuit some unnecessary discussion as a game suggestion proposal.

TBH, I think the approach your taking (prescribing a game feature but not really thinking in detail about it like a mod) discourages both approaches by giving neither a fair and dedicated shake. Though, like I said above, I don't think in the end it's that big of a deal as it's a pretty niche audience that is interested.

[Deadlock989]

These "abstraction" threads remain hilarious.

If people don't feel like wading through the whole torrent of spitballing above, this is the TL;DR:

Take Factorissimo, one of the most UPS-intensive mods ever made for Factorio, and use it to make magical UPS savings to a game written in C++. These savings will be enormous because apparently we all have multiple huge areas in our games where everything is absolutely identical almost all of the time, down to the spacing of items on belts and the angle of every inserter arm, and they're all fully supplied all the time and they're all fully powered all the time and they're all using the exact same pair of circuit networks all the time. If there is ever a tiny change in those conditions, such as tiny gap in one belt somewhere, then massive sections of the world will suddenly grind to a halt and freeze in time, and that won't look weird or be frustrating, in fact it will literally be artistic. Of course all these massive parts of the world have to be indestructible and unchangeable because it's not like the game has literal monsters rampaging through it biting things or anything like that and no-one ever edits their bases once they've been built.

I think some people have taken their interest in optimisation too far, past the point where things actually make sense, and certainly far past the point where it would be a game that is fun to play. I certainly wouldn't play it.

[blazespinnaker]

Now now, if you're not interested, best to just not post.

You can really say this about anything.

There are frightfully large discourses on golf, golf etiquette, golf clothes, golf technology, golf anything that I frequently shudder to think about. I don't go there and tell them they are wasting their time, however. I'm sure there are people out there who would question any amount of time you have spent playing or talking about factorio itself.

The motivation behind much of this, at least for me, is I have a base that has hit its UPS limits. I am interested in devising a larger and more complex train network, but there really isn't much game motivation, because everything already is getting what it needs. So I'm using a hacked version of factorissimo to cut down on all the redundant copies of factories.

Also, there is an interesting aspect to thinking about factorio in terms of core factory designs rather than thinking in terms of relentlessly duplicating them.

Yet another way of looking at it, is the problem itself - how to encapsulate factories - is just yet another puzzle to solve much like anything else in factorio.

[ptx0]

[...]
Where did I say nobody is interested in more UPS? That is a very unfair and rather hostile misreading[...]

ssilk is actually very much in your corner on all of this, it only seems like they're trying to help you present your idea in a way that will get more engagement that you were asking for earlier..

i was going to comment and ask how you plan on handling certain problems, but you've already indicated you aren't interested in my opinions, even though I also would very much like to build larger before UPS is a problem, I disagree fundamentally with how you've chosen to go about it.

personally, I'll go for things like "Schall Machine Scaling" which present their own logistical challenges - they're enormous buildings that cost the same as a large number of vanilla buildings to craft, and consume a scaled number of inputs/outputs.

in other words, 16 assemblers/chemplants/refineries/nuclear reactors are condensed into one large entity that maintains all of the vanilla gameplay characteristics - no need to surround them with walls or have perfectly synchronised "anything" going on - they're just normal machines using the existing C++ prototypes, highly performant, and very reliable - there's a recipe scaling mod that helps to make them work better in the game engine.

I still have to manage their inputs and outputs effectively and without a loader mod, this is more of a challenge than you'd think - though their increased footprint gives more I/O possibilities.

that said, it just seems that you might be, as some people might say, "a bit too attached" to your idea. people find it difficult to discuss the implementation when you take everything as an attack. even deadlock989's input has value.

[blazespinnaker]

It's not that I'm not interested in your opinions, I do read them for sure. You tend to be a little more reactionary than most, but I don't mind. It's all relative. The issue really is I just think the mods would prefer we didn't do the back and forth thing that inevitably results.

[Impatient]

Oh, that escalated quickly. Grabbing popcorn.

[blazespinnaker]

Going to ignore the pointless attempt there to provoke drama.

Just to be clear, in case there is some confusion, this really is all ssilks idea, plus maybe a few minor suggestions from me.

I did find it very creative and innovative and quite inspiring, as I mentioned above. Mostly the issue is that the mechanics, as I worked through them, seem a bit forced.

But it could be the seed of something even better, so seriously worth considering.

[Jap2.0]

[Moderated by Koub : off topic]

[NotRexButCaesar]

[Moderated by Koub : off topic]

[Jap2.0]

[Moderated by Koub : off topic]

[Deadlock989]

[Moderated by Koub : off topic]

[Jap2.0]

[Moderated by Koub : off topic]

[NotRexButCaesar]

[Moderated by Koub : off topic]

[Jap2.0]

[Moderated by Koub : off topic]