Optimization idea: abstraction

[BicycleEater]

The problem a script would have is hat it has to be faster than the base-game engine, which is pretty fast really. This means that even a small UPS hit from the script would outweigh any benefit, and the script would be quite complex in order to avoid the issues discussed, something lua does not suit.

If I were to write the mod we are talking about, I would disable the machines in question, take items off of the input, handle them internally, before putting them in the output. It would not quite be the same as the game would do it, but should be close.

There would be two ways of doing it:
1: Have the script use one area of the factory as a reference, and the others as 'clones'
2: Have the script handle the whole factory process internally.

1 has the advantage of making the game engine do the thinking, saving the most performance, and being most accurate to the game, but requires everything to be the same.
2 has the advantage of being able to handle individualities of several different factories, including slightly different properties on each input/output, e.g. non-full belts
The whole system would be pretty easy to game - it is unlikely that there wouldn't be a way of persuading the script that something exists in one area, when it doesn't (e.g. powering the reference area, not the others), it would also have to ensure that it can copy the total state of the reference/internal model onto the fake when it had to let the game deal with any differences. These issues could result in lost/gained material, and a cloning system could probably be made.

[Deadlock989]

I do not think that at this stage of development, the developers would add something like this -

What tipped you off?

Was it the Factorio developer who said they thought the team was most likely not interested in the idea, even it if it worked, and it probably wouldn't work?

Or maybe it was the other developer who said it would be ugly if it worked but that was fine because it probably wouldn't work.

Or maybe it was the third developer who pointed out examples where people have made the wrong assumptions about how Factorio works under the hood and that their perceived "abstractions" and "optimisations" either simply don't apply or are already implemented.

Or maybe it was the fourth developer who suggested that people might be better off designing a brand new game from scratch if this is the kind of thing they are really interested in, because the game as described is no longer Factorio.

I mean, it looks like maybe they're not interested, if you'll forgive the rough paraphrasing.

Just a hunch.

[BicycleEater]

Oh, I hadn't read that much about it - sorry about coming into the topic under informed, I was just putting my perspective on it down - I had guessed that no massive opinions had been voiced, as the topic was still under discussion. The fact that they say this is no surprise, and I couldn't agree more that this would basically involve a whole new game (in fact the only reason I said it so weakly was because I hadn't bothered researching it).

Even the way the belts work would be a pain for this kind of optimisation, with lots of dense packing etc. A more Infinifactory style approach to conveyors would be necessary, or a satisfactory style game with more roofs, and less determinism. As I said in my answer, it is also off topic for Factorio, which is fundamentally about the pre-rocket-launch section of the game, so has much more emphasis on smaller factories.

My answers were more focussed on what it would take to mod this in - I'm not sure that is possible either, although something not-as-good might be.

[blazespinnaker]

The code is actually surprisingly simplistic. It's just a matter of adding counters to a table of what gets transferred in and out from the chest connection above and then using that to measure input and generate output where the factory and chests are duped.

In terms of UPS hit, maybe, I'll do some tests this weekend since you brought it up as an issue.

It didn't occur to me for some reason that lua was really that suboptimal. Not sure what engine they are using, tho.

And yes, we're well aware of the lack of appetite from wube on this. That doesn't make the problem any less interesting. Some people like to optimize green circuit output, others like to optimize UPS.

It's just a different kind of game puzzle.

[blazespinnaker]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

Encapsulation and abstraction could help ameliorate BP spraying significantly.

That said, I'm not a huge fan of the phrase and not entirely sure why it's the issue Wube thinks it is. What else do you do with all that robots tech they made?

I think the core issue they should be concerned about is actually newer players spoiling their own fun by reaching out to shared blueprint libraries before launching a rocket on their own. Heavy use of BPs shouldn't be an issue, and in fact is a credit to their hard work making a fun game to play.

[ptx0]

I think the core issue they should be concerned about is actually newer players spoiling their own fun by reaching out to shared blueprint libraries before launching a rocket on their own. Heavy use of BPs shouldn't be an issue, and in fact is a credit to their hard work making a fun game to play.

this seems completely off-topic for this thread, maybe you start a new one to discuss this?

[Koub]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

[Jap2.0]

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

Different terminology; instead of spraying entities over the blueprint, they're referring to spraying blueprints themselves, i.e. having a lot of copies of the same blueprint/making one blueprint for something and using it over and over again.

[blazespinnaker]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

Not at all, we are absolutely talking about the same thing - in fact that was precisely what I suggested; though I encouraged them to go all the way via ghostplacer. viewtopic.php?p=531615#p531615

But you need to examine the root issue. What leads to such a thing? The need to relentlessly copy/paste. Encapsulation and abstraction would exactly eliminate that need.

As a quick example of what I mean by copy/paste, in order to get production up you can't just build a factory with one of each type assembler, you have to copy/paste the assembler (and assorted inserters/power poles/belts) multiple times.

Imagine now, if you could build a core factory design, encapsulate/abstract it, and use that newly invented archetype henceforth. By reducing redundancy, the repetition leaves the game and need for so called "blueprint spraying."

It's tempting to worry, but bots and blueprints would still have a place - but I think they would be more purposeful and authentic rather than a bit of an artificial byproduct of factorio mechanics.

As an aside, I'm not trying to provoke drama here. I truly believe the point I made and I think if you sincerely examine it further you'll at least start to see where I'm coming from.

But I admit there is a possibility, that copy/paste is easy for folks to get a handle on, and makes the game more palatable for casual gaming. New software developers frequently resort to such WET methods, DRY principle isn't something that comes easy to everyone. Like recursive blueprints, it's possible that there was a conscious decision not to make the game too programmy.

Pushing back on the copy paste though, or at least making it more tedious and not well integrated into beginner game play seems like an odd choice, however. But that point, I agree, is off topic for this thread.

edit: Actually, re-reading your post and mine, one potential misunderstanding / wrong assumption I may have made. I immediately tried to empathize with their root implied criticism and I assumed it could only be either cargo cult or mindless use of blueprints. It never really occurred to me that they thought making blueprints easier to use pre-bots was somehow inherently bad.

However, I'm still optimistic that my initial assumption was the correct one, which lead me to think about how it was the relentless copy/pasting aspect that was causing this problem and how encapsulate/abstract would solve it.

That said, I might have been wrong. Maybe they are all any% speedrunners at heart and see Factorio, at least pre bot factorio, as some kind of challenge of your mouse and keyboarding skills.

[Dimava]

I made a mod for this
https://mods.factorio.com/mod/dupebox
please test and review

[ptx0]

But you need to examine the root issue. What leads to such a thing? The need to relentlessly copy/paste. Encapsulation and abstraction would exactly eliminate that need.

relentlessly? how does encapsulation and abstraction eliminate the need to copy/paste? instead of copy/pasting a blueprint you're just copy/pasting some "abstract idea" of a blueprint.

As a quick example of what I mean by copy/paste, in order to get production up you can't just build a factory with one of each type assembler, you have to copy/paste the assembler (and assorted inserters/power poles/belts) multiple times.

but you can absolutely get away with one building for everything. it takes a lot longer but the game is finishable without it.

some people don't actually use copy/paste until they have bots to make use of it.

Imagine now, if you could build a core factory design, encapsulate/abstract it, and use that newly invented archetype henceforth. By reducing redundancy, the repetition leaves the game and need for so called "blueprint spraying."

so you don't have to think about it ever again? no, to me, that removes the fun of factorio and i'm pretty sure the devs feel the same.

look at how Satisfactory has refused to add copy/paste or blueprints yet because it doesn't solve the problem you're aiming to resolve of being unable to permanently grow the factory.

It's tempting to worry, but bots and blueprints would still have a place - but I think they would be more purposeful and authentic rather than a bit of an artificial byproduct of factorio mechanics.

i have no idea what the heck you mean by that. can you please write a whole page long post about what you mean?

As an aside, I'm not trying to provoke drama here. I truly believe the point I made and I think if you sincerely examine it further you'll at least start to see where I'm coming from.

the same things i don't understand in this post are the same things you've been repeating relentlessly just like the copy/paste'rs you mentioned.

But I admit there is a possibility, that copy/paste is easy for folks to get a handle on, and makes the game more palatable for casual gaming. New software developers frequently resort to such WET methods, DRY principle isn't something that comes easy to everyone. Like recursive blueprints, it's possible that there was a conscious decision not to make the game too programmy.

have you actually played with recursive blueprints? it's not very fun - it's available for the mod, but it's not as popular as just adding more tiers like bob's does. it's nonintuitive and difficult to use.

Pushing back on the copy paste though, or at least making it more tedious and not well integrated into beginner game play seems like an odd choice, however. But that point, I agree, is off topic for this thread.

edit: Actually, re-reading your post and mine, one potential misunderstanding / wrong assumption I may have made. I immediately tried to empathize with their root implied criticism and I assumed it could only be either cargo cult or mindless use of blueprints. It never really occurred to me that they thought making blueprints easier to use pre-bots was somehow inherently bad.

However, I'm still optimistic that my initial assumption was the correct one, which lead me to think about how it was the relentless copy/pasting aspect that was causing this problem and how encapsulate/abstract would solve it.

That said, I might have been wrong. Maybe they are all any% speedrunners at heart and see Factorio, at least pre bot factorio, as some kind of challenge of your mouse and keyboarding skills.

actually, you are wrong. this is why they don't make it impossible to drop items on ghosts, otherwise blueprint use before bots would become more common.

[blazespinnaker]

I was able to get this work at scale (~ 30k spm) with minimal UPS impact from the script. Admittedly I'm using one uber virtual factory duped several times which takes incoming raw resources and outputs lds/jet/rcus and I'm only producing white science at the moment.

The issue I'm running into now is entity update on the miners/inserters. I figured out that I wanted to avoid waiting for space in destination which cut down on UPS consumption, but it looks like the ore -> train loading is becoming a bottleneck now. I don't want to virtualize that, but I may consider using a more optimal inserter mod or make one.

I thought I had 60k spm working, but I had a bug that was doubling output which I caught when I looked at my production and realized I wasn't producing enough raw resources for 60k spm.

I tried and got an 81 wagon train working. With 1000% productivity bonus, individual patches can produce a lot and very quickly and I'm experimenting with longer trains versus shorter ones.

I'm curious what the limit on trains are in terms of length. I remember counting some very long trains as a kid. BHP has the record with 682 cars IRL.


https://en.wikipedia.org/wiki/Longest_trains

edit: hmm, looks like my electric network is the issue. Should be easy to fix and scale up more.

[NotRexButCaesar]

I'm curious what the limit on trains are in terms of length

I’m not sure that there is a practical limit. I have had trains with 1000 cars and 100 locomotives before.

[ssilk]

I made a mod for this
https://mods.factorio.com/mod/dupebox
please test and review

Looks cool. I’ll try it out and replace some structures in my biggest map with this.

[ptx0]

edit: hmm, looks like my electric network is the issue. Should be easy to fix and scale up more.

just virtualize the electric network updates

[blazespinnaker]

I made a mod for this
https://mods.factorio.com/mod/dupebox
please test and review

Looks cool. I’ll try it out and replace some structures in my biggest map with this.

woah, totally missed this. Not enough fanfare

Trying now..

edit: played around with it. Looks like a very fantastic start. My one comment is that the way I did it with my hack on factorissimo, the duped factories are not required to be adjacent. This is important as it doesn't constrain train network construction, which at least for me, was the whole point of the mod.

I'm sure others may have different goals, however, and it does help allow for general abstraction which enables DRY factories. This mod will absolutely help cut down on the "relentless copy/paste" problem I outlined above: viewtopic.php?p=531915#p531915

I didn't test time usage at scale for the mod, but theoretically this mod will help folks with slower computers produce the same as those with faster computers (but not using the mod, obviously).

Which IMHO are both significant achievements.

Though as I mentioned elsewhere, the requirement to have same input/output seems rather artificial. There's a phrase for it in software engineering, "leaky abstraction". But again, others may have a different perspective, and as Spolsky said - all non-trivial abstractions, to some degree, are leaky.

I was a bit saddened to see that you had to resort to using remnants. Was that because there was no alternative? If so, perhaps we can ask Wube to enable some technical approach to using their tile graphics.

[blazespinnaker]

One tweak I've been thinking about here, is inspired by ssilks approach, but a bit different and doesn't force the player to deal with artificial mechanics caused by what are really CPU limitations and not really factory related limitations.

Rather than having to stay in sync, a player will create a parent factory and will give it a certain amount of buffer input and then 'switched on'. This buffer input will include everything currently available, including! anything in chests and assemblers inside the factory.

At some defined end period of ticks, the factory will have completed a cycle and the total output will be recorded, including! any intermediates produced.

When a player puts down a virtual factory, it will function only after a similar amount of buffered supplies is made available in chests, and at the end of the same amount of ticks as the parent factory, it will produce precisely the same amount of output - including intermediates. Everything will be made available in externalized outputs.

The advantage of this approach is that it doesn't require synchronization across all the child factories and in fact is more accurate than the sync approach as we also now account for intermediates generated.

The disadvantage is you lose certain flow characteristics of the factory - rather we now have 'production cycles'. However, I believe as a mechanic this could be interesting from a conceptual point of view. Flow characteristics can be difficult to fully grasp sometimes whereas production cycles are nicely compartmentalized for thinking about how that production occurs.

Also, by breaking the factory and cycles into small enough atomic virtual units, you'll still end up with similar flow characteristics at a higher megabase level. However, you'll now be able to better appreciate how that flow is generated as it doesn't require thinking about it down to the raw resource level (except, of course, when you design the atomic factories)

You can make observations such as when having a certain amount of input at time T, then at time T+N, these resources will be consumed and we will be left with the specified generated output. For planning and play purposes, I think it is both more realistic and interesting to me looking at the supply chain graph with its discrete / well defined edges and vertices rather than just worrying about maintaining flow 24/7 on compressed belts.

YMMV.

Yes, output could be impaired compared to a fully flowing factory, but it's not clear as to how bad that impairment will be. It will all depend on the abstractions used and how atomic they are. And, of course, as this will be very UPS efficient, you will be able to produce many x times what you can produce with a non abstracted factory.

You could also maintain a certain amount of flow characteristics by integrating intermediates into the input. This is very similar to the nuclear processing, advanced oil production and coal liquefaction technologies currently in the game. These are fun puzzles indeed.

This approach also aligns somewhat better with my train networks and how I'm devising them as the rules around loading/offloading are more precise now and related specifically to the production cycles.

[ssilk]

Saying that in factory A an item is ready, then so in an identical factory B is wrong. Both needs to be in the same state, when you start this. Then this is valid. It doesn’t depend only on what is feed into it.

Otherwise it could be, that factory A stops production, while the simulation B just continues to produce. Well, that is what players want , but has nothing to do with realistic simulation.

[blazespinnaker]

Saying that in factory A an item is ready, then so in an identical factory B is wrong. Both needs to be in the same state, when you start this. Then this is valid. It doesn’t depend only on what is feed into it.


Otherwise it could be, that factory A stops production, while the simulation B just continues to produce.

In a game without simulation the reverse of your original approach is true, and aligns much better to my tweak. If I copy / paste a factory, and the parent factory I copied from stops producing (eg, I cut its power) -> it has no impact on the child factory which is a separate entity. I don't understand your desire to link the two in the way you are; that is not reflective of factorio in a way I understand.

I don't understand what you mean by state. Virtual factories don't really have state as the entities do not exist. Intermediates as input? But clearing out the factory on initial start is fairly trivial, though the result is not very interesting or fun to me. Handling intermediate input/output backpressure is far more complex and interesting and leads to fascinating designs that you find in things like kovarex / liquefaction production.

When you say " Well, that is what players want". Yes, and no. Some players are like myself and will enjoy the added accuracy and complexity of this suggested improvement, but not all.

Power draw? Sure, but both approaches will need to figure that out and seems unrelated to this.

I encourage you to take a moment and think through the implications of the improvement. Not only is it more accurate by integrating intermediates, it also leads to many simplifications, implementation wise. It will also be exceedingly UPS efficient as you only have to measure at end time and produce at end time. So everything is O(1)

In some ways, this is like a 'viritual meta assembler'. An assembler doesn't flow production as it builds, there is just the input and the output. Eg, a coil assembler doesn't produce one coil at 1/2 time, and then another. You get 2 or none at all.

The key to all this, is the train compatibility. If this was about belts, this solution wouldn't be so great. But once you focus on trains, the concept of shipments and completed production cycles become more interesting than continual flow. Signals and train schedules will be hooked up to combinators which have an understanding of production required and being produced. I see no need to mod for that to make this work.

For example, I'm currently building a city block megabase in my DW. Eventually each block will be a virtualized factory. I am likely going to use common areas per borough for stacking trains. Trains will be released from their line in the stack when a production cycle is complete and ready to be shipped.

In regards to sub optimal production, one interesting aspect to this is not necessarily doing DI from/to trains, but rather pulling/pushing to buffering chests. In that way you can get very close to flow semantics without impairing production.

Belts are still relevant, but only on factory internals, which I think is as it should be.

One thing I'd like to add to this, is the notion of 3D factories. Think like factorissimo, but with factory floors. Pipes / Belts can link to floors. The ability to add virtual factories as floors would be useful as well. Another improvement is necessarily constraining the size of the factory itself on nauvis must be exactly equal to the size of the floors. Dealing with space constraints is a big part of factorio, and the subversion factorissimo does there I think is somewhat inappropriate.

[ssilk]

In a game without simulation the reverse of your original approach is true, and aligns much better to my tweak.

I just answered to your former post, and that simulating a sub-factory like so is not guaranteed to be a correct simulation.

I don't understand what you mean by state.

The state is for me the current condition of a closed system.

Virtual factories don't really have state as the entities do not exist.

They know for example how long it takes to produce an item. Or how long it will take now.

In some ways, this is like a 'viritual meta assembler'.

That’s what it makes this idea for me totally uninteresting.