Optimization idea: abstraction

[blazespinnaker]

What I find currently much more appealing than this, is the idea, that you create many of equal “elements” and you calculate only one.
Because of the deterministic nature of Factorio all others must behave the same, if they have the same input.

See this thread: viewtopic.php?f=6&t=91587 Nuclear Reactors BluePrint Cloning

This is as good a thread as any for this topic.

. Adding a new child would mean then just to pause all child’s, until the new child is in synchronized state.

The only way that statement makes sense is if you don't believe a child factory needs to follow the complete production path of the parent. In that case, it is pretty much the same as simply estimating the input/output, but you have placed some concrete, fixed rules around it. I fail to see much benefit over estimation, except maybe it can work better in multiplayer. It certainly wouldn't be a more accurate simulation and some of the constraints around it seem rather awkward for the player to manage for no particular advantage.

More accurate would be what I originally thought you were suggesting, which was synchronizing a child so that it must follow the entire production path (inputs and outputs) of the parent exactly.

A simplified way to think about a production path is this:
Starting from zero state
- at time 1 (after a wipe), you begin with Input at time 1, this creates intermediates plus any outputs at time 1.
- at time 2, you use the intermediates from time 1 + Input at time 2, to create intermediates, plus any outputs at time 2
- at time 3, you use the intermediates from time 2 + Input at time 3, to create intermediates, plus any outputs at time 3
- at time N, you use the intermediates from time N-1 + input at time N, to create intermediates, plus any outputs at time N

We are given the inputs and outputs via tracking the parent factory so do not need to calculate or know the intermediates for a virtual child factory. If a virtual child factory had timely inputs matching the above inputs at times 1 through N (ie, synchronized), then it would be reasonable and accurate to assume that the outputs would be the same as the ones above - outputs 1 through N.

However, total times 1 through N will be arbitrarily, infinitely long unless you periodically reset it to zero state. All of the inputs and outputs at time t will be dependent on the child factory first following the entire series of inputs and outputs from time 1 to t-1

In this case, there is no such practical thing as "waiting until the new child is in synchronized state" if the series length can be arbitrarily long. It could be hours, maybe, perhaps dozens of hours that you would be waiting, depending on how long things had been working.

[ssilk]

I don’t understand what you mean, but I mean you think too complicated.

What I meant with “waiting until synchronized“ is that robots actively bring a sub-factory into synchronized state by adding/removing items (to assemblies, on belts, in inserters etc.), changing modules, “magically” changing production-cycle etc. Then, when everything is exactly as in the other child’s, the production can continue. Yet I haven’t thought deeper than that, because it is clear that this needs some additions to Factorio’s state-engine.

And I don’t want to continue discussion in this thread, it’s already messy. Please wait, for coming thread about this subject (I think today, but can’t promise).

[blazespinnaker]

Sure, but I suggest before you post you think through the implications a bit more as I think you're missing some rather critical subtleties. Determinism is a nice benefit, but making awkward leaps and compromises just to get there doesn't add that much unless you end up tracking the base game closely.

That said, I enjoyed the base idea quite a bit and found it very creative, intriguing and inspiring.

Too bad there doesn't seem to be much interest beyond a very niche crowd

[BicycleEater]

I do love the idea, but a mod could never do it that well.
I don't know, but I'm pretty sure the UPS hit from doing the lua scripting required would out do any benefit. When it comes to performance, an interpreted scripting language replacing hard-coded c++ is not a good move.
The problem with determinism is more than just idealogical, it is fundamental to the multi-player aspects of the game: The game has to be able to ensure that its state is the same in every client, as otherwise they will diverge. If this isn't guaranteed, then the clients and server have to communicate every tiny aspect of the operation of the game.
I do not think that at this stage of development, the developers would add something like this - in my opinion Factorio is mostly focussed on the early to mid game, where most players spend their time, and not the super-late game, where UPS becomes a considerable concern, meaning that these changes would be difficult, bug-inducing, hard to use, and rarely used.
If I were to guess, the best thing might be in train to train areas of the factory, where the inputs and outputs are most distinct, and isolated from the rest of the world, and the change would include a full simulation of the inside, but not in separate entities - the biggest hit to UPS will not be positions on belts, but entity-to-entity interactions, at least from a basic guess.
This would be very hard to make mod compatible - it would have to know the recipes and how they interact.
The hardest thing to simulate would be the odd conditions which limit throughput - e.g. an inserter waiting while the input is full.

Other problems also arise: if the game moves back to reality whenever something breaks the abstraction, if you ran out of power, the game might crash, as it tries to replace everything with the entity version.

[blazespinnaker]

Yeah, scripting can be a hit, but if you use chest like objects rather than belts, the hit isn't bad from what I've seen so far. It'd be nice if we were given some UPS efficient transfer apis tho.

For MP, there are global variables according to the tutorial and that these are synced. There is in fact very little data that needs to be synced up here, so if that works, determinism shouldn't be an issue w/ regards to MP.

Train to train is exactly how i'd like to do it, if we're thinking the same thing. Train -> Simulated Factory -> Train

I haven't had much time so I'm just using the chest connections right now in f2 - https://github.com/MagmaMcFry/Factoriss ... st.lua#L94 but will switch to cargo/fluid wagons when I do get some time.

I dunno if mod compatibility is an issue, tbh. The mod is agnostic when it comes to what it's dealing with in terms of input and output, but it's an interesting point.

Simulated impaired input is the challenge for sure. Not sure that's doable, tbh.

[BicycleEater]

The problem a script would have is hat it has to be faster than the base-game engine, which is pretty fast really. This means that even a small UPS hit from the script would outweigh any benefit, and the script would be quite complex in order to avoid the issues discussed, something lua does not suit.

If I were to write the mod we are talking about, I would disable the machines in question, take items off of the input, handle them internally, before putting them in the output. It would not quite be the same as the game would do it, but should be close.

There would be two ways of doing it:
1: Have the script use one area of the factory as a reference, and the others as 'clones'
2: Have the script handle the whole factory process internally.

1 has the advantage of making the game engine do the thinking, saving the most performance, and being most accurate to the game, but requires everything to be the same.
2 has the advantage of being able to handle individualities of several different factories, including slightly different properties on each input/output, e.g. non-full belts
The whole system would be pretty easy to game - it is unlikely that there wouldn't be a way of persuading the script that something exists in one area, when it doesn't (e.g. powering the reference area, not the others), it would also have to ensure that it can copy the total state of the reference/internal model onto the fake when it had to let the game deal with any differences. These issues could result in lost/gained material, and a cloning system could probably be made.

[Deadlock989]

I do not think that at this stage of development, the developers would add something like this -

What tipped you off?

Was it the Factorio developer who said they thought the team was most likely not interested in the idea, even it if it worked, and it probably wouldn't work?

Or maybe it was the other developer who said it would be ugly if it worked but that was fine because it probably wouldn't work.

Or maybe it was the third developer who pointed out examples where people have made the wrong assumptions about how Factorio works under the hood and that their perceived "abstractions" and "optimisations" either simply don't apply or are already implemented.

Or maybe it was the fourth developer who suggested that people might be better off designing a brand new game from scratch if this is the kind of thing they are really interested in, because the game as described is no longer Factorio.

I mean, it looks like maybe they're not interested, if you'll forgive the rough paraphrasing.

Just a hunch.

[BicycleEater]

Oh, I hadn't read that much about it - sorry about coming into the topic under informed, I was just putting my perspective on it down - I had guessed that no massive opinions had been voiced, as the topic was still under discussion. The fact that they say this is no surprise, and I couldn't agree more that this would basically involve a whole new game (in fact the only reason I said it so weakly was because I hadn't bothered researching it).

Even the way the belts work would be a pain for this kind of optimisation, with lots of dense packing etc. A more Infinifactory style approach to conveyors would be necessary, or a satisfactory style game with more roofs, and less determinism. As I said in my answer, it is also off topic for Factorio, which is fundamentally about the pre-rocket-launch section of the game, so has much more emphasis on smaller factories.

My answers were more focussed on what it would take to mod this in - I'm not sure that is possible either, although something not-as-good might be.

[blazespinnaker]

The code is actually surprisingly simplistic. It's just a matter of adding counters to a table of what gets transferred in and out from the chest connection above and then using that to measure input and generate output where the factory and chests are duped.

In terms of UPS hit, maybe, I'll do some tests this weekend since you brought it up as an issue.

It didn't occur to me for some reason that lua was really that suboptimal. Not sure what engine they are using, tho.

And yes, we're well aware of the lack of appetite from wube on this. That doesn't make the problem any less interesting. Some people like to optimize green circuit output, others like to optimize UPS.

It's just a different kind of game puzzle.

[blazespinnaker]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

Encapsulation and abstraction could help ameliorate BP spraying significantly.

That said, I'm not a huge fan of the phrase and not entirely sure why it's the issue Wube thinks it is. What else do you do with all that robots tech they made?

I think the core issue they should be concerned about is actually newer players spoiling their own fun by reaching out to shared blueprint libraries before launching a rocket on their own. Heavy use of BPs shouldn't be an issue, and in fact is a credit to their hard work making a fun game to play.

[ptx0]

I think the core issue they should be concerned about is actually newer players spoiling their own fun by reaching out to shared blueprint libraries before launching a rocket on their own. Heavy use of BPs shouldn't be an issue, and in fact is a credit to their hard work making a fun game to play.

this seems completely off-topic for this thread, maybe you start a new one to discuss this?

[Koub]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

[Jap2.0]

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

Different terminology; instead of spraying entities over the blueprint, they're referring to spraying blueprints themselves, i.e. having a lot of copies of the same blueprint/making one blueprint for something and using it over and over again.

[blazespinnaker]

There is an interesting point, here. One of the wube developers recently mentioned they wish to discourage "blueprint spraying."

"Blueprint spraying" is partly a byproduct of the relentless amount of the copy/pasting you need to do in order to get reasonable production. Modules help, but by then you just want more production.

I might be wrong, but I think you misunderstood what blueprint spraying was. As I understand it, BP spraying is : place a blueprint out of conbot coverage, grab one of the entities placed as ghost, and spray it over the BP to build only where ghosts of this specific entity are, to build super fast pre-bot. If I'm right, that blueprint spraying thing is indeed quite off topic here.

Not at all, we are absolutely talking about the same thing - in fact that was precisely what I suggested; though I encouraged them to go all the way via ghostplacer. viewtopic.php?p=531615#p531615

But you need to examine the root issue. What leads to such a thing? The need to relentlessly copy/paste. Encapsulation and abstraction would exactly eliminate that need.

As a quick example of what I mean by copy/paste, in order to get production up you can't just build a factory with one of each type assembler, you have to copy/paste the assembler (and assorted inserters/power poles/belts) multiple times.

Imagine now, if you could build a core factory design, encapsulate/abstract it, and use that newly invented archetype henceforth. By reducing redundancy, the repetition leaves the game and need for so called "blueprint spraying."

It's tempting to worry, but bots and blueprints would still have a place - but I think they would be more purposeful and authentic rather than a bit of an artificial byproduct of factorio mechanics.

As an aside, I'm not trying to provoke drama here. I truly believe the point I made and I think if you sincerely examine it further you'll at least start to see where I'm coming from.

But I admit there is a possibility, that copy/paste is easy for folks to get a handle on, and makes the game more palatable for casual gaming. New software developers frequently resort to such WET methods, DRY principle isn't something that comes easy to everyone. Like recursive blueprints, it's possible that there was a conscious decision not to make the game too programmy.

Pushing back on the copy paste though, or at least making it more tedious and not well integrated into beginner game play seems like an odd choice, however. But that point, I agree, is off topic for this thread.

edit: Actually, re-reading your post and mine, one potential misunderstanding / wrong assumption I may have made. I immediately tried to empathize with their root implied criticism and I assumed it could only be either cargo cult or mindless use of blueprints. It never really occurred to me that they thought making blueprints easier to use pre-bots was somehow inherently bad.

However, I'm still optimistic that my initial assumption was the correct one, which lead me to think about how it was the relentless copy/pasting aspect that was causing this problem and how encapsulate/abstract would solve it.

That said, I might have been wrong. Maybe they are all any% speedrunners at heart and see Factorio, at least pre bot factorio, as some kind of challenge of your mouse and keyboarding skills.

[Dimava]

I made a mod for this
https://mods.factorio.com/mod/dupebox
please test and review

[ptx0]

But you need to examine the root issue. What leads to such a thing? The need to relentlessly copy/paste. Encapsulation and abstraction would exactly eliminate that need.

relentlessly? how does encapsulation and abstraction eliminate the need to copy/paste? instead of copy/pasting a blueprint you're just copy/pasting some "abstract idea" of a blueprint.

As a quick example of what I mean by copy/paste, in order to get production up you can't just build a factory with one of each type assembler, you have to copy/paste the assembler (and assorted inserters/power poles/belts) multiple times.

but you can absolutely get away with one building for everything. it takes a lot longer but the game is finishable without it.

some people don't actually use copy/paste until they have bots to make use of it.

Imagine now, if you could build a core factory design, encapsulate/abstract it, and use that newly invented archetype henceforth. By reducing redundancy, the repetition leaves the game and need for so called "blueprint spraying."

so you don't have to think about it ever again? no, to me, that removes the fun of factorio and i'm pretty sure the devs feel the same.

look at how Satisfactory has refused to add copy/paste or blueprints yet because it doesn't solve the problem you're aiming to resolve of being unable to permanently grow the factory.

It's tempting to worry, but bots and blueprints would still have a place - but I think they would be more purposeful and authentic rather than a bit of an artificial byproduct of factorio mechanics.

i have no idea what the heck you mean by that. can you please write a whole page long post about what you mean?

As an aside, I'm not trying to provoke drama here. I truly believe the point I made and I think if you sincerely examine it further you'll at least start to see where I'm coming from.

the same things i don't understand in this post are the same things you've been repeating relentlessly just like the copy/paste'rs you mentioned.

But I admit there is a possibility, that copy/paste is easy for folks to get a handle on, and makes the game more palatable for casual gaming. New software developers frequently resort to such WET methods, DRY principle isn't something that comes easy to everyone. Like recursive blueprints, it's possible that there was a conscious decision not to make the game too programmy.

have you actually played with recursive blueprints? it's not very fun - it's available for the mod, but it's not as popular as just adding more tiers like bob's does. it's nonintuitive and difficult to use.

Pushing back on the copy paste though, or at least making it more tedious and not well integrated into beginner game play seems like an odd choice, however. But that point, I agree, is off topic for this thread.

edit: Actually, re-reading your post and mine, one potential misunderstanding / wrong assumption I may have made. I immediately tried to empathize with their root implied criticism and I assumed it could only be either cargo cult or mindless use of blueprints. It never really occurred to me that they thought making blueprints easier to use pre-bots was somehow inherently bad.

However, I'm still optimistic that my initial assumption was the correct one, which lead me to think about how it was the relentless copy/pasting aspect that was causing this problem and how encapsulate/abstract would solve it.

That said, I might have been wrong. Maybe they are all any% speedrunners at heart and see Factorio, at least pre bot factorio, as some kind of challenge of your mouse and keyboarding skills.

actually, you are wrong. this is why they don't make it impossible to drop items on ghosts, otherwise blueprint use before bots would become more common.

[blazespinnaker]

I was able to get this work at scale (~ 30k spm) with minimal UPS impact from the script. Admittedly I'm using one uber virtual factory duped several times which takes incoming raw resources and outputs lds/jet/rcus and I'm only producing white science at the moment.

The issue I'm running into now is entity update on the miners/inserters. I figured out that I wanted to avoid waiting for space in destination which cut down on UPS consumption, but it looks like the ore -> train loading is becoming a bottleneck now. I don't want to virtualize that, but I may consider using a more optimal inserter mod or make one.

I thought I had 60k spm working, but I had a bug that was doubling output which I caught when I looked at my production and realized I wasn't producing enough raw resources for 60k spm.

I tried and got an 81 wagon train working. With 1000% productivity bonus, individual patches can produce a lot and very quickly and I'm experimenting with longer trains versus shorter ones.

I'm curious what the limit on trains are in terms of length. I remember counting some very long trains as a kid. BHP has the record with 682 cars IRL.


https://en.wikipedia.org/wiki/Longest_trains

edit: hmm, looks like my electric network is the issue. Should be easy to fix and scale up more.

[AmericanPatriot]

I'm curious what the limit on trains are in terms of length

I’m not sure that there is a practical limit. I have had trains with 1000 cars and 100 locomotives before.

[ssilk]

I made a mod for this
https://mods.factorio.com/mod/dupebox
please test and review

Looks cool. I’ll try it out and replace some structures in my biggest map with this.

[ptx0]

edit: hmm, looks like my electric network is the issue. Should be easy to fix and scale up more.

just virtualize the electric network updates