Optimization idea: abstraction

[Klonan]

That's not great. They can be paused with circuits but not by mods?
So do belts require CPU cycles if some items are on them but nothing is taken on or off them? (Inserters active false).

I'm don't care about circular belts, so the items will be stationary at the ends after a short while. It would look kind of bad. Maybe i need to add circuits to shut the belts off? If that makes them inactive.

If a belt is 100% full, or 100% empty, all the belts in that transport line will sleep.
Don't use circuit network in attempt to optimise things, circuit network typically forces things to stay awake checking each tick if the circuit condition is met

[Qon]

If a belt is 100% full, or 100% empty, all the belts in that transport line will sleep.

That's good at least.

Don't use circuit network in attempt to optimise things, circuit network typically forces things to stay awake checking each tick if the circuit condition is met

Does this mean I can't active = false things that are connected to circuit network? Or can I both connect belts to circuit network and then active=false'ing them, making them both stop and use no CPU? Though I was told belts are immune to active=false and I expect this to also be true then, which is a shame. But asking anyways.

Also, can things become active=true if I set them to active=false (assuming no other mod messes with them) or is it permanent? I'm not sure if active state is the same as the sleep state or if active=false forces sleep but isn't the same thing.

[Rseding91]

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Also setting active=false sticks until something sets active=true through script.

[yagaodirac]

A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.
The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.
In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.

[Qon]

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Also setting active=false sticks until something sets active=true through script.

Ok, good to know, thanks.

Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?

Is there a non-constant cost to modifying active state? If the entities need to be removed from an array of active_entities and it ends up being an O(n), with n being entities in the game, operation to toggle parts of the factory on and off would have to be rate limited to not end up hurting UPS.

[Qon]

A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.

I'm not interested in "cheat" machines. I want everything to be vanilla (and it should be entity agnostic and work with modded assemblers too for the most part), and gameplay should not be altered to the furthest extent reasonably possible. Ideallay I want it to be just installation and then your UPS is just no longer affected by the entity count and you should be able to keep playing as if your computer was infinitly fast and with no affect on gameplay. Vanilla gameplay is amazing already. I just want everyone, even those with potato computers with over the top megabase aspirations, to be able to not be limited by the realities of actual iteration. I want it to be more like Clusterio than an infinity chest, everything should still be managed the same. Actually better than Clusterio because it doesn't include teleportation which alters gobal logistics planning. There might be some ways this perfect mod idea doesn't quite become the utopia I envision, but I'm not interested in using a mod with your suggestion. The point of making a base 100x more productive than any other base built so far should be that you engineer the solutions to that kind of base and don't just replace all the entities with 100x throughput buildings. I want it as close to kovarex vision of "a lot of things" moving, not just big numbers. This is about taking shortcuts in the computation of the game, not shortcuts in the factory.

The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.

I'm not doing it in Factorio chunks.
Also first you say it's impossible, but if I can figure out how to do it then it is possible?

In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.

Actually scaling up production by 10x brings it's own set of new challenges, it doesn't need to be done "differently" by a mod that changes things. Megabases are not built like 100 starter bases, so it is not just "pushing numbers". It sounds like you haven't made any really big megabase in vanilla Factorio in quite a while.

[yagaodirac]

A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.

I'm not interested in "cheat" machines. I want everything to be vanilla (and it should be entity agnostic and work with modded assemblers too for the most part), and gameplay should not be altered to the furthest extent reasonably possible. Ideallay I want it to be just installation and then your UPS is just no longer affected by the entity count and you should be able to keep playing as if your computer was infinitly fast and with no affect on gameplay. Vanilla gameplay is amazing already. I just want everyone, even those with potato computers with over the top megabase aspirations, to be able to not be limited by the realities of actual iteration. I want it to be more like Clusterio than an infinity chest, everything should still be managed the same. Actually better than Clusterio because it doesn't include teleportation which alters gobal logistics planning. There might be some ways this perfect mod idea doesn't quite become the utopia I envision, but I'm not interested in using a mod with your suggestion. The point of making a base 100x more productive than any other base built so far should be that you engineer the solutions to that kind of base and don't just replace all the entities with 100x throughput buildings. I want it as close to kovarex vision of "a lot of things" moving, not just big numbers. This is about taking shortcuts in the computation of the game, not shortcuts in the factory.

The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.

I'm not doing it in Factorio chunks.
Also first you say it's impossible, but if I can figure out how to do it then it is possible?

In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.

Actually scaling up production by 10x brings it's own set of new challenges, it doesn't need to be done "differently" by a mod that changes things. Megabases are not built like 100 starter bases, so it is not just "pushing numbers". It sounds like you haven't made any really big megabase in vanilla Factorio in quite a while.

1, I didn't mean a 100x assembler. If the recipe is 100x assembler into a big one, then the big one is 300x3 in size, and 100x in speed. When it's built on ground, it should look like 100 separated original things in a line. It's not cheating at all. It's cpu friendly only. Nothing logical changes. You literally misunderstood me.
2, Good luck figuring out the parsing algorithm.
3, You shouldn't talk about this idea here. You know amd and intel? Ask them for the solution. But in fact, I just started knowing that, lua can call functions written in cpp. If you mod a interface in lua, take the advantage of cpp and graphics card, with some specially designed algo, I think it might have a chance to do what you want.
4, The most practical way. Mod to remake everything, remove all the useless things. Simplify all the images. If you read through the source code of __base__, you would find a lot things that waste cpu time. If you get rid of all the decorations, all the transition sprites, at least you save a lot cpu time which is on the graphics task in vanilla. This is not cheating at all, it only make your game ugly. I know you wouldn't mind this.

[eradicator]

There was once a mod that implemented "furnace arrays" (can't find it on the portal right now). Basically the idea is to make a fake entity that looks exactly like n furaces in a row, and is n times as fast. That approach has it's own set of problems though: It only works for rows (or whatever pattern you decide to optimize), it's not hideable from the player because mass-entity replacement is expensive, and the "array"-entity has no way to enforce "normal" base layout, the player can just input/output with a bunch of loaders on one end. Also beacon effects get totally messed up due to the sheer size of the entity.


@Qon: What i was wondering is: How do you intend to create items with the correct rate when the machines are off? Spawning stuff via lua is too expensive and infinity chests have very limited rate control (only full integers each tick).

[Qon]

Not exactly sure what 1 - 4 are responses to, but I'll try to answer anyways.

1, I didn't mean a 100x assembler. If the recipe is 100x assembler into a big one, then the big one is 300x3 in size, and 100x in speed. When it's built on ground, it should look like 100 separated original things in a line. It's not cheating at all. It's cpu friendly only. Nothing logical changes. You literally misunderstood me.

I took into acount that you might have meant a faster and larger assembler. But it doesn't change things that much. Larger assemblers prevents you from using your own blueprint layouts, ie it limits your engineering and kind of forces you into using someone elses layout/blueprint. It also forces you to rework your factory to use these kinds of entities and you can't enable or disable the mod on the fly.

And to supply the 300x3 assembler with items you need a lot of inserters (which are more costly in terms of CPU than assemblers). Or you need some 300x1 inserter. Fed from some 300x1 logistics chest or some kind of super belt. And now the speedup becomes limited or you would have to incorporate it all into a single unit. But now you have completely abandoned actually making your design, you literally have a big box that just produces things super fast and all you need is the space for it. You don't need to think about how to incorporate roboports and radars into the design (which is needed when making exceptionally large structures to make them buildable from map, which is a part of the challenge of building at this scale).

And this only covers a single step, you are limited to a single recipe which limits the speedup. Unless you want to make an assembler that takes iron and copper plates and spits out green chips. Or ores and spits out science packs. But then you have removed the whole building and designing aspect of the game. This method removes the game from the game. I don't like it.

2, Good luck figuring out the parsing algorithm.

Thanks, it's going to be non-trivial. But I believe I can do it with enough time and effort, with some slight imperfections allowed.

3, You shouldn't talk about this idea here. You know amd and intel? Ask them for the solution. But in fact, I just started knowing that, lua can call functions written in cpp. If you mod a interface in lua, take the advantage of cpp and graphics card, with some specially designed algo, I think it might have a chance to do what you want.

Not sure what I shouldn't talk about here... maybe you can say what you are talking about instead of saying "this" and quoting my entire post!?
Also you can't call C++ librararies from Lua code executed by Factorio. That would break the security model and the determinism guarantees and everything else.

4, The most practical way. Mod to remake everything, remove all the useless things. Simplify all the images. If you read through the source code of __base__, you would find a lot things that waste cpu time. If you get rid of all the decorations, all the transition sprites, at least you save a lot cpu time which is on the graphics task in vanilla. This is not cheating at all, it only make your game ugly. I know you wouldn't mind this.

There are mods that reduce the graphics. But Factorio is RAM bandwidth limited and CPU limited. The graphics are to some extent handled by the CPU, but reducing animations isn't going to give you 200k SPM. Basically only things on screen this tick are affected by these kinds of changes, and most of it is handled by the GPU anyways. And what is on screen is nothing compared to the 99% of the factory that is outside your screen, which is what I want to abstract away.

Also I prefer the game to look good if possible.

Your suggestions don't provide nearly enough benefits compared to all the compromises they introduce and compared to the much, much faster abstractions I'm thinking of implementing. If I can just do it. If I can active=false all entities and just do a handful of API calls to remove a few tiles of ores and just change the research progress bar by a calculated amount once per minute I can achieve about 0.0% CPU utilisation for an entire gigabase, including the mod. And it will all be dynamic with no interaction by the player and "no" restrictions.

Listen, I'm not talking about speeding things up, I'm talking about eliminating computations. Like almost all of them. Making things 100 times faster is not enough, I'm going for something closer to constant time compution for any factory size and for any time step. The ideal is to be able to set game.speed = 100 and have it run at 6k UPS with a 1M SPM factory. That is going to run into many practical issues which means that might not be possible. But I want the closest thing to that and which is still possible. Without altering the factory building process. If I can calculate that a mining outpost will last for 100 hours at the rate I'm mining from it (total/mining rate) and that it produces X science/minute, then why not just use that calculation instead of simulating all the steps? As long as the player playing the game can't really notice any difference then that should be good enough.

[Nidan]

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.

[Qon]

@Qon: What i was wondering is: How do you intend to create items with the correct rate when the machines are off? Spawning stuff via lua is too expensive and infinity chests have very limited rate control (only full integers each tick).

I will only be able to work with buffers, inserting individual items is too expensive. That means a single Lua call/chest once every X minutes and fill it completely with that single call, X being minutes it would otherwise take to fill it. Filling a chest with a call should make up for all the overhead of Lua compared to all the entities required to fill the chest normally.

Also I need to be able to abstract whole outposts or factories with several steps, skipping intermediate buffers completely. I want ot measure from as early as possible (potentially ore on ground) to as late as possible (research progress bar) and spawn as little items as possible. Most things should preferable only be created as a number in the stats window and would never actually be realised as an item in the world.

This means some factories will not really be abstractable, at least not if the player is within them and they are constructed to break all my assumpions. Like as long as they contain buffers that fill up without any local consumers (like a mall that hasn't been filled up completely). And you would need buffers in your factory and reasonably sized outpost blocks to be able to abstract partial factories efficiently.

[eradicator]

I will only be able to work with buffers, inserting individual items is too expensive.

Ah, so it'll only work for "natural" buffers and you're not going to artificially insert any. Interesting approach. As far as i remember the input/output slots in assemblers ignore the maximum stack size. That might or might not be better than size-limited chests.

Most things should preferable only be created as a number in the stats window and would never actually be realised as an item in the world.

As far as i can see production statistics are not moddable.

[Rseding91]

Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?

Inserters have such a high update cost that adding the circuit overhead and stopping the inserter still comes out ahead. But yes the inserter control behavior never sleeps. None of the active control behaviors sleep because they have nothing to run events off of. The circuit network is constantly changing each tick as all of the inputs get summed together and sent out over the wire(s).

Is there a non-constant cost to modifying active state? If the entities need to be removed from an array of active_entities and it ends up being an O(n), with n being entities in the game, operation to toggle parts of the factory on and off would have to be rate limited to not end up hurting UPS.

Setting active=false/true is a constant time operation.

[Rseding91]

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.

Event-based is inventory: on-item-added: send event, on-item-removed: send event. Circuit logic is polled: each tick go over each thing that has signals to send and add them together. Then that's the state of the circuit network for that tick. Next go over each thing that wants to read it and have them do their logic.

There's no event for fluids flowing into a storage tank that's connected to a circuit network. There's no event for any of the circuit network; it's all sum-all, iterate users and calculate.

[Qon]

I will only be able to work with buffers, inserting individual items is too expensive.

Ah, so it'll only work for "natural" buffers and you're not going to artificially insert any. Interesting approach. As far as i remember the input/output slots in assemblers ignore the maximum stack size. That might or might not be better than size-limited chests.

It's worse. And probably not for the reason you think. It doesn't matter how fast it is to modify assemblers. The assemblers still need the output extracted from them. That means I can't abstract away the inserters and the belts and the inserters again to the train station chests. This could work for production outposts with assembler to train direct insertion, but those are almost never built so it seems like a waste of effort.
I don't want to abstract away assemblers. I want to abstract away everything. Except combat and building and some things that are not feasible.

As far as i can see production statistics are not moddable.

Code: Select all

game.player.force.item_production_statistics.on_flow('automation-science-pack', 10000)

[Qon]

Thanks for the info. Some more questions

Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?

Inserters have such a high update cost that adding the circuit overhead and stopping the inserter still comes out ahead. But yes the inserter control behavior never sleeps. None of the active control behaviors sleep because they have nothing to run events off of. The circuit network is constantly changing each tick as all of the inputs get summed together and sent out over the wire(s).

Inserter control behaviour never sleeps? Even when active=false?
Which entities are not affected by (and/or don't stop requiring CPU cycles if that is something separate) being active=false except belts, underneathies and splitters? Inserters and circuits and wired things also?
Is there a complete list of things that (a) continue to work while active=false and a complete list of things that (b) might stilll use CPU while active=false?

And how does the debug option "show-active-state" relate to this?
The red ring is on inserters that are disabled with a circuit wire (regardless of active value)
and those that can't insert because target is full while (regardless of active value)
for me and on inserters that are stopped mid-swing by being active=false.
So does this mean the red ring doesn't signify that things don't use CPU cycles, since from what I understand now some of those things with red rings still use CPU according to what you described?
Tested with

Code: Select all

/c local ents = game.player.surface.find_entities_filtered{position = game.player.position, radius = 100} for k, v in pairs(ents) do v.active = false end

[eradicator]

It's worse. And probably not for the reason you think.
[...]
I don't want to abstract away assemblers. I want to abstract away everything.

I understand that just fine. But you also said that you want to keep the visible part running - why else would you need to even consider filling chests at all. But belt based factories usually don't have any chests you can abuse. Anyway it was just a bit of info. You're the only one who knows how you want to get this done. I certainly have no idea how one would make parts of a long belt seem like they're moving when everything around them is actually frozen. And train-based city-blocks is gonna be pretty obviously ugly on the map view :D.

As far as i can see production statistics are not moddable.

Code: Select all

game.player.force.item_production_statistics.on_flow('automation-science-pack', 10000)

Sorry, I was fooled by the read-only tag.

Code: Select all

item_production_statistics :: LuaFlowStatistics [Read-only]

[Qon]

I will simply not freeze things in view. And anything that is connected to the thing in view in a way that isn't separated by a buffer will unfortunately also have to be running. That is why I needed active state to be quickly toggleable, or at least one reason.

Belt based factories with no buffers at all are a clear weakness of this method. There might be a need to alter the design slighty in that case to take advantage of the speedups. You would only have to do a handful of "cuts"/buffers to allow the mod to abstract away things out of view though. Maybe interacting with assembler buffers would be an option then to avoid those issues. If its possible to abstract away all mining and smelting or similar impact then that might be worth considering.

Outposts by train would look the same from map view when zoomed out. The trains will still run (if the (complete?) Rail network isn't abstracted. When zoomed in to world I would have to use selected entity position instead of player position to decide which parts to run.

[Rseding91]

When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:

Entities
Belts
Pipes
Electric flow (producer -> consumer)
Control behavior
Circuit Networks
Heat flow
Trains

[Qon]

When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:

Does that mean that active=false only affects the entity loop and every other loop is unaffected?
Are inactive entities affecting the CPU for power calculations? They still have power drain in my test. But regardless I'm imagining that if all solar panels can be virtually combined as one entity then the same should be possible for all entities with drain (min consumption) also?

Pipes are not affected by .active it seems. Do full/empty pipe systems also constantly use UPS? If the sources and drains are all inactive then that should mean that no update will affect the state of fluid levels anyway.

Setting active=false on all my entities improved UPS, but destroying all entities on the surface improved UPS more. Trains were not affected.
If trains are stationary at a stop waiting for some condition (circuit/time/inactivity/inventory I guess are the main ones) to leave then will it use UPS when inventory isn't being updated? They have no real reason to do path updates when they are stopped, I hope.

Is there somewhere I can read up more on how the game engine handles things like these? Or is there some way to find it out for myself? show-active-state doesn't really tell me if things stop using the CPU since there's things like control behaviour (what is that?) that keeps draining my UPS needlessly.

I'm doing some experiments to see what can be stopped and if I need to actually .destroy() things to stop the engine from wasting cycles on things. I might need some more powerful pause entity functionality for the mod to actually speed up things properly.