Optimization idea: abstraction

Post your ideas and suggestions how to improve the game.
Hannu
Filter Inserter
Filter Inserter
Posts: 803
Joined: Thu Apr 28, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Hannu »

I do not agree. What is fine thing in Factiorio is that there are belts transport and inserters move every item exactly and factory seem to be working real thing. Practical limits of Factorio are huge. Far more then can be shown of seen. I do not see any reason to go for larger number of entities. And I am especially against replacing simulated function with boring black boxes just to be able to show larger numbers on production stats.

Maybe Factorissimo like mod, which handle factories as simple black boxes, would be good if there is much demand for this kind of thing.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11423
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

For me this is not really needed, because - I already talked about it - there are some mods, that will make the need for speed of this suggestion more ore less obsolete.

That are these points:
- boxing mod (deadlocks stacking inserters and extensions are able to assemble stacked items with stacked items. That means to produce 200 (a stack) of green circuits you need only one update of the assemblies, instead of 200 and you need to belt transport 1 item instead of many.

- Klonans Mining drones. That mod reduces also the needed CPU power.

- transport drones (also Klonan). This mod in conjunction with the boxing is sooo cool. No big bus. Instead streets with a lot of traffic.

All in all this brings a lot of CPU power back. I’m yet not sure how much. But it seems to be 3 times faster with the same amount of produced items.

What I mean: this suggestion/solution is in my eyes too obvious. Just make everything faster, bigger, more. And if it the simulation is not really exact anymore: who cares? (I know what I said some month ago, but I changed my opinion).

Instead of that in my eyes it’s about making things more clever, neater, smaller. Which the above mentioned combination of mods does.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

I've been thinking of making a mod for this (or something similar at least, to make it possible). I'm almost far enough with my specification to start coding it. But I'm too busy at the moment to do it. But it might be my next big mod project.

This post is going to be a bit of a raw brain dump that might not make sense if you haven't read the topic, or maybe even then.
Goals
My goal is to maintain 60 UPS pretty much regardless of factory size and having production/minute be as close to vanilla as possible but not really paying attention to /tick production stats. It wouldn't be exactly as the devs intended it with each item actually being accounted for precisely. And I do love that about Factorio. But the game is great without it also and it doesn't hurt the game if it's implemented as a mod. If players can continue to play and expand their factories instead of abandoning them because they start dropping UPS then at that point the tradeoff seems to be pretty ok.
Features and requirements spec:
I absolutely need active = false to mean that the entities don't use CPU cycles. This makes it "simple" and fast to enable/disable abstraction quickly. And belts and bots etc will stay in position carrying material as if it was always running. With the assumption that the running factory looks "the same" all the time it should be OK to just pause the belts and bots and then unpause when a player comes to inspect. If materials have dried up at inputs inputs then that's just some buffered items that can be accounted for and won't really affect actual production but just when those things get used.
  • It should just be plug-and-play, install the mod and after a while (if it's added to an already running big base) UPS should increase so that you can play without slowdowns any more.
  • No input output marking
  • to greatests extent should work with any building style
  • no new entities
  • This of course excludes Factorissimo style buildings and building with blurry glass roofs etc. The actual entities represent themselves, just active=false'd.
  • This also means the player have to manage the factory just the same as if there really was proper simulation of all the items. No boxing of items that reduce belts, all the same space requirements, no difference to blueprints or calculations need to be made etc. Should "feel" like the real thing, except better UPS.
techical features:
  • should be compatible with most mods
  • should be able to handle belts, pipes, bots and trains in local networks. Maybe even new logistics methods like Klonans transport drones could be handled, maybe even without specific code to handle those if they are used within local production cells, but that might be strecting it.
  • shouldn't be need recomputation for non-affecting edits
  • should enable and disable parts as seamlessly as possible, instantly. After initial measurements the goal should be that only what you are currently working on and looking at is working. As soon as you look away it becomes abstracted and the new thing you look at is running instead.
  • Less power means slower production, abstraction just adjusts proportionally (depends on power abstraction precision).
Limitations of requirements:
For this to be achievable I'm fine with minor losses of accuracy. A 20.00k SPM factory should still produce 20.00k SPM, losing or gaining less than 1% is not of great concern if I can get everything else good enough. But it should be as close as reasonably possible for number of items consumed and produced. The speed of production depends on power (might be less precise) and isn't as critical to get close to perfect imo, as long as it is within a few %. If things run 3% faster or slower, who cares if it's almost count perfect? The inputs and outputs might be constrained by some other non-abstracted factory anyways so in that case the abstraction could run it 100 times faster and you would just get "spikier" production but not actually any faster production. Sources of inaccuracy would be productivity modules. 2 things in might lead to 2 things out, but 3 things in could give 4 things out. The productivity bars could be read or the output calculated, but then you are talking about possible doing abstracted output higher than what you have measured so it just seems like a lot of work that generates more work for minor accuracy increase but at the risk of accuracy decrease if done with wrong assumptions and more CPU time by the mod. So not worth it I think, just measure for a bit longer for more accurate results.

The mod API doesn't give view position, but character position and player selected gives enough that it is almost achievable to know where the players are looking. So for that reason I'm fine with players seeing frozen machines if they use map view and accidentally/deliberately don't mouse over any entities.

Power draw might be tricky to get right, both to estimate, it's probably ok to approximate that with slightly less precision (so within a few %) instead of aiming to get it almost perfect like production stats. People overbuild on solar and nuclear on scales like 25% power excess or sometimes much more.

I think it's ok to only start measuring (abstraction preparation) when input buffers are filled and power is always at 100%. Should be reasonable to maintain for a few minutes.

Measurement really only needs to check that power is consistent every frame. Inputs and outputs are buffered and only need to be checked sometimes. So measurement is almost free, the heavier calculations only happen once or twice for the system at start and end of measurement. Finding and maintaining areas to abstract is another part that could require a fair ammount of work. But this isn't time critical and can be spread out over many tick, over minutes or hours. Doing it spread out over more time just delays the time it takes for a completed factory unit to be abstracted for the first time after being completed/modified and then up and running. For big factories the cell units that have not been modified by the player in the last 10 minutes is probably 90% or more, typically.

I'm not planning on making it completly impossible to "abuse" into behaving differently when abstracted from what it otherwise would do. It aims to do "the same thing as previous minute, repeated over the next hundred minutes". If a factory cell produces one thing at odd hours and other things at even hours then it should be accurate in it's production count, it just wouldn't switch to the other behaviour if there's circuits that route things differently certain times, or factory cells with circuits wouldn't be abstracted maybe.

Buffered items should be counted or assembler types should have crafts counted. Or buffered items should be detected and not be part of the measured and cached output for repetition. So you don't get free outputs because buffers were emptying while measuring. But otherwise it doesn't need to be impossible to abuse with all mod configurations etc. It just shouldn't accidentally give the wrong output when abstracted. If people want to abuse or trick the abstraction then they might as well just install Creative Mod or use infinity chests so it's not like real cheats aren't available if my mod isn't perfect...

The parts that will have items drawn from and outputted to will have to be chests. There items can be created or destroyed faster by Lua than Factorio by itself since items are not inserted individually but filled/drained completely instead with an API call, and big chains of interactions of entities can be abstracted away.

Ores on ground and labs/research also need to be handled. Especially ore deposits since most of material flow happens with mining ores and then smelting so it would be a great boost if even just that was sped up.
Difficulties
Most of the hard things to do will be to maintain data structures over a dynamic factory, find input outputs, not spend too much time on checking if things interfere with earlier abstractions, finding connections, maintaining hierachies and overlapping abstractions, simulating power draw etc, splitting work over several ticks efficiently in Lua...

But a more limited form that skips cells with logistics chests in it and doesn't pay as much attention to accuracy of the speed, power draw and splits the world into simpler disjointed cells etc might be possible to prototype up a bit quicker and then I'll see if the project is worth improving up to full spec.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4361
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

Qon wrote:
Tue Oct 13, 2020 11:54 am
Features and requirements spec:
I absolutely need active = false to mean that the entities don't use CPU cycles. This makes it "simple" and fast to enable/disable abstraction quickly. And belts and bots etc will stay in position carrying material as if it was always running.
Belts don't care about "active".
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

That's not great. They can be paused with circuits but not by mods?
So do belts require CPU cycles if some items are on them but nothing is taken on or off them? (Inserters active false).

I'm don't care about circular belts, so the items will be stationary at the ends after a short while. It would look kind of bad. Maybe i need to add circuits to shut the belts off? If that makes them inactive.

User avatar
Klonan
Factorio Staff
Factorio Staff
Posts: 4518
Joined: Sun Jan 11, 2015 2:09 pm
Contact:

Re: Optimization idea: abstraction

Post by Klonan »

Qon wrote:
Wed Oct 14, 2020 7:58 am
That's not great. They can be paused with circuits but not by mods?
So do belts require CPU cycles if some items are on them but nothing is taken on or off them? (Inserters active false).

I'm don't care about circular belts, so the items will be stationary at the ends after a short while. It would look kind of bad. Maybe i need to add circuits to shut the belts off? If that makes them inactive.
If a belt is 100% full, or 100% empty, all the belts in that transport line will sleep.
Don't use circuit network in attempt to optimise things, circuit network typically forces things to stay awake checking each tick if the circuit condition is met

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

Klonan wrote:
Wed Oct 14, 2020 8:00 am
If a belt is 100% full, or 100% empty, all the belts in that transport line will sleep.
That's good at least.
Klonan wrote:
Wed Oct 14, 2020 8:00 am
Don't use circuit network in attempt to optimise things, circuit network typically forces things to stay awake checking each tick if the circuit condition is met
Does this mean I can't active = false things that are connected to circuit network? Or can I both connect belts to circuit network and then active=false'ing them, making them both stop and use no CPU? Though I was told belts are immune to active=false and I expect this to also be true then, which is a shame. But asking anyways.

Also, can things become active=true if I set them to active=false (assuming no other mod messes with them) or is it permanent? I'm not sure if active state is the same as the sleep state or if active=false forces sleep but isn't the same thing.
Last edited by Qon on Wed Oct 14, 2020 5:10 pm, edited 1 time in total.

Rseding91
Factorio Staff
Factorio Staff
Posts: 11354
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Optimization idea: abstraction

Post by Rseding91 »

Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Also setting active=false sticks until something sets active=true through script.
If you want to get ahold of me I'm almost always on Discord.

yagaodirac
Long Handed Inserter
Long Handed Inserter
Posts: 65
Joined: Sun Jun 16, 2019 4:04 pm
Contact:

Re: Optimization idea: abstraction

Post by yagaodirac »

A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.
The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.
In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

Rseding91 wrote:
Wed Oct 14, 2020 7:28 pm
Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.

Also setting active=false sticks until something sets active=true through script.
Ok, good to know, thanks.

Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?

Is there a non-constant cost to modifying active state? If the entities need to be removed from an array of active_entities and it ends up being an O(n), with n being entities in the game, operation to toggle parts of the factory on and off would have to be rate limited to not end up hurting UPS.
Last edited by Qon on Thu Oct 15, 2020 11:48 am, edited 1 time in total.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.
I'm not interested in "cheat" machines. I want everything to be vanilla (and it should be entity agnostic and work with modded assemblers too for the most part), and gameplay should not be altered to the furthest extent reasonably possible. Ideallay I want it to be just installation and then your UPS is just no longer affected by the entity count and you should be able to keep playing as if your computer was infinitly fast and with no affect on gameplay. Vanilla gameplay is amazing already. I just want everyone, even those with potato computers with over the top megabase aspirations, to be able to not be limited by the realities of actual iteration. I want it to be more like Clusterio than an infinity chest, everything should still be managed the same. Actually better than Clusterio because it doesn't include teleportation which alters gobal logistics planning. There might be some ways this perfect mod idea doesn't quite become the utopia I envision, but I'm not interested in using a mod with your suggestion. The point of making a base 100x more productive than any other base built so far should be that you engineer the solutions to that kind of base and don't just replace all the entities with 100x throughput buildings. I want it as close to kovarex vision of "a lot of things" moving, not just big numbers. This is about taking shortcuts in the computation of the game, not shortcuts in the factory.
yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.
I'm not doing it in Factorio chunks.
Also first you say it's impossible, but if I can figure out how to do it then it is possible? :roll:
yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.
Actually scaling up production by 10x brings it's own set of new challenges, it doesn't need to be done "differently" by a mod that changes things. Megabases are not built like 100 starter bases, so it is not just "pushing numbers". It sounds like you haven't made any really big megabase in vanilla Factorio in quite a while.

yagaodirac
Long Handed Inserter
Long Handed Inserter
Posts: 65
Joined: Sun Jun 16, 2019 4:04 pm
Contact:

Re: Optimization idea: abstraction

Post by yagaodirac »

Qon wrote:
Thu Oct 15, 2020 11:42 am
yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
A solution is there and very simple. For functional areas like smelting, or basic products, you could mod some new, big furnaces and assemblers. Let's say, a long furnace. The recipe to get it is with gradient of 24 stone furnaces. And it can be crafted back to 24 separated furnaces. It not only saves the time you build it, it also saves cpu.
I don't know if you accept something like big assemblers. Or some polar panel of different tiers. Or simply mod some faster machine, tier 4 assemlbers, tier 2 electron miners.
I'm not interested in "cheat" machines. I want everything to be vanilla (and it should be entity agnostic and work with modded assemblers too for the most part), and gameplay should not be altered to the furthest extent reasonably possible. Ideallay I want it to be just installation and then your UPS is just no longer affected by the entity count and you should be able to keep playing as if your computer was infinitly fast and with no affect on gameplay. Vanilla gameplay is amazing already. I just want everyone, even those with potato computers with over the top megabase aspirations, to be able to not be limited by the realities of actual iteration. I want it to be more like Clusterio than an infinity chest, everything should still be managed the same. Actually better than Clusterio because it doesn't include teleportation which alters gobal logistics planning. There might be some ways this perfect mod idea doesn't quite become the utopia I envision, but I'm not interested in using a mod with your suggestion. The point of making a base 100x more productive than any other base built so far should be that you engineer the solutions to that kind of base and don't just replace all the entities with 100x throughput buildings. I want it as close to kovarex vision of "a lot of things" moving, not just big numbers. This is about taking shortcuts in the computation of the game, not shortcuts in the factory.
yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
The original idea of freezing a chunk, and caring only about the input and output in total is not possible. We talked about this 2 years ago in a Chinese social media software. The conclusion is about the parsing. If you can figure out the algo to parse the chunk and calc the property of it, then it's possible. My suggestion of modding some standard pseudo chunk prototype is an implement of this parsing.
I'm not doing it in Factorio chunks.
Also first you say it's impossible, but if I can figure out how to do it then it is possible? :roll:
yagaodirac wrote:
Thu Oct 15, 2020 8:53 am
In fact, you shouldn't push the count of entities to infinity only because the game doesn't limit with some mechanism. You all know cpu has its limitation. You should try something funny other than literally pushing the number to extremely big.
Actually scaling up production by 10x brings it's own set of new challenges, it doesn't need to be done "differently" by a mod that changes things. Megabases are not built like 100 starter bases, so it is not just "pushing numbers". It sounds like you haven't made any really big megabase in vanilla Factorio in quite a while.
1, I didn't mean a 100x assembler. If the recipe is 100x assembler into a big one, then the big one is 300x3 in size, and 100x in speed. When it's built on ground, it should look like 100 separated original things in a line. It's not cheating at all. It's cpu friendly only. Nothing logical changes. You literally misunderstood me.
2, Good luck figuring out the parsing algorithm.
3, You shouldn't talk about this idea here. You know amd and intel? Ask them for the solution. But in fact, I just started knowing that, lua can call functions written in cpp. If you mod a interface in lua, take the advantage of cpp and graphics card, with some specially designed algo, I think it might have a chance to do what you want.
4, The most practical way. Mod to remake everything, remove all the useless things. Simplify all the images. If you read through the source code of __base__, you would find a lot things that waste cpu time. If you get rid of all the decorations, all the transition sprites, at least you save a lot cpu time which is on the graphics task in vanilla. This is not cheating at all, it only make your game ugly. I know you wouldn't mind this.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4361
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

There was once a mod that implemented "furnace arrays" (can't find it on the portal right now). Basically the idea is to make a fake entity that looks exactly like n furaces in a row, and is n times as fast. That approach has it's own set of problems though: It only works for rows (or whatever pattern you decide to optimize), it's not hideable from the player because mass-entity replacement is expensive, and the "array"-entity has no way to enforce "normal" base layout, the player can just input/output with a bunch of loaders on one end. Also beacon effects get totally messed up due to the sheer size of the entity.


@Qon: What i was wondering is: How do you intend to create items with the correct rate when the machines are off? Spawning stuff via lua is too expensive and infinity chests have very limited rate control (only full integers each tick).
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

Not exactly sure what 1 - 4 are responses to, but I'll try to answer anyways.
yagaodirac wrote:
Thu Oct 15, 2020 2:14 pm
1, I didn't mean a 100x assembler. If the recipe is 100x assembler into a big one, then the big one is 300x3 in size, and 100x in speed. When it's built on ground, it should look like 100 separated original things in a line. It's not cheating at all. It's cpu friendly only. Nothing logical changes. You literally misunderstood me.
I took into acount that you might have meant a faster and larger assembler. But it doesn't change things that much. Larger assemblers prevents you from using your own blueprint layouts, ie it limits your engineering and kind of forces you into using someone elses layout/blueprint. It also forces you to rework your factory to use these kinds of entities and you can't enable or disable the mod on the fly.

And to supply the 300x3 assembler with items you need a lot of inserters (which are more costly in terms of CPU than assemblers). Or you need some 300x1 inserter. Fed from some 300x1 logistics chest or some kind of super belt. And now the speedup becomes limited or you would have to incorporate it all into a single unit. But now you have completely abandoned actually making your design, you literally have a big box that just produces things super fast and all you need is the space for it. You don't need to think about how to incorporate roboports and radars into the design (which is needed when making exceptionally large structures to make them buildable from map, which is a part of the challenge of building at this scale).

And this only covers a single step, you are limited to a single recipe which limits the speedup. Unless you want to make an assembler that takes iron and copper plates and spits out green chips. Or ores and spits out science packs. But then you have removed the whole building and designing aspect of the game. This method removes the game from the game. I don't like it.
yagaodirac wrote:
Thu Oct 15, 2020 2:14 pm
2, Good luck figuring out the parsing algorithm.
Thanks, it's going to be non-trivial. But I believe I can do it with enough time and effort, with some slight imperfections allowed.
yagaodirac wrote:
Thu Oct 15, 2020 2:14 pm
3, You shouldn't talk about this idea here. You know amd and intel? Ask them for the solution. But in fact, I just started knowing that, lua can call functions written in cpp. If you mod a interface in lua, take the advantage of cpp and graphics card, with some specially designed algo, I think it might have a chance to do what you want.
Not sure what I shouldn't talk about here... maybe you can say what you are talking about instead of saying "this" and quoting my entire post!?
Also you can't call C++ librararies from Lua code executed by Factorio. That would break the security model and the determinism guarantees and everything else.
yagaodirac wrote:
Thu Oct 15, 2020 2:14 pm
4, The most practical way. Mod to remake everything, remove all the useless things. Simplify all the images. If you read through the source code of __base__, you would find a lot things that waste cpu time. If you get rid of all the decorations, all the transition sprites, at least you save a lot cpu time which is on the graphics task in vanilla. This is not cheating at all, it only make your game ugly. I know you wouldn't mind this.
There are mods that reduce the graphics. But Factorio is RAM bandwidth limited and CPU limited. The graphics are to some extent handled by the CPU, but reducing animations isn't going to give you 200k SPM. Basically only things on screen this tick are affected by these kinds of changes, and most of it is handled by the GPU anyways. And what is on screen is nothing compared to the 99% of the factory that is outside your screen, which is what I want to abstract away.

Also I prefer the game to look good if possible.

Your suggestions don't provide nearly enough benefits compared to all the compromises they introduce and compared to the much, much faster abstractions I'm thinking of implementing. If I can just do it. If I can active=false all entities and just do a handful of API calls to remove a few tiles of ores and just change the research progress bar by a calculated amount once per minute I can achieve about 0.0% CPU utilisation for an entire gigabase, including the mod. And it will all be dynamic with no interaction by the player and "no" restrictions.

Listen, I'm not talking about speeding things up, I'm talking about eliminating computations. Like almost all of them. Making things 100 times faster is not enough, I'm going for something closer to constant time compution for any factory size and for any time step. The ideal is to be able to set game.speed = 100 and have it run at 6k UPS with a 1M SPM factory. That is going to run into many practical issues which means that might not be possible. But I want the closest thing to that and which is still possible. Without altering the factory building process. If I can calculate that a mining outpost will last for 100 hours at the rate I'm mining from it (total/mining rate) and that it produces X science/minute, then why not just use that calculation instead of simulating all the steps? As long as the player playing the game can't really notice any difference then that should be good enough.

Nidan
Inserter
Inserter
Posts: 39
Joined: Sat Nov 21, 2015 1:40 am
Contact:

Re: Optimization idea: abstraction

Post by Nidan »

Rseding91 wrote:
Wed Oct 14, 2020 7:28 pm
Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.
Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

eradicator wrote:
Thu Oct 15, 2020 3:06 pm
@Qon: What i was wondering is: How do you intend to create items with the correct rate when the machines are off? Spawning stuff via lua is too expensive and infinity chests have very limited rate control (only full integers each tick).
I will only be able to work with buffers, inserting individual items is too expensive. That means a single Lua call/chest once every X minutes and fill it completely with that single call, X being minutes it would otherwise take to fill it. Filling a chest with a call should make up for all the overhead of Lua compared to all the entities required to fill the chest normally.

Also I need to be able to abstract whole outposts or factories with several steps, skipping intermediate buffers completely. I want ot measure from as early as possible (potentially ore on ground) to as late as possible (research progress bar) and spawn as little items as possible. Most things should preferable only be created as a number in the stats window and would never actually be realised as an item in the world.

This means some factories will not really be abstractable, at least not if the player is within them and they are constructed to break all my assumpions. Like as long as they contain buffers that fill up without any local consumers (like a mall that hasn't been filled up completely). And you would need buffers in your factory and reasonably sized outpost blocks to be able to abstract partial factories efficiently.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4361
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

Qon wrote:
Thu Oct 15, 2020 3:28 pm
I will only be able to work with buffers, inserting individual items is too expensive.
Ah, so it'll only work for "natural" buffers and you're not going to artificially insert any. Interesting approach. As far as i remember the input/output slots in assemblers ignore the maximum stack size. That might or might not be better than size-limited chests.
Qon wrote:
Thu Oct 15, 2020 3:28 pm
Most things should preferable only be created as a number in the stats window and would never actually be realised as an item in the world.
As far as i can see production statistics are not moddable.
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

Rseding91
Factorio Staff
Factorio Staff
Posts: 11354
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Optimization idea: abstraction

Post by Rseding91 »

Qon wrote:
Thu Oct 15, 2020 11:21 am
Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?

Inserters have such a high update cost that adding the circuit overhead and stopping the inserter still comes out ahead. But yes the inserter control behavior never sleeps. None of the active control behaviors sleep because they have nothing to run events off of. The circuit network is constantly changing each tick as all of the inputs get summed together and sent out over the wire(s).
Qon wrote:
Thu Oct 15, 2020 11:21 am
Is there a non-constant cost to modifying active state? If the entities need to be removed from an array of active_entities and it ends up being an O(n), with n being entities in the game, operation to toggle parts of the factory on and off would have to be rate limited to not end up hurting UPS.
Setting active=false/true is a constant time operation.
If you want to get ahold of me I'm almost always on Discord.

Rseding91
Factorio Staff
Factorio Staff
Posts: 11354
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Optimization idea: abstraction

Post by Rseding91 »

Nidan wrote:
Thu Oct 15, 2020 3:23 pm
Rseding91 wrote:
Wed Oct 14, 2020 7:28 pm
Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.
Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.
Event-based is inventory: on-item-added: send event, on-item-removed: send event. Circuit logic is polled: each tick go over each thing that has signals to send and add them together. Then that's the state of the circuit network for that tick. Next go over each thing that wants to read it and have them do their logic.

There's no event for fluids flowing into a storage tank that's connected to a circuit network. There's no event for any of the circuit network; it's all sum-all, iterate users and calculate.
If you want to get ahold of me I'm almost always on Discord.

Qon
Smart Inserter
Smart Inserter
Posts: 1523
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

eradicator wrote:
Thu Oct 15, 2020 5:19 pm
Qon wrote:
Thu Oct 15, 2020 3:28 pm
I will only be able to work with buffers, inserting individual items is too expensive.
Ah, so it'll only work for "natural" buffers and you're not going to artificially insert any. Interesting approach. As far as i remember the input/output slots in assemblers ignore the maximum stack size. That might or might not be better than size-limited chests.
It's worse. And probably not for the reason you think. It doesn't matter how fast it is to modify assemblers. The assemblers still need the output extracted from them. That means I can't abstract away the inserters and the belts and the inserters again to the train station chests. This could work for production outposts with assembler to train direct insertion, but those are almost never built so it seems like a waste of effort.
I don't want to abstract away assemblers. I want to abstract away everything. Except combat and building and some things that are not feasible.
eradicator wrote:
Thu Oct 15, 2020 5:19 pm
As far as i can see production statistics are not moddable.

Code: Select all

game.player.force.item_production_statistics.on_flow('automation-science-pack', 10000)

Post Reply

Return to “Ideas and Suggestions”

Who is online

Users browsing this forum: No registered users