Optimization idea: abstraction

Post your ideas and suggestions how to improve the game.
Qon
Smart Inserter
Smart Inserter
Posts: 1521
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

Thanks for the info. Some more questions ;)
Rseding91 wrote:
Thu Oct 15, 2020 6:40 pm
Qon wrote:
Thu Oct 15, 2020 11:21 am
Does that mean that those that use clocked inserters to force sleep to improve UPS in megabase smelting arrays and similar are not really getting anything out of it? I'm guessing inserters are event based though and can be made more effiencient like this, correct?
Inserters have such a high update cost that adding the circuit overhead and stopping the inserter still comes out ahead. But yes the inserter control behavior never sleeps. None of the active control behaviors sleep because they have nothing to run events off of. The circuit network is constantly changing each tick as all of the inputs get summed together and sent out over the wire(s).
Inserter control behaviour never sleeps? Even when active=false?
Which entities are not affected by (and/or don't stop requiring CPU cycles if that is something separate) being active=false except belts, underneathies and splitters? Inserters and circuits and wired things also?
Is there a complete list of things that (a) continue to work while active=false and a complete list of things that (b) might stilll use CPU while active=false?

And how does the debug option "show-active-state" relate to this?
The red ring is on inserters that are disabled with a circuit wire (regardless of active value)
and those that can't insert because target is full while (regardless of active value)
for me and on inserters that are stopped mid-swing by being active=false.
So does this mean the red ring doesn't signify that things don't use CPU cycles, since from what I understand now some of those things with red rings still use CPU according to what you described?
Tested with

Code: Select all

/c local ents = game.player.surface.find_entities_filtered{position = game.player.position, radius = 100} for k, v in pairs(ents) do v.active = false end

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4360
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

Qon wrote:
Thu Oct 15, 2020 8:25 pm
It's worse. And probably not for the reason you think.
[...]
I don't want to abstract away assemblers. I want to abstract away everything.
I understand that just fine. But you also said that you want to keep the visible part running - why else would you need to even consider filling chests at all. But belt based factories usually don't have any chests you can abuse. Anyway it was just a bit of info. You're the only one who knows how you want to get this done. I certainly have no idea how one would make parts of a long belt seem like they're moving when everything around them is actually frozen. And train-based city-blocks is gonna be pretty obviously ugly on the map view :D.
Qon wrote:
Thu Oct 15, 2020 8:25 pm
eradicator wrote:
Thu Oct 15, 2020 5:19 pm
As far as i can see production statistics are not moddable.

Code: Select all

game.player.force.item_production_statistics.on_flow('automation-science-pack', 10000)
Sorry, I was fooled by the read-only tag.

Code: Select all

item_production_statistics :: LuaFlowStatistics [Read-only]
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

Qon
Smart Inserter
Smart Inserter
Posts: 1521
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

I will simply not freeze things in view. And anything that is connected to the thing in view in a way that isn't separated by a buffer will unfortunately also have to be running. That is why I needed active state to be quickly toggleable, or at least one reason.

Belt based factories with no buffers at all are a clear weakness of this method. There might be a need to alter the design slighty in that case to take advantage of the speedups. You would only have to do a handful of "cuts"/buffers to allow the mod to abstract away things out of view though. Maybe interacting with assembler buffers would be an option then to avoid those issues. If its possible to abstract away all mining and smelting or similar impact then that might be worth considering.

Outposts by train would look the same from map view when zoomed out. The trains will still run (if the (complete?) Rail network isn't abstracted. When zoomed in to world I would have to use selected entity position instead of player position to decide which parts to run.

Rseding91
Factorio Staff
Factorio Staff
Posts: 11352
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Optimization idea: abstraction

Post by Rseding91 »

When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:

Entities
Belts
Pipes
Electric flow (producer -> consumer)
Control behavior
Circuit Networks
Heat flow
Trains
If you want to get ahold of me I'm almost always on Discord.

Qon
Smart Inserter
Smart Inserter
Posts: 1521
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

Rseding91 wrote:
Thu Oct 15, 2020 10:31 pm
When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:
Does that mean that active=false only affects the entity loop and every other loop is unaffected?
Are inactive entities affecting the CPU for power calculations? They still have power drain in my test. But regardless I'm imagining that if all solar panels can be virtually combined as one entity then the same should be possible for all entities with drain (min consumption) also?

Pipes are not affected by .active it seems. Do full/empty pipe systems also constantly use UPS? If the sources and drains are all inactive then that should mean that no update will affect the state of fluid levels anyway.

Setting active=false on all my entities improved UPS, but destroying all entities on the surface improved UPS more. Trains were not affected.
If trains are stationary at a stop waiting for some condition (circuit/time/inactivity/inventory I guess are the main ones) to leave then will it use UPS when inventory isn't being updated? They have no real reason to do path updates when they are stopped, I hope.

Is there somewhere I can read up more on how the game engine handles things like these? Or is there some way to find it out for myself? show-active-state doesn't really tell me if things stop using the CPU since there's things like control behaviour (what is that?) that keeps draining my UPS needlessly.

I'm doing some experiments to see what can be stopped and if I need to actually .destroy() things to stop the engine from wasting cycles on things. I might need some more powerful pause entity functionality for the mod to actually speed up things properly.

Rseding91
Factorio Staff
Factorio Staff
Posts: 11352
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Optimization idea: abstraction

Post by Rseding91 »

Qon wrote:
Fri Oct 16, 2020 9:58 am
Rseding91 wrote:
Thu Oct 15, 2020 10:31 pm
When you set active=false on an inserter the *inserter* stops updating. The control behavior is separate and always updates.

There are several unrelated update loops:
Does that mean that active=false only affects the entity loop and every other loop is unaffected?
Yes
If you want to get ahold of me I'm almost always on Discord.

Nidan
Inserter
Inserter
Posts: 39
Joined: Sat Nov 21, 2015 1:40 am
Contact:

Re: Optimization idea: abstraction

Post by Nidan »

Rseding91 wrote:
Thu Oct 15, 2020 6:43 pm
Nidan wrote:
Thu Oct 15, 2020 3:23 pm
Rseding91 wrote:
Wed Oct 14, 2020 7:28 pm
Things with circuit logic that can't run event-based (most of them): the circuit logic never sleeps. Setting active=false doesn't do anything for it.
Can you elaborate why the circuit logic can't be sleep?
For all vanilla circuit entities, the result they compute can only change when their input changes. This sounds like a prime example for an event based system, so I wonder why it can't be.
Event-based is inventory: on-item-added: send event, on-item-removed: send event. Circuit logic is polled: each tick go over each thing that has signals to send and add them together. Then that's the state of the circuit network for that tick. Next go over each thing that wants to read it and have them do their logic.

There's no event for fluids flowing into a storage tank that's connected to a circuit network. There's no event for any of the circuit network; it's all sum-all, iterate users and calculate.
The fluid system not having events makes sense.
But if you were to store the signal the tank provided in the current tick and, in the next tick, compare the signal the tank would provide with the stored one, you could decouple both systems. The same could be done for other entities with a similar situation.
Given that, you'd have a method for detecting change and the decision whether to poll each tick or to use events becomes a CPU vs memory trade. You'd have to store the signals provided by each entity and (for display and faster updating) the sum of each circuit network, but save time when no input changes. In addition mods could make use of the event and wouldn't have to rely on on_tick.

Impatient
Filter Inserter
Filter Inserter
Posts: 634
Joined: Sun Mar 20, 2016 2:51 am
Contact:

Re: Optimization idea: abstraction

Post by Impatient »

StephenLynx wrote:
Sat Mar 07, 2020 1:08 am
...
But if this is possible, think about how much UPS it would save. How larger factories could be or how better lower end machines could run it. I dare to say it might be the biggest optimization for a modular factories this game have ever seen.


The basic idea you are bringing on the table there, is the idea of black boxes. This idea has been juggled around time and time again.

Now, IMO and as stated by many posts already, in regards of micro simulation, this can not be done in factorio with the same degree of accuracy. Currently the accuracy in factorio is 100%.

The next thing that comes to mind is, just to accept lower accuracy as trade off for more UPS. Why not? That is something that would be possible.

But, ... the question is IMO, how should factoio determine what the production rate or throughput of such a black box is/should be ( without it being possible to be exploited? ) That is a big thing and has yet to be answered.

Btw, some modders took a very simple approach and created new entities which have the same throughput and production stats as known blueprints ( eg a whole smelting line ). That is a great way to improve UPS, if you want something like that.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 4360
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Optimization idea: abstraction

Post by eradicator »

Impatient wrote:
Fri Oct 16, 2020 3:18 pm
Now, IMO and as stated by many posts already, in regards of micro simulation, this can not be done in factorio with the same degree of accuracy. Currently the accuracy in factorio is 100%.

The next thing that comes to mind is, just to accept lower accuracy as trade off for more UPS. Why not? That is something that would be possible.
Actually that is a point i've thought about with @Qon's idea and i have to admit that for me personally the accuracy might actually be quite important. There's so many games out there that give you only a vague idea of what is actually going on, and factorios precision clockwork is a very welcome change from that. So a mod that manipulates statistics and lies straight into my face about *everything* that's going on might not be as appaealing to me to play as it is technically intriguing to talk about.
Author of: Hand Crank Generator Deluxe, Screenshot Hotkey 2.0, /sudo
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11421
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

eradicator wrote:
Fri Oct 16, 2020 3:53 pm
Actually that is a point i've thought about with @Qon's idea and i have to admit that for me personally the accuracy might actually be quite important. There's so many games out there that give you only a vague idea of what is actually going on, and factorios precision clockwork is a very welcome change from that. So a mod that manipulates statistics and lies straight into my face about *everything* that's going on might not be as appaealing to me to play as it is technically intriguing to talk about.
I think that’s an important point.

And my automatic next question is:
How much can a simulation of Factorio “lie”, to be accepted?

To be more exact: how many percent exactness do we need, to say “this is an exact simulation of Factorio.”?

And to measure it I suggest a simple way: we measure how many items are made in reality (over very long time) and measure the items produced by the simulation. And the error of the simulation can be expressed in items per time.

E.g. an error of 1 item per hour means, that there was produced 1 item too much r too less in an hour for that part of the factory. And because “that part of factory” can be expressed as an area, that produces some items per hour we came to the unit of
errors per items

... and the question that follows is, how much errors per items is acceptable and what not? Some say zero, others don’t bother with one error per 10 produced items.

How much would be ok for you??
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

yagaodirac
Long Handed Inserter
Long Handed Inserter
Posts: 61
Joined: Sun Jun 16, 2019 4:04 pm
Contact:

Re: Optimization idea: abstraction

Post by yagaodirac »

I literally accept all the changes, all the levels of changes.
One thing I have to point out, within all the similar games I've played or at least watched the videos or streaming, Factorio is very special. The design of belts with 2 sides, inserters, loaders, and even the variation of shapes of train stations, are only be seen in this game. They are too complicated. If you accept new entity, replacing is a great implementation which looks like merging. If you receive some new mechanism of communication between nodes(furnaces, assemblers) and segments(belts), all the inserters, loaders, are gone. If you accept a pseudo power system or removing the whole power system from this game, the game could be simplified a lot. People can literally concentrate on crafting better.
But, you are not gonna accept any of them.
In fact I knew some studies on how to save cpu at least 1 year ago. But those dudes didn't do any actual experience. But even without clues, we still think the belts and drones harm ups the most. We even tried to carry every single item inside wagons, but nobody eventually made any experience on that. I even invented a railway net pattern to simplify and speed up the trains, but yet any experience is done about it. And very few people know how to put items from a wagon into another wagon with out red inserter or extra vihicle.
So, which would you prefer? Modding some half cheating entities? Modding some algo to lie? Reinventing the way you build up the super factory? Or even make scenario to separate things into multiple surfaces. I don't know if this make use of multi core of your cpu.

yagaodirac
Long Handed Inserter
Long Handed Inserter
Posts: 61
Joined: Sun Jun 16, 2019 4:04 pm
Contact:

Re: Optimization idea: abstraction

Post by yagaodirac »

OK, besides. Some of you are interesting in algorithm to directly simplify the computation in vanilla. But you need to know, if you parse a specific structure, like smelting zone, only the corresponding zones benefit. The more a zone is logically near to mining, the practical to do this.
If an assembler or furnace is built, no matter how, by player or by robot, you can parse the belt lane provides to it and the lane receive products from it. And if some other assembler carries out the same producing task, you could merge them logically. But if you don't involve the inserters and belts, it doesn't help very much imo. But if it's possible to parse a whole zone, 48furnaces,nearly 200 belts, 96inserters,3 splitter 2 underground belts, 18 poles, it's very signifisantly. I think this is what you truly need.
The disadvantage is that, this method only works well with some traditional and widely used structure and logically near to mining. Imo, it works only with mining, smelting and some very basic assembler zones like circuits, and the very final lab.
But as I know, people invented at least 5 smelting zone layout. And if people link 2 assemblers with inserter (the case when producing green and red circuit), it makes the parsing complex. And 2 zones are not possible to share any part. It means, you have to put corper cables on a belt, and send them to somewhere else to craft circuit. The issue harms the designing of green, red circuit, belt, inserter, red bullet, and some more.
Besides, ore zones are irregular, no easy way to keep track of all the tiles with some specific kind of resource. If you track them in lua, it's slow. I asked some container in cpp to help us handle data in lua. But the staff is not interesting in this. Obviously, very few mod would make use of this container.
So, I really can't help. Good luck.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 11421
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Optimization idea: abstraction

Post by ssilk »

@Yagaodirac parsing is not useful (to take up your words), the calculations you make there are just theoretical output. The true output can only be measured against fully working assemblies.

Hm, how to I explain this?

Let’s assume you can mark some part of your factory to be measured some time. And what we are searching for are the assemblies (furnace, refineries etc.).

Now we can have two situations:
A) not all assemblies work 100%. Some need to wait for input items, some cannot output items.
B) all assemblies work 100% all the time.

B is what we are looking for!! Because only then the real output is equal to the theoretical output and only then we can really simulate it, because we can calculate the exact throughput so, that the error per item is minimized (I think lower than one error than 100,000 items).

And to achieve that we need to build such areas a little bit differently than now.

Every assembly needs to work 100% of time, which means we need to make sure, that there is always enough items to be inputted and the output never get stuck.

This can get quite complicated, because we need also to take in account the buffers (in chests, on belts, in the assemblies, in the hands on the inserters). We can only ignore them, if we have another buffer that feeds the whole construction, which is much bigger, than the internal buffers. The same for the output.

So again the steps:
A) built an “area”, that can be simulated (separated). Which means: no circuits, no bots, small internal buffers, assemblies work always 100% and the only connection is power and some input- and output-buffers (chests?). There shouldn’t be also no big internal buffers, because that enables cheating.
B) before we can start to simulate that, we need to measure for a while, if this is really the case. We can calculate the turnaround time for all assemblies (length and types of belts + production time + inserter time) and double that. But at least input and output needs to have been stabilized, before we can stop measuring. Otherwise the possibilities to trick this out somehow is too high.
C) when that is stabilized, we can simulate it. Calculate the theoretical throughput. Pick items from input buffer and put items into output buffer. The area closes its roof, so we cannot see, what’s happening inside (and not need to simulate the inserters etc.)
D) if one condition changes (no power, not enough input item or output is full), stop simulation.

I surely have forgotten some things, but I say it is possible.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

yagaodirac
Long Handed Inserter
Long Handed Inserter
Posts: 61
Joined: Sun Jun 16, 2019 4:04 pm
Contact:

Re: Optimization idea: abstraction

Post by yagaodirac »

ssilk wrote:
Sat Oct 17, 2020 8:30 am
The @ symbol didn't show any notice to me. Anyway.
We share the same way. Except for some details, which don't count at all.
I think we should review the design of this game, rather than base on it and then try something epic but not very practical.
1, What is the purpose of scaling base. Does giant base give us fun? Personally I don't believe.
2, What is the purpose of this game? I don't really take this game as a game. I spend time studying its api, the same way I did when I learnt UE4 and Unity. For now, even the Mtn Fortress or Crash Site is not good enough. This game is not playable at all. It's more like 1/2 game engine + 1/2 pseudo physical experiment simulator. I believe we should spend some time figuring out what is funny enough and possible to base on the api.
3, Some questionable design. Why inserters? Why loaders? Why put poles on the same layer with other entities. Why 2 sides for the belts? Any alternative to the assembler mechanism? Better way to manage the belts?
4, I saw only one one-mod-scenario, which is named singistics. It shows a great idea, but not good enough. I tried to improve it, but didn't push it any further. Is vanilla mode holy enough and worth people paying that much attention?
I prefer to abstract every question before I try answering it.
The problem shouldn't be how to parse the "block", how to simulate it. The point is, what should a game do, how to design a mechanism, how to design the rule, and eventually, what is fun?

Qon
Smart Inserter
Smart Inserter
Posts: 1521
Joined: Thu Mar 17, 2016 6:27 am
Contact:

Re: Optimization idea: abstraction

Post by Qon »

yagaodirac wrote:
Sat Oct 24, 2020 1:26 pm
I think we should review the design of this game, rather than base on it and then try something epic but not very practical.
I already know it to be great. Why should we review it, in this context? If we are modding then if we think something sounds cool and worth implementing and do it then that's justification enough. If others like it then that also tells us a bit about if the idea is good or not.
yagaodirac wrote:
Sat Oct 24, 2020 1:26 pm
1, What is the purpose of scaling base. Does giant base give us fun? Personally I don't believe.
Yes, giant bases provide more fun. Every time you scale up production by an order of magnitude you find a new type of bottleneck that gives you a new type of play experience and new design problems to solve. Basically it's another "level" of play.

You might go from belts to trains because belts don't scale and become too cumbersome when going from 10 to 100 SPM. But when going from 100 to 1000 SPM you might need to redesign your rail network because you get weird congestion.

When going from 1k SPM to 10k SPM you might do it by building 10x 1k SPM factories. But now you need a modular 1k SPM design that can be replicated easily and can easily accept resources by rail. You will want to make the factory compact enough that the space becomes manageble and you might want to reduce trains for each 1k SPM factory to make it easier to keep track of and set up trains for each new factory.

At 100k SPM with 100x 1k SPM factories you will get tired of managing everything about clearing and claiming space, connecting mining outposts, traveling etc. If you didn't make the whole 1k SPM factory and all the other infrastructure needed to build that buildable from map view then you will have to redesign your blueprints again. You will also want to think about how to automate the logistics of shipping building materiel and how to manage outposts that only build more infrastructure like modules, rail and assemblers, because the robot flying speed for building things this far away is going to be really limiting your expansions and you need separate construction networks but need automated logistics of entities for building. At this stage you are not really optimising for SPM, but for SPM added per hour (or Science/(minute^2)), that is, how fast can you scale production? Because if you don't speedrun building and do it all from map view then you are easily get stuck at creating a megabase every 20 hours or so and you won't get there within thousands of hours.

For 1M SPM you have probably already hit the limit a while ago of keeping UPS at 60 from just building with hundreds of thousands of construction robots, even if the factories you have built require 0 processing time. But you need ConMan/Recursive Blueprints and completely programmatically expand your base to practically be able to reach something like this. Which is another fun challenge to do even if reality of physical hardware stops anything like this from actually reaching this stage.

Disclaimer: numbers are wildly approximated within a few orders of magnitude. CPU cost of running factories is assumed to be about 0.

posila
Factorio Staff
Factorio Staff
Posts: 4715
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: Optimization idea: abstraction

Post by posila »

Qon wrote:
Sat Oct 24, 2020 4:21 pm
yagaodirac wrote:
Sat Oct 24, 2020 1:26 pm
I think we should review the design of this game, rather than base on it and then try something epic but not very practical.
I already know it to be great. Why should we review it, in this context? If we are modding then if we think something sounds cool and worth implementing and do it then that's justification enough. If others like it then that also tells us a bit about if the idea is good or not.
yagaodirac is trying to design a new game around the abstraction optimization tech. Which is a proper way to design a game if you indend to push some optimization to extreme limits - don't do things that make the optimization too complex or impossible to implement.

But I think it's a bad idea attempting to do it as a mod for Factorio. You'd be attempting to circumvent the game simulation in a game that's built around everything (that is built by player) being simulated and accounted for a all times. Making a new game in UE4/Unity seems easier than trying to bend the game this way through its modding API.

Post Reply

Return to “Ideas and Suggestions”

Who is online

Users browsing this forum: ptx0