Friday Facts #421 - Optimizations 2.0

[raven2k3]

I am not sure if devs are reading the forums, I am very rarely posting something here.

Regarding robo ports I would love to finally see something like selectable robo port channels.

This would make things a lot easier if you want to build a sub network.
I usually use custom made city blocks where I automatically build decent robo network.
It was always a pain that you have to remove the roboports to split a network, because at the borders you will lose the coverage and
therefore robots can't reach some areas anymore..

[Koub]

I am not sure if devs are reading the forums, I am very rarely posting something here.

They do.
But the answer is not always "Yes" .

[raven2k3]

They do.
But the answer is not always "Yes" .

That's okay, I was just wondering

[zggz]

Here it is! (beep boop)

https://factorio.com/blog/post/fff-421

all time when i read about optimisation i hope to see multicore...

[cappie]

Are you using hardware acceleration for the bot movements.. asin.. vertex shaders for the positional drawing of the sprites?

[VampireSilence]

This is absolutely great! I love it!

The "keep revelead" counter made me think about something. What if there would be a building decreasing it? That means we could have a kind of "jammers" that prevent revealing certain areas of the map from enemy factions. :>
Every faction would have to have its own counter for this to work ofc. But it would be possible.

[dohnpa]

I am honestly surprised you guys are iterating over everything to compute the electricity. I know very little of Factorio inner workings and I am quite definitely missing something, but anyway...

When I looked at the fff-209 I saw that you are splitting entities per prototype to make the structures smaller for cache. Do you really need those for each prototype? Don't you just need a sum of their power demands? Those do not change very often in some cases, do they?
Radars, beacons etc has constant power draw.
Drills have either full output or not. Which does not change most of the ticks.
Crafting machines are often continually crafting and crafting an item usually takes multiple frames.

So the way I would expect it could work would be something like:
- Exclude entities that are powered by multiple networks. We will handle them later.
- Electric network only holds variable for total demand.
- Each entity "registers" to one electric network. The "registration" is just a change in the total demand value.
- Every time, when in Update the entity changes it's power demand, it updates the total demand in electric network.
- Entities powered by multiple networks are resolved iteratively like they are resolved now
- Electric networks looks at the power input, power demand and calculate power satisfaction
- Power satisfaction is then a public number that all the entities can read in next update
- Again, satisfaction for entities powered by multiple networks will be resolved iteratively like they were before.

I do not know if you had individual satisfaction for each entity separately, but if you did, then I am not sure if this is really worth it. At the scale of the factories, small rounding errors in memory consumption probably do not matter all that much.
For some entities (ie inserters) this approach probably does not make much sense, but for most of the others...
You could have two types of registration to the network. The fast one and the iterative one. It would make the lists of entities much shorter. The "iterative registration" will be for some prototypes with every frame changing entities and for entities powered by multiple networks. The fast one will be used for the rest.

Again, I am pretty sure there is a reason why this will not work, but I could not help myself .

[OnisDWA]

Regarding the optimization of the electric network
My assumptions may be wrong, but I think they may be partially useful.
I'm not sure, but I think that machine processing is already paralleled, then you can do the following
I think that in general it is possible to abandon a separate pipeline for processing the electrical network, thus:

In the network object, it will be enough for us to store
- current consumption (just for display) (ECUR)
- current consumption of the prev cycle (ECUR_PREV)
- expected consumption (maximum consumption on past cycles) (EEXP)
- expected consumption of the prev cycle (EEXP_PREV)
- maximum possible consumption (EMAX)
- current performance (just for display)
- maximum possible performance (EPRODUCE)

Build aggregates when networks change (when machines are added/removed)
1. Calculation of the maximum possible electricity generation
2. Each machine in the network will have a link to each object of the network in which it consists (in priority for the use of renewable sources)
3. Together with p.2, the machine will register its energy drain as a basic consumption, in the highest priority network (if there is available energy in it)
4. Together with p.2, the machine will register its maximum possible consumption into the network, taking into account all effects (EP)

Сalculate the energy consumption in the pipeline of the machines themselves. Thus, we will remove a separate pipeline for calculating the network, and we will be able to calculate it together with the change in the state of the machine: namely, when they are turned on/ off

I believe that if the electricity in the network is not produced by steam engines, then it does not matter what the expected consumption (EEXP) is, if it is below the maximum possible performance (EPRODUCE)
In this case, I believe that the calculation of energy consumption can generally be skipped and equal to its maxima (EMAX)
For such a radical optimization, you can switch to some console command or when the UPS ceiling has been reached, usually at those volumes you no longer care what consumption is there, because the production margin is almost always huge
Why can this be done if we do not have steam engines: because the consumption of resources for reactor operation does not correlate with the load (except for water consumption, but players do not carry it by trains, I hope)

How do I see consumption miscalculation:
The priority of where to get electricity will be set based on p.1
When an event occurs that the machine should become active, it should add its consumption (EP) and expected consumption (EEXP) to the network
When the machine finishes its work, it must subtract its consumption (EP) and expected consumption (EEXP) from the network
If the car does not have enough electricity from the first network, then it should try to take it from the next one. To do this, you will have to remember where and how much electricity was taken from

A machine can take 100% of its consumption (EP) if EMAX < EPRODUCE or EEXP_PREV < EPRODUCE, otherwise in order for all machines to receive the same amount of energy, then we need to understand how much it can potentially be used, for this we need to remember the consumption for N past cycles or its average (EEXP_PREV) then we can calculate how much energy (EP) we can take from the network like this: EP * ((EPRODUCE - ECUR_PREV ) / (EEXP_PREV - ECUR_PREV)
Example: ECUR_PREV=1, EEXP_PREV=2, EPRODUCE=1.5, and 5 machines should add an EP by 0.2
0.2 * ((1.5-1) / (2 -1)) = 0.1, which is exactly 50%
And so on, each machine will add its consumption (EP) to the network (ECUR)
But still there is a problem, because it is not clear how many more machines are connected, then not all machines can get their consumption (EP), so for 1 cycle some may be completely without energy, but already on the second cycle they will receive +- the same % as all the others

As I see the production calculation:
Electricity generation should be calculated after calculating the consumption of all machines
The main thing is to figure out how to make the potential calculation as accurate as possible
At this stage, you can equate ECUR_PREV = ECUR and EEXP_PREV = EEXP
If it so happened that the maximum production did not coincide with the real one (and I think this will happen often), then you can determine the error
If the error was not large, then in general you can do nothing and leave the recalculation for subsequent cycles,
otherwise if the discrepancy was large, then start the second cycle of recalculation of all machines (which I consider unnecessary)

I think this would allow us to maintain multithreading (provided that the machines themselves are processed asynchronously) and would avoid memory problems (because there would be network objects in the cache all the time, and we bypass all the machines at the moment, and not several times and only to calculate electricity), as well as calculate the entire electrical the network in 1 pass, while in the same pipeline where the machines will be processed, without unnecessary requests to memory, as well as save performance and do not touch those parts of the electrical network if no objects have been activated (you can use the cache)

[Big_Snarly]

Okay, I can only hope this gets looked at...

I love that the lamps can have auto switch on their own now, and they also don't need an outside influence to change their color, right? The RGB sliders can be inherent without a network connection?

It would be great if we could have swathes of lamps without the need for electrical poles. Too bad we couldn't have an interconnect Lamp version that powers neighboring lamps, or a tech research to enable lamps to have inherent power connectivity to each other.

Yes, it's purely for aesthetic. I'd like to design more pictures with lamps and make some fun displays. It's one of the most versatile ways I can customize my layouts.

And since there's an introduction to bio fuel and manipulating bio matter, is there any chance we can research something agriculturally to grow plants and trees? (Yes, I'm hoping there's other things I might be able to decorate with. I'm still hoping for more tiles. The 5 I can make are good, but I wish I could use the colored concrete or more)

[Panzerknacker]

I really hope that because of those and additional optimizations, we can keep the simulated fluid system.

[Koub]

I really hope that because of those and additional optimizations, we can keep the simulated fluid system.

No. And please stop bringing back the fluids 2.0 change in every single thread. We have understood you don't like it, but you don't have to pollute the whole forum with that.

[OhYeahItsAshley]

Not one to know much about optimisation and that kind of stuff, but Intel VTune was mentioned specifially, so I assume Wube's rigs are Intel-based, is there any way to know if these optimisations will have the same effect on AMD processors? I personally run AMD so I'm curious as to wether I'd see the same 10-25% performance boost or it would be less on a different architecture

[raven2k3]

Tbh I don't know how the logic works of computing the electricity works but I suggested to somehow
force the player to split the energy network, e.g. via needed transformers or big electric fuses (like https://mods.factorio.com/mod/PowerOverload).

The idea would be that If you have then more than one electricity network, each network can be handeld by one separate thread / core independently.
Devs really did a great job in optimizing the logic but more multi threading would be really nice, if possible..

[BrainGamer_]

The idea would be that If you have then more than one electricity network, each network can be handeld by one separate thread / core independently.
Devs really did a great job in optimizing the logic but more multi threading would be really nice, if possible..

read the FFF again. They did manage to make the electric network update multithreaded. The problem is not a lack of processing power but memory throughput. Multithreading does not improve anything here at all which is why it was dropped again.

[GregoriusT]

I have no idea about the inner workings of the Energy Network, but I am wondering if there would be some possible optimization angle where all machines whose buffers are full can be optimized out of the equation or something, or merging things like accumulators or solars into one bulk entity so only one memory value needs to be changed (as long as they have identical charge levels and such). But i bet the Devs already thought of this angle and did something about it, just me thinking of ideas that probably dont matter. ^^

[BrainGamer_]

merging things like accumulators or solars into one bulk entity so only one memory value needs to be changed (as long as they have identical charge levels and such)

this is in fact already a thing which is how solar power is so UPS efficient

The solar panels and accumulators are now merged in the electric network, so they don't have to be simulated individually. The electric network distribution calculations are also heavily simplified by this. That basically means, that the amount of solar/accumulators have no effect on the speed of the factory update times any more.

[BattleFluffy]

Yes, this is what I wait for, the optimizations! gogo devs!! :D

[maniak1349]

Not one to know much about optimisation and that kind of stuff, but Intel VTune was mentioned specifially, so I assume Wube's rigs are Intel-based, is there any way to know if these optimisations will have the same effect on AMD processors? I personally run AMD so I'm curious as to wether I'd see the same 10-25% performance boost or it would be less on a different architecture

Intel VTune is just a profiler - it tells you which parts of code run fast and which run slow. Optimizations are done in generic C++ code and for desktop AMD and Intel CPUs binaries are built for generic x86_64 architecture which they share. So - yes, you should see the same 10-25% performance boost on AMD. You should see similar increase in performance on ARM too, also numbers may be somewhat different there.

[boskid]

Usage of Intel VTune means that just my personal PC is known to be intel based, other machines can be anything. Its just a profiler but one that gives a really nice explanation about why code was slow to execute.

[atomicstew]

One of the best FFF so far! keep up the good work!
kovarex you are my hero