Friday Facts #421 - Optimizations 2.0

[Dev-iL]

For a couple of years now, I've fantasized about asking Factorio devs for NDA access to the engine source.

Theoretically not impossible. People are known to have been given acces to the source. Typically, said people were well-known modders and/or those who wanted to work on features that were low priority for the devs, but they found important enough.

[Galista]

I settled on a registration style system where anything that wants to reveal an area of the map simply registers the chunks as "keep revealed" by increasing a counter for that chunk in the map system. As long as the counter is larger than zero the chunk stays visible. Things can overlap as much as they want and it simply increases the counters.

I may be missing something here, but why even bother to count reveal requests? Aren't boolean operations faster?

[samcday]

For a couple of years now, I've fantasized about asking Factorio devs for NDA access to the engine source.

Theoretically not impossible. People are known to have been given acces to the source. Typically, said people were well-known modders and/or those who wanted to work on features that were low priority for the devs, but they found important enough.

Aye, I'm aware that notable non-Wube folks have/had access. That it could actually happen is what makes my little fantasy such quality daydream material Similarly, I like to daydream about winning the lottery (also theoretically possible, though I've never bought a lottery ticket myself....) more than having superpowers (because Compound V sadly/thankfully does not exist in our particular local timeline).

[samcday]

I settled on a registration style system where anything that wants to reveal an area of the map simply registers the chunks as "keep revealed" by increasing a counter for that chunk in the map system. As long as the counter is larger than zero the chunk stays visible. Things can overlap as much as they want and it simply increases the counters.

I may be missing something here, but why even bother to count reveal requests? Aren't boolean operations faster?

It needs to be a counter because otherwise how do you know when the last registration was cleaned up? That is, a boolean would be "lossy" here - you would be throwing away important information. Another way of thinking about this, is that what was described in the blog post is very much like a "semaphore", and you're wondering why not use a "mutex" instead.

[JKopter]

What if the logistic robot has a curved path (chasing a mobile entity like the player or spidertron)?

I was also concerned about this. Also if the location for the robot is removed entirely - it would need to choose a different direction, possibly even exact opposite. I'm thinking it would glitch and jump around.

Perhaps swapping the order would make things smoother? So the 'real' robot moves, and *then* the 'ghost robot' moves from where it was, to the new real robot's position.

[Ieuan]

Can't wait to try this out on my most recent (SE) save, my world has gotten rather big with lots of roboports and radars;

screenshots

I don't know if the optimizations to radars and robots would solve it 1:1, but the game is almost entirely unplayable now lol

[mmmPI]

I think many FFF's are about optimization, but i can understand there's so much of it that it needs to have its own FFF, to point at for when someone ask again "what about multithreading the electric network".I think biters must have been optimized too, given the level of detail everything receive. But i don't think there is any mention of some, just a single base on map view that has disappeared after the rechart. I noticed while searching for some, that there is a nice interchange visible on the first picture, despite the numerous (low-quality?) robotport.

Very interesting to read the technical as always. Not as thrilling as new things ( like biters ) , but it's a way to learn things that we won't discover when playing the expansion as in a regular game and it gives little insight on how to be more efficient when building. I was expecting to read that lamps can transmit power to neighbouring lamps for a moment when seeing the pictures, but i think it's funny to have them work like this on the space platform only, as if it's a better place to send light signals to other planets/far away.

I'm not sure i understand the part about the control behavior apart from " (a benchmark time went down from 131s to just about 8.2s to complete). " how it also relate to lamps, but for both i'm curious to see how it will apply on editor maps with only combinators. I had forgotten the RGB for lamps were implemented for 2.0, i thought it was only mentionned as a possibility. It's a pleasant reminder !

The bot logic part was easier to understand for me, it's also pleasant to hear, none of those optimizations seems like trade-off on anything on the game itself. It's time spent by devs to allow me to build larger crappy design before my machine give up ? I say YES THANK YOU !

[X3KJ]

With the radar update - what happens, if one radar is out of power? Is the registering calculated every tick?

[adam_bise]

Just want to say thanks for all of the work that has gone into optimization over the years. It is things like this that really make Factorio stand out.

[Galista]

It needs to be a counter because otherwise how do you know when the last registration was cleaned up? That is, a boolean would be "lossy" here - you would be throwing away important information. Another way of thinking about this, is that what was described in the blog post is very much like a "semaphore", and you're wondering why not use a "mutex" instead.

Oh I see, I'm stupid! Thank you for the explanation! Time to get my coffee...

[morse]

Hm, this attempt at optimizing the power update make me think. So, if the memory read is a limiting factor, could it be beneficial to compactly store the relevant values separate from the main objects? Modern memory is good at reading large chunks, so iterating over an array should be much faster than dereferencing a list of objects.

Regardless of the outcome, I hope the code for multithreaded update will be kept in the game. Hardware evolves, and what is a limiting factor now might not be so in 10 years.

[cynicalwanderer]

Hm, this attempt at optimizing the power update make me think. So, if the memory read is a limiting factor, could it be beneficial to compactly store the relevant values separate from the main objects? Modern memory is good at reading large chunks, so iterating over an array should be much faster than dereferencing a list of objects.

Regardless of the outcome, I hope the code for multithreaded update will be kept in the game. Hardware evolves, and what is a limiting factor now might not be so in 10 years.

Yeah this is what I was thinking too. The cure for memory limited when iterating over an O(n) list is usually to try to pack the relevant data more tightly in contiguous memory so as to reduce cache misses. Kind of like what DOTS in Unity tries to do

[morse]

It would be interesting to know, why don't process each planet in a separate thread/CPU core?

Because it won't be enough. Modern CPUs have tens of cores, and pin just one of them to a whole planet is just wasteful.

[keks244]

It would be great if lamps could be powered without the need for power poles on the planets too.

Perhaps you could add a special ground tile that can distribute the power between the entities.
A planet version of the space platform if you will.

Nice side effect:
You can use your space-platform blueprints even on planets, if you place the "energy tiles" underneath.

[Tertius]

Optimizing the game this way is great! I really don't like it to always have in mind: "If I build it this way, what would be the ups impact?" instead of: "If I build it this way, is this nice enough for my technical/aesthetic senses?"

My biggest time consumer is always the inserters. They require 1/4 to 1/3 of my update time. Since they are one of the earliest entities built into Factorio, revisiting them might perhaps give some potential. For example the arm swing. Once the arm starts swinging, it's similar to the robots: the speed is known and doesn't change, so the inserter could sleep up to the tick where the arm reached its target location and wants to grab or release items. Displaying the swing is just something for the visuals.

[Silari]

It would be interesting to know, why don't process each planet in a separate thread/CPU core?

Already been discussed multiple times in other threads. That only works if surfaces are truly independent, and they aren't. Linked belts/chests exist, and scripts can touch anything anywhere at any time, and connect things like power poles across surfaces IIRC, so you can't just cut surfaces apart.

It's amazing that you managed to hit the point where multithreading would slow you down at least in some places, very well done folks

IIRC they already had done that. Specifically, heat pipe updates were attempted to be multithreaded but it turned out it was slower due to RAM cache misses, mentioned in an FFF a while ago. I think it was before the heat logic was changed though, to the current form parallelized with the fluid and electric networks. Not sure on that.

[Agamemnon]

Ok first: Nice job!

Second: I wonder if we see some similar improvements for Biters. Their pathing and swarm mechanics are still something that has potential for improvement - most notably in the cases where they (for whatever reason) just accumulate over time and eat your UPS instead of your ammo.

[samcday]

Ok first: Nice job!

Second: I wonder if we see some similar improvements for Biters. Their pathing and swarm mechanics are still something that has potential for improvement - most notably in the cases where they (for whatever reason) just accumulate over time and eat your UPS instead of your ammo.

Don't forget that the original Space Age announcement included that tantalizing sneak peek at some kind of new life form (and other than our engineer, those things exist purely for us to gleefully tear apart with laser energy, bullets, missiles, poison gas, nuclear warheads and who knows what else?). Since then we've heard exactly NOTHING about any form of organic life in any subsequent FFF, besides the non-sentient Fulgora flora

So my prediction is our heroes at Wube are saving some of the best/juiciest updates for last, right before launch

[idgod]

So my prediction is our heroes at Wube are saving some of the best/juiciest updates for last, right before launch

Kovarex:

[kerrin]

Why do I find the optimisations so exciting? I don't think any of my bases have become noticeably slow. My mega bases were never really very mega.