Increase bot queue processing rate

[rain9441]

In the current version, the system checks 10 (EDIT: CORRECTION, it checks between 1 and 3 by default depending on successes or failures) ghosted entities per tick to see if a construction bot is able to perform the job of building that entity. In many games players will have 1000s of ghosted entities across their base (or multiple bases). Any entities not built are reappended to the back of the queue. Any new entities ghosted are also appended to the back of the queue. Then the system cycles through all of the queue at a rate of 10 (EDIT: CORRECTION, 1-3) entities per tick. (The Editor Extensions mod changes the maximum failures per tick to 10 behind the scenes which is why I had the original value wrong)

This can easily cause construction bots to lag and not pick up jobs for 10-20 seconds while the system processes the queue. EDIT: In some games players have a delay of as high as 200 seconds to over an hour

The rate of checking 1 ghosted entities per tick seems low. I would suggest raising it by an order of magnitude (10/100?) or a percentage of the queue (so that it guarantees checking every item within 1 second). This seems relatively safe to me to change for the main bot processing queue (which does not seem to include modules or concrete) and would bring times down from 10-20 seconds to 1-2 seconds.

[ptx0]

...

not everyone has such a capable system and it would overall drag performance down for everyone. this is achievable with mods already.

[ssilk]

I run also into that several times.

But increasing this may cause big performance problems.

I have this in mind as a solution:
It should pick up 10 items as yet, if they are different; but (maybe) 200 if they are the same. And the bots can place 20 items instead of one at once.

See also this mod https://mods.factorio.com/mod/Cursed-BCR

[rain9441]

If increasing the bot queue causes significant performance, then I'm okay with not making the change. If we believe it *might* impact performance, but there is no actual evidence to support that, then I still push for it to be considered.

The thing about performance of the bot queue is...
* If you have a low limit, then you could have long delays for your construction bots to build things, but still have good UPS on very low end machines. However, you cannot change the bot construction queue processing rate, so you are bound by this 10-20s latency.
* If you have a high limit, then you may have lower UPS on very low end machines. You can always place less blueprints to get your UPS higher. So you get either the low latency or the high UPS (or both if your CPU is from the last 5 years).

Furthermore, the bot queue size usually scales with base size and inversely with technology. The players who really suffer from this are the 0.00001% that push the limits of construction bots immediately in the blue science era. Most players who have bot speed technology and large bases filled with material don't have many issues with bot construction queue because they don't place blueprints faster than the bots.

[ptx0]

However, you cannot change the bot construction queue processing rate

yes you can

You can always place less blueprints to get your UPS higher

you could say the same for the person whose construction queue got too deep for the bots to be used effectively.

[ickputzdirwech]

I think this would be a perfekt candidate for a setting. The issue is: What kind of setting? Since it needs to be the same for all players in multiplayer.

This would really only make sense as a mod startup (or map?) setting. Or, to hide it in “the rest” and apply it only in single player.

[ptx0]

This would really only make sense as a mod startup setting

yes, and it can be done already as a mod, no need for Wube to change anything.

[rain9441]

From my testing, I changed max_failed_attempts_per_tick_per_construction_queue to 200 and max_successful_attemps_per_tick_per_construction_queue to 200. Game update went from approximately 0.900ms to 1.00ms. I changed it from 200 back to 1, and it went back to 0.900ms. I then painted a large chunk of the map with ghosts, 10's of thousands of ghosts. The peak update was 1.600ms, but usually stayed around 1.300ms. When I filled in the entire visible map with ghosts, the update time was 3.600ms. This was hundreds of thousands of ghosts with a setting of 200 instead of the default (which is 1 and 3 for successes/failures respectively).

I realize that there are many things you can do in mods in Factorio, but the mere fact that you can do something in a mod does not disqualify it from being a valid suggestion in vanilla.

I think I have shown there is evidence to support that this could be increased safely and would be a positive change to vanilla factorio.

[ptx0]

I think I have shown there is evidence to support that this could be increased safely and would be a positive change to vanilla factorio.

if you could reproduce it across a number of CPU types on Amazon EC2 or Google Cloud Platform (or even Azure - they have AMD EPYC systems), then that would start to appear as acceptable evidence but a single system / non-scientific test does not evidence make.

the aversion to mods doesn't make any sense to me.

[ickputzdirwech]

the aversion to mods doesn't make any sense to me.

I don't thin rain9441 qualifies as "mod aversionist". see https://www.youtube.com/watch?v=RW_yxSpsRFo

I said this before: The point of this subforum is to present ideas for the base game. If it is moddable, there are mods already or people make mods following ideas from this forum, linking them is great. It doesn't invalidate the suggestion however. If the suggestion get's implemented is entirely the devs decision.

[thuejk]

the system checks 10 (CORRECTION, it checks between 1 and 4 by default depending on successes or failures) ghosted entities per tick

I think it is between 1 and 3? max_successful_attempts_per_tick_per_construction_queue is 3 when I print it out ingame. max_failed_attempts_per_tick_per_construction_queue is 1.

/c game.players["thuejk"].print(game.players["thuejk"].force.max_successful_attempts_per_tick_per_construction_queue)

[thuejk]

My main issue is the failed attempts, which slows down to 1 per tick. What often happens is that there are many unconstructable ghosts in my world. And the queue will then repeatedly cycle through these unconstructable ghosts at one per tick, each time taking many seconds.

So the suboptimal thing is that the game is often slowly cycling though ghost it should know are unlikely to be placable, holding up stuff which is placable.

A suggestion which should not increase update time too much, but will make construction feel much faster: Have a separate internal queue for ghosts which have little hope of being built, i.e. failed construction at least once by having no materials or in network with no total construction bots. This means that the main queue can be set to always advance at max_successful_attempts_per_tick_per_construction_queue speed.

[rain9441]

Thue is correct in that it is between 1 and 3. I mistakenly added 1+3 assuming that it could have 1 failure and 3 successes in a single tick. But instead it is just 1 failure or up to 3 successes, but not both. Thue is also correct in regards to the failures being the main problem. If it stayed at 3 successes per tick and the failures per tick was increased to 50, it would solve this problem just the same as 50 successes per tick and 50 failures per tick. There are a lot of fast players, but I don't know many players who are so fast that their bots build 3000 different entities each second!

I also agree that some more advanced logic to optimally apply construction bots would be beneficial as well. However I don't believe that it is necessary to remove the symptoms here. I think that just bumping the rate up by 10 or 100 will improve the performance of the queue dramatically without any noticeable cost in UPS to the user (nor any cost in development time ).

[ickputzdirwech]

Any new entities ghosted are also appended to the back of the queue.

Have a separate internal queue for ghosts which have little hope of being built

Adding new ghost to the beginning of the queue would increase responsiveness drastically in some cases I imagine. I also really like thue's suggestion. Essentially a separate queue for new ghosts and when they fail they get moved to the regular queue.

I think that just bumping the rate up by 10 or 100 will improve the performance of the queue dramatically without any noticeable cost in UPS to the user

I don't think it's even necessary to increase the rate that much. In cases were only a small percentage of ghost could be build (for example because most are outside of a network), doubling max_failed_attempts_per_tick_per_construction_queue could already nearly double the building speed.

What I don't get is why failed and successful attempts are handled differently i the first place? Is a failed attempt more costly than a successful one? Couldn't they be merged in max_attempts_per_tick_per_construction_queue?

[ssilk]

This all is interesting, because it doesn’t explain to me, why roboports and poles seem to have a higher build-priority than anything else. They are really build earlier, even if enough items of everything is available.

[rain9441]

I did not have a similar experience as you ssilk. Poles and roboports were not prioritized and this is why players place skeleton blueprints that have only poles and roboports in 100% speedruns prior to placing all other entities.

Last night I did a test in my 100% planning practice. I changed the bot queue failure rate from 1 to a higher number mid run (10 or 100 I forget, it was not unreasonably high). It had a significant effect on the rate at which my construction bots built things (very positive!) without any sort of performance hit. Prior to the change, most bots were sitting in roboports doing nothing. As soon as I made the change, 100s of bots across multiple roboports began working furiously.

Hoping this gets noticed by the dev team at some point!

[Nexarius]

I think it would also be good if it didn't have to cycle through 5000 concrete until it started to build the newest placed blueprint. It would be good if item types would be grouped instead.

[jodokus31]

I think it would also be good if it didn't have to cycle through 5000 concrete until it started to build the newest placed blueprint. It would be good if item types would be grouped instead.

Tiles are already separate, afaik

Last night I did a test in my 100% planning practice. I changed the bot queue failure rate from 1 to a higher number mid run (10 or 100 I forget, it was not unreasonably high). It had a significant effect on the rate at which my construction bots built things (very positive!) without any sort of performance hit. Prior to the change, most bots were sitting in roboports doing nothing. As soon as I made the change, 100s of bots across multiple roboports began working furiously.

Hoping this gets noticed by the dev team at some point!

Sounds nice!

[Nexarius]

I think it would also be good if it didn't have to cycle through 5000 concrete until it started to build the newest placed blueprint. It would be good if item types would be grouped instead.

Tiles are already separate, afaik

Okay but that doesn't help when some players put down large blue belt blueprints on a server before you can even make blue belts.

[ptx0]

in 100% speedruns
[...]
Hoping this gets noticed by the dev team at some point!

ah, that explains the aversion to mods.