Performance issues when force-placing 22k element blueprint?

[GenericKen]

I'm building my Von Neuman Mk.3 on belts, and I'm seeing some weird, non-deterministic behavior whenever I plop an expansion.

The attached screenshot has the original 21.2k element expansion, and it actually built pretty smoothly off of ~3k construction robots.
I've since:
-added about 600 solar/accumulator elements to the plan
-added about 5k construction robots
-and started plopping expansions over dead trees and lakes with force place

And at some point, one of those factors caused the construction robots to act weird. Sometimes, they would sit idle for 2-4 minutes, ignoring construction ghosts in the range of the previous expansion's roboports, before suddenly exploding with activity. Sometimes, they would ignore the ghosts in the range of a new powered, plopped roboport for minutes before they started construction.

What could be causing this behavior? It's not the personal roboport - this could reproduce (sometimes) when I took my armor off. Are they assigning jobs in an exponential manner? If I increased my number of construction robots from 8k to 24k, should I expect a 3 fold decrease in performance? Maybe 9 fold? Or would the problem instantly go away once I saturated the network with more than 22k construction robots?

[DaveMcW]

The game loops through 1 ghost per tick, and either assigns it a construction bot or gives up.

Most of your ghosts are initially outside the range of a roboport, so there is a long pause while the game examines each and gives up.

A partial solution could be to randomize the ghost search order, instead of searching row-by-row outside the range of a roboport. It would be nice if the developers did this, but you could also do it by editing the source code of your blueprint.

Another solution is to use 2 blueprints, one with only roboports and power lines, and one with everything. Place the roboport skeleton first.

[GenericKen]

Ah. Ahhh... my blueprint are going from the bottom up, not from the top down.
How would I hack the source code of my blueprint to go from top-down to bottom-up? Does it matter which direction I draw it from?

Weird that ghosts don't have priority.

Also weird that it only examines one per tick. That implies that blueprints max out at, best-case, 18k elements that can be assigned construction robots before the 5 minute timeout.

Is this on the devs' radar?

[orzelek]

Ah. Ahhh... my blueprint are going from the bottom up, not from the top down.
How would I hack the source code of my blueprint to go from top-down to bottom-up? Does it matter which direction I draw it from?

Weird that ghosts don't have priority.

Also weird that it only examines one per tick. That implies that blueprints max out at, best-case, 18k elements that can be assigned construction robots before the 5 minute timeout.

Is this on the devs' radar?

I'd recommend looking at one of mods that expand time of ghosts when playing around with such huge blueprints.

[GenericKen]

I'd like them to build more quickly than 5 minutes. The ghost expiry is more of an inconvenience.

Starving out all my outpost blueprints while the main factory builds itself is actually more of a problem. I guess I could create a skeleton blueprint with just power and roboports, but inventory space is at a bit of a premium.

Are there mods for tweaking the order of object placement in blueprints? I'm unfamiliar with what the underlying blueprint data structure looks like, nor how to tweak it. Does the game always can ghosts from top to bottom, or does it scan in the order of ghost creation defined in the blueprint?

[codewarrior]

The bug we discovered here might be related.

In short, whenever a roboport is mined or runs out of power, it will cancel ALL build orders for its entire network. This causes the construction bots to become very stupid for a short time, returning all materials to storage and then taking them right back out of storage to build elsewhere.

And yes, it's a bit unfortunate that there's an 18k hard limit on blueprint size due to the 1-ghost-per-tick interacting with the 5 minute timeout. I would like to beg the developers for an "Automated Construction II" technology, that not only increases the timeout, but also allows a construction bot to carry more than one of an item and build them both in a single trip. This would make paving the planet go so much faster.

I second DaveMcW's suggestion to use two overlapping blueprints. But to overcome the "unpowered roboport cancels ALL construction" bug, it would work better to have the first blueprint have only power lines, and drive around using your personal roboport to place them just to make sure the roboports never become unpowered.

[GenericKen]

Glad to know I'm not alone.

The unpowered roboports cancelling construction is more of a minor inconvenience than the fact that only one construction robot is assigned per tick. That max limit makes it kind of pointless to have more than 18k construction robots.

The real developer solution to "unpowered roboport cancels construction" and "robots sit idle while game analyzes ghosts outside of construction areas" problems would be to simply tweak the ghost evaluation. Instead of having a single queue for evaluating ghosts, have a high priority queue for power and roboport ghosts, and a normal priority queue for everything else (just like having a skeleton blueprint, but less dumb).

[GenericKen]

Okay, watching construction, I see now that roboports powering-off is in fact killing a lot of my construction time.

Are there any mods out there that will either:
1) Make robots prioritize electric/roboport ghosts? or
2) Allow me to clone and edit a blueprint to remove everything but electric, roboport, and train elements? Maybe a lua command for removing ghosts that aren't of type whatever?

I've found that building my own bare robo-electric skeleton blueprint to be time consuming and error prone.

[codewarrior]

I don't know about any code-based solutions, though I'm sure it's possible.

I usually edit blueprints by placing them well outside of my roboport network, then removing ghosts with right-click and adding more ghosts by shift-clicking items in. Then, I just take a blueprint of the ghosts.

Good news: The unpowered/removed roboport bug will be fixed in 0.12.5!

[Rseding91]

The construction manger actually sends up to 9 robots per tick if you have enough robots. 3 for ghosts, 3 for deconstruction, and 3 for repairing.

It could be changed to send robots to random ghosts-to-be-built but I don't see how that would be better than the current system (first in first out).

[GenericKen]

So... what are the lua properties of a ghost?

It would make life much easier to be able to simply strip the non-electric/roboport ghosts out of my massive plan with a command. Something like

for _, entity in ipairs(game.player.surface.find_entities_filtered{area = area, name = "ghost"}) do
entity.destroy()
end

but I don't know if destroy is the correct command for ghosts. Also, ghosts aren't named "ghost", and "ghostname = "redbelt"" seems to return true for everything? It's weird.

Any modders happen to know the correct way to interact with ghosts?

[DaveMcW]

The construction manger actually sends up to 9 robots per tick if you have enough robots. 3 for ghosts, 3 for deconstruction, and 3 for repairing.

It could be changed to send robots to random ghosts-to-be-built but I don't see how that would be better than the current system (first in first out).

Randomizing the order of ghosts in a single blueprint could help this specific case (22k entities, 80% of them out of roboport range, 3k robots).

But a better solution would be to add a 10th robot, reserved only for roboport and power line ghosts. If another robot impacts performance too much, I would even argue that 2 general ghosts and 1 roboport/power ghost is better than 3 general ghosts.