Logical requests delayed

Don't know how to use a machine? Looking for efficient setups? Stuck in a mission?
Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Logical requests delayed

Post by Kahnugo »

I have started to have increasingly severe issues with provider chests clogging due to missing deliveries assigned (even though there are at least 5k unanswered requests in the same network).

My assessment currently is that buffer chest requests is not able to be handled in time before it starts over, but this is merely a guess based on what I see in game. I don't actually know how the game deals with logistic requests, so if someone has a more enlightened perspective or even potential solutions it would be highly appreciated.

The specific situation is a factorissimo2 factory that is nuclear powered from within, hence it needs a tonne of water. I am using a large amount of buffer chests both for the incoming water barrels and the outgoing empty barrels to avoid long delivery routes for the factory connections. At the moment I have about 40 buffer chests for water barrels and empty barrels respectively.
The problem seems to be affecting the empty barrel output chests the most and I've observed chests being full without any planned pickups for up to 7-8 seconds, even though when I look at the buffer chests for empty barrels, most are almost empty with less than 100 incoming barrels.

Zavian
Smart Inserter
Smart Inserter
Posts: 1641
Joined: Thu Mar 02, 2017 2:57 am
Contact:

Re: Logical requests delayed

Post by Zavian »

Post the save.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

TheNest.zip
(40.63 MiB) Downloaded 117 times
Edit: Updated

2.Edit: Updated
TheNest.zip
(43.31 MiB) Downloaded 102 times
Last edited by Kahnugo on Sun Nov 22, 2020 11:47 pm, edited 2 times in total.

User avatar
Nosferatu
Fast Inserter
Fast Inserter
Posts: 228
Joined: Fri Jan 20, 2017 4:48 pm
Contact:

Re: Logical requests delayed

Post by Nosferatu »

Can't check your safe at the moment but here is one clue:
Buffer chests are the last thing on the priority list. They only get filled if your bots have nothing else to do

Full info from https://wiki.factorio.com/Logistic_network
A requested item is first looked up in active provider chests and in the player's trash slots, then in the storage chests and buffer chests, then the passive provider chests. So, the active provider chests are emptied first, then the storage chests and buffer chests, then the passive provider chests.
Requests are assigned first for player logistics, then for requester chests, then for buffer chests.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I forgot to mention that available bots moves between around 1k and 3.5k. Often more than 25% of the robots are idle while this is going on.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I seems like switching to active provider chests might help solve the issue. At least the immediate problem seems to disappear after the switch. I didn't really want to use active providers, but in reality it doesn't really interfere with anything as there wont be any storage chests in near proximity and the empty barrel requests need to be handled by circuit logic anyway.

It might cause a ripple effect if the cause is the processing speed of logistic requests, but it might not be visible if it pushes the problem to areas with much less traffic.

Edit: It makes me think that maybe there's a time limit on each update for each logistic network, and that makes buffer chest requests not be processed as fast as you might expect in large networks.

2. Edit: One of the requester chests supplying water barrels to the nuclear power seems to be having similar issues. It might have been struggling all along, but seems to confirm that the game struggles getting through all the requests. (I'm expecting active provider chests to be handled before requester chests.)

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I've expanded a little bit and it's becoming more noticeable. Specifically with the requester chests that are the most sensitive elements in the supply chain to the power plants.
I've had a single nigh blackout when one of the chests emptied out for some time and power dipped below full saturation (starts a death spiral because the water supply slows down at that point) and it seems like the issue is also the culprit (at least partially, congestion at charging stations probably plays a role as well) behind the wild flailing of the number of available robots (varies from <10% of total to >45% of total, with total being 8550 robots atm) , while the consumption of the base is roughly stable.

It is a little confusing to me, I can see that the update time on the logistic manager varies quite a bit and I've not been able to pinpoint anything that should trigger some sort of stall, but I'm playing blind in the sense that I don't know how things are handled under the hood.

User avatar
Nosferatu
Fast Inserter
Fast Inserter
Posts: 228
Joined: Fri Jan 20, 2017 4:48 pm
Contact:

Re: Logical requests delayed

Post by Nosferatu »

I don't know the root problem.
Probably to much logistic requests but I don't know the limits there - I'm a belt base guy.

But one suggestion: Do switch to active providers.

Passive providers are the last thing on the list when a robot chooses where to pick up stuff.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I guess there's not really much I can do at this point. I think I can conclude that it's affecting logistic networks globally as I'm starting to have issues on the base floor of the factory which is a significantly smaller network that the outside one.
Changing priorities only works to a limited degree as it seems it's mainly moving the problem around a bit as it seems to affect low priority requests more than high priority ones. By now every critical point is seeing an effect and it's making the bot requirements explode as the requests come in bursts bouncing the network load from 25% use to 75% in a split second even with stable demand by production. As a side effect this also makes congestion problems worse as effectively the network is affected as if it had much higher demands than it actually has.

I don't really have much hope getting an answer on this as not a lot of people seems to have knowledge of how the logistic manager works, which I guess is what you can expect since it doesn't really affect normal play :p. It is frustrating though that it stalls the logistic manager before it starts having a real impact on overall performance. Definitely room for improvement on that.

Guenni7
Fast Inserter
Fast Inserter
Posts: 144
Joined: Thu May 18, 2017 5:53 am
Contact:

Re: Logical requests delayed

Post by Guenni7 »

Hi Kahnugo,

the behavior you describe sounds much like charging congestion at the roboports.
As far as I know bots that are charging (or waiting to be charged) but without delivery task yet, are shown as idle,
doesn't mean they're ready to deliver something.
You can turn on "show bots on map" and look for crowds near roboports, if there are many bots waiting to recharge, you need more roboports.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

It is not charging congestion, as that affects jobs already assigned (but the delivery is delayed). What I am experiencing, on top of congestion (in fact it makes congestion a lot worse because it causes jobs to be created in bursts), is that tasks are not created for the robots and they go idle and then it sort of catches up and creates a lot of tasks in a split second.

This is detrimental for several reasons.

1. It increases the practical load on the network as more robots can be active at the same time (load on the network fluctuates wildly from less than half of robots used to almost no idle robots in a second or two, in a network consisting of almost 12k robots by now).

2. The 'bursty' nature of the way tasks get assigned creates a lot of congestion every time a bunch of tasks are created at the same time overloading the charging stations because a lot of the robots run out of charge at the same time, more so than if tasks were created as demand surface.

3. It is creating a huge issue on my supply chain, starving my very sensitive nuclear power setup of water (I've installed buffers that helps patching this issue a bit, but earlier even 10 seconds if no supply could on occasion kill some of my power plants and if power at any point went below 100% a blackout would be just around the corner (roboports can't keep full charge if they are in constant use, thus creating more congestion which leads to more severe shortage).

4. Furthermore it puts a computational strain on the save causing UPS to be affected earlier than it would have been if not for this issue.

Edit: It is a bit hard to assess the robots' distribution in the network precisely because of its size in terms of robots and roboports (I have already packed the area with as many roboports I could fit in, and the most crowded area has just shy of 500 roboports). As far as I can tell robots very rarely travel more than 2 roboports further than the closest when they reach the charging threshold, which as far as I have been able to gather means that the queue of the most congested roboports generally do not reach more than around 15 (penalty queue + distance*2, assuming 7-8 distance to neighbor's neighbor that is 14 to 16 minus queue on the center, so if I see 2-4 robots on those roboports and 0-1 on the one beyond that should mean that the queue on the worst ones are at about 15 robots, which does not explain the 4-5k jumps in tasks being created.

Guenni7
Fast Inserter
Fast Inserter
Posts: 144
Joined: Thu May 18, 2017 5:53 am
Contact:

Re: Logical requests delayed

Post by Guenni7 »

When you have bursts of assigned bots while other times they're sleeping while they could just work steady, I think your Processor can't handle that much bots.
You might try to come up with a design more efficient, to be honest - I never had 12K bots in one network because that doesn't make much sense.
Two things I can suggest: Limit the size of the chests (that spares a lot of computing power) and make smaller, seperated bot-networks with much less bots.
Bot networks > 2K bots start to be ineffective IMHO.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

The context is that I'm placing an entire factory inside a factorissimo factory building so the whole setup is going to be extreme and I'm entirely prepared to accept that I'm going to run into hardware and software limitations. I think though that the issue I'm running into here is a software issue rather than a hardware issue, because it started being an issue before I even started dipping below 60 UPS (still get 60 UPS when nothing in particular is going on on my screen).

There's no way around having a large bot network transferring items into the main building (the water supply alone for the power is an enormous amount of barrels), my throughput is technically limited by chest size, so smaller chests are actually bad for performance in some way in this setup, specifically in those connections where this issue is prevalent chest size is fully utilised. (chests are inefficient performance wise if the game has to scan empty space to search for nonexistent items).

I'm confused as to what exactly is causing this behaviour to appear this early, as it is as I see it detrimental to performance, for example the bot network is 20-30% larger than it would have been if tasks had been created continuously and not in bursts (requiring way more robots than it "should" had it been done efficiently). In the case that it is something specific that is causing the process to stall, it might be able to design it in a way to lessen the issue, hence the post (also I am curious :)).

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Logical requests delayed

Post by orzelek »

It might be a little separate but a question:
Is Factorissimo2 fluid delivery mechanism to factories not enough to deliver water for your power plant?

AFAIK you can deliver water directly to buildings instead of water barrels. Trying to feed nuclear power plant with barells sounds like main underlying cause of your problems.

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

Fluid connections are not even close to being enough sadly. The largest issue being handling the in and output which I don't think is possible at all given the space constraints. Consider that the theoretical max throughput is 15000 units of water (2500 storage in the in/output factorissimo pipes) you have to handle 12.5 pipelines of fluid for each connection used. And if you look at the water requirement for nuclear power you get that each fluid connection can theoretically supply up to 1.456 GW of power (and that is assuming that the logistic nightmare of pipes can actually be solved :p) Currently I might be able to squeeze 10GW out of my nuclear setup, I hope to get a bit more once I redesign the water barreling system to be less wasteful of logistic network capacity. That's at the very least 7 fluid connections (where you have to handle 83 pipes worth of fluid both on the inside and outside of a single factorissimo building along with the rest of the materials required for a full base).

Theoretically the maximal throughput of chest connections with barrels are 72000 fluid units per second compared to 15000 (that is taking barrels out into account), that too is impossible in practise I think, but I think around 30k is possible per connection used. It's still the most challenging logistic challenge by far (I estimated 70%+ of the total material transport used to supply the base could be attributed to the power supply when I tried to run the numbers)

User avatar
Nosferatu
Fast Inserter
Fast Inserter
Posts: 228
Joined: Fri Jan 20, 2017 4:48 pm
Contact:

Re: Logical requests delayed

Post by Nosferatu »

Kahnugo wrote:
Sun Nov 22, 2020 12:14 pm
I'm entirely prepared to accept that I'm going to run into hardware and software limitations. I think though that the issue I'm running into here is a software issue rather than a hardware issue, because it started being an issue before I even started dipping below 60 UPS (still get 60 UPS when nothing in particular is going on on my screen).
I agree can't be a hardware limitation.
Factorio is deterministic.
The logistic network has to perform identical on every hardware or it would cause desyncs in multi-player.

But I think there are hard limits. For example I have a big construction job going on in my factorio and the missing materials counter magically stops at 600.
It looks like that limit is per network.
If I start constructing something in a different network the number rises beyond 600.

In case the logistic network behaves the same you could split your network in 4 parts...

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I had that thought, but it doesn't seem to be the case as I could see the same effect worsening in the smaller networks inside the factory as well. As with everything regarding this, it's hard to make precise observations, I could try to force the game speed down, though that could change the dynamics completely.

I don't think it's all that relevant if it's deterministic or not, it's more like it's stalling even though the update itself is not delayed at the same rate. The logistic manager update is fluctuating between 1 and 4.5 ms which seems odd to me, causing me to think that something it does makes it stop mid loop delaying the processing of some requests, though I haven't been able to see if there is any synchronicity in the delays (is the affected containers affected at the same time or does it happen more randomly, I suspect it happens at the same time).

Edit: It's not really an option to split the network as it delivers material to and from the same 25x25 grid.

2. Edit: Interestingly enough it's still doing it at 20% speed (12 UPS).

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Logical requests delayed

Post by orzelek »

How factorissimo2 transfers the items?
Doesn't it do it in batches with some delay between transfers?

What you are seeing seems to point to fact that items are being moved to chest in big batch then bots go out and grab them. That would also explain high variability of logic manager time consumption.

Another solution might be going to author of Factorissimo2 and asking for bigger transfer pipes. (I'm assuming that building nuclear power plant outside is not an option for some reason)

Kahnugo
Inserter
Inserter
Posts: 48
Joined: Thu Nov 12, 2020 1:10 am
Contact:

Re: Logical requests delayed

Post by Kahnugo »

I'm building it inside because I want it to be inside :p solar outside would have been a vastly superior option.

It does transfer items in pulses, the intervals can be adjusted between a number of different values. The fastest is 10 ticks between transfers. That sets the absolute limit for throughput through connections to container size times number of transfers per second.

It does not explain why the logistic manager sometimes stalls for up to several seconds and stop creating tasks for deliveries. For example empty barrels come out of the factory at a quite high rate. I currently have chest transfer connections transporting barrels out of the factory (updates every 10 tick, transporting all barrels from the inside chest into the outside chest until the outside chest is full). I can watch these chests (now outside chests for empty barrels are all active providers and I have plenty of free storage) and most of the time they have between 20 and 100 barrels in them and incoming deliveries, but regularly the tasks cease to be created and I can see them dwindle and the content rise, sometimes until the chest is full and occasionally it stays full for some seconds too.
Why this happens is not clear, all the conditions are met for tasks to be created. There are robots resting in roboports, there is large quantities of unfulfilled requests and available barrels in the logistic system. The obvious answer that comes to mind is that the system is overloaded for some reason, but why and how is not obvious. And it's not because the update is taking too long either, the logistic manager update time drops to less than 25% of what it is when it picks back up and start filling all these orders that weren't filled the last couple of seconds and the UPS is not typically low when it does this.

The main issue with pipes in factorissimo is handling the fluids before and after transfer, not the potential transfer rate (even if it is still lower than the potential barrel transfer rate). I just can't build something using 50+ pipes of fluid coming into the building with actual pipes.

Guenni7
Fast Inserter
Fast Inserter
Posts: 144
Joined: Thu May 18, 2017 5:53 am
Contact:

Re: Logical requests delayed

Post by Guenni7 »

I had a look into your save today and it looked totally normal to me.
There are bursts when copper or iron comes in by train, water barrels are transported quite steady.
It's true that the buffer boxes are not getting much (lowest priority), but that's because you always have around 2-3K bots charging at a time and the others are
busy. Guess that is normal behavior on a high troughput bot network. I wasn't able to see idle bots sitting in roboports, they all got dispatched after ~1 second.
I tried to erase the buffer chests and the network got less busy, I think they are slowing the troughput down because bots needs to handle every barrel two times more.

Post Reply

Return to “Gameplay Help”