Round robin scheduling for trains / Train Limits can Soft-Deadlock when Incoming and Outgoing are Both Full

[GPFault]

Hello,

>> What did you do?

I have several unload and load stations. I found that some trains are never moving. I created a test setup to highlight the problem.
I placed three "A1" stations and one "B" station with a limit of 1 train. I placed three trains with the same schedule: move to "A1", wait 5 seconds, move "B", wait 20 seconds (time enough to move from B to A1).

>> What happened?

The train above never moves.

>> What did you expect to happen instead? It might be obvious to you, but do it anyway!

I expect that all trains will visit "B" station one after one.
Perhaps it needs to make that train early found that "Destination full" will move early than a train that found the same later. It will balance the load evenly.

Screenshot_6.png (1.46 MiB) Viewed 5618 times

Save file:

nevermove.zip

(1.74 MiB) Downloaded 170 times

Import Blueprint:
0eNqdnO1OGzsQhu9lfwfk77FzK0cVSiFqVwoBJaE6COXem5BsWpLZ7vvOPyjwMOvxrMePXT6676u35eumX++6+UfXP76st938v49u2/9YL1bHf9u9vy67edfvls/drFsvno+fbRb9qtvPun79tPy/m/v9t1m3XO/6Xb88/fznJ+8P67fn78vN4RsuP/n4tvm1fLr7BMy615ft4Wde1sdfdOCE6tyse+/mRcKB/tRvlo+nL5f97AYaYGjVoVGBxgt0uzvwfvzcjWGlyRmbv2JFwSYYW73XsV7BZnAIpKUztE5Dy5c0352ngjoA93mI9f4q2qyABU2Y/xOtgqlEiuIJVKMGakRS8r9A3hlCyirJG2LSScEQk6ikaIhJJyVDTE0lZXCSVh+HSVrbvT5SxfB8elTCP1/zKqnyMY2QmiEmdZYHZ4hJJ3lDTGruQjDEpJMi/C4NnxxxbvqVF5IhPrV2QjaQ1Fka+PkuTp1bAX2jSy1nTvk6ZkGjEi/42s7cOL20hwYvbP5+eO6ILGzRGUZUrYzoDSR1Pke0PZLkzpw63XNFoj9KQ3wynfNIrAxJCG4muI3g4jUk2RNcYgXJkeASNZWZvBGrSybyloiOKhN5S8TKU4i8JaLbKkTeElFvhchbIuqtMHkj6q0weSPqTZi8EfUmTN6IehMmb0S9CZG3TNSbEHnLRL1VIm850P3/offQ+/8cDWuu2qNloqriOTveT9uDjO52JNXL09brzkVbz/OfujoGvL7b7l5e/5n0dpWc2eFRF6ePu077DWIYXbVvzRXtaGI6j20CFApvAY4507yJgzuuoMenZah4vjPycbqCSuA7I4gb+c4I4ia+M4K4me+MIG7hOyOIK3xnBHEr3xlB3MZ3RghXHN8ZQVzPd0YQN/CdEcSNfGcEcRPfGUHczHdGELfwnRHEFb4zgriV74wgLlFvlchbJeqtEnmrTIdI5K0G3iJBXOaEiMhb5d04xiXqrTF5K/RJGcZlHDqTN8KoOyZvRP/oiLw1Rx5v3nSTmgNtBr/o9cMzor7ikCeZ3uu0CO912rDX8QnZ67RE7nV8Jvc6Dd6ntUrGXugD5JvR1nY/zbA78+ouulXD3rcB86EZuDJdCd458jhagtMPfy01pZ+POqao4mUQVVRkd8shTM8X75LhYUcizPzDBv2M0xmOtMZQYohq5FZAZY9gAiAEvGv8GUwAVhzPXGMYVAMG9rxrwMCBlw0YOPK2AQMnXjdg4Mz7BgxceOGAgYU3Dhi48soBAzfeOUDg4HjpgIE9bx0wcOC1AwaOvHfAwIkXDxg48+YBAxdePWBg4d0DBq68fMDAjbcPEDg6Xj9gYM/7BwwceAGBgSNvIDBw4hUEBs68g8DAhZcQGFh4C4GBK68hMHBjPUQAPIRPhitTI202c6fj0rHrN0GZaxzDZjMAUsMng9UIkBnwf13lmECLZ9GZNCaBNSY+4bfN6YER3pkEwJn4VA1zd2TCNXaXHh0QIXOd41ISupLIntUkUdckORhGbSSmaHi8kXvkBp8xhjL4jKhLiGzwGWMog8+I+os2G+b9GKoZohq5vW9YScZQhpUk6lO0GC4oRRn7DwqGG0pjcSXDI+rzvcC2O6bLI7brV3dS0Zb5PxKlYf4nvZSKYf6Poej3firAuAnttK+xWcV6vi1KDQHTl9RzALotiWzPeo1VW2FJcE27YcKnm2t46oItme9VrkdY7bMF7q/ypTs8foQMh4Dd4aAQ8tVVx/KlO5x1z4v122L18AnbPqz65343NlwV7hod+1QGu56RjVQ12HUMbLDrGNhg1zGwwa5jYINdx8AGu46BDXYdAxvsOgY22HUMbLDrELgZ7DoGNth1DGyw6xjYYNcxsMGuY2CDXcfABruOgQ12HQMb7DoGNth1BBycwa5jYINdx8AGu46BDXYdAxvsOgY22HUMbLDrGNhg1zGwwa7fgr/NTn9GZf7XX12Zdb+Wm+3pGw6bMmmHHq20UFzd738D1A9+nA==

[mrvn]

I'm split on this.

On the one hand round-robin of trains is what other cases use.

On the other hand activating the closest train is also good.

Is ore from one mine any better than ore from another? Or iron plates from a smelter? What situation do you have where round-robin is better?

[Klonan]

I am moving this to ideas and suggestions, nothing here is to suggest this is a bug

[GPFault]

Is ore from one mine any better than ore from another? Or iron plates from a smelter? What situation do you have where round-robin is better?

Please consider if you have two Red Circuits block and one Blue Circuits block as "A1" stations (Blue is on top). And station "B" is Green Circuits supplying block. Blue Circuits will never get a chance to start production until Red Circuits will not be filled with trains. It is pretty difficult to manage. With round-robin production of Blue and Red Circuits will be more balanced.

[mrvn]

Is ore from one mine any better than ore from another? Or iron plates from a smelter? What situation do you have where round-robin is better?

Please consider if you have two Red Circuits block and one Blue Circuits block as "A1" stations (Blue is on top). And station "B" is Green Circuits supplying block. Blue Circuits will never get a chance to start production until Red Circuits will not be filled with trains. It is pretty difficult to manage. With round-robin production of Blue and Red Circuits will be more balanced.

True. If you look at this from a supply point of view with trains stationed at the consumers.

I tend to have the trains at the producer except for ore. Because:

1) generally there are more consumers than producers so it saves trains
2) trains as the producer will be filled already when a consumer needs goods and only need to travel 1 way instead of a round trip. So delivering items is faster.
3) factorio is more tuned for over-producing than over-consuming, balancing happens when one consumer backs up and stops consuming

But it's a valid example to have it the other way around. Not sure how to fix this unless you manually balance your stops with circuit logic.

In the long run the red circuits will back up at some point unless you constantly drain them with something else and then blue circuits and red circuits both get trains. It just takes forever till that happens with the amounts of green circuits consumed. And every time the blue circuits eat a red circuit train they get exclusive deliveries till they are backed up again.

The simplest solution is: Produce more green circuits.

[Khagan]

The simplest solution is: Produce more green circuits.

In fact, just produce more of everything. The factory grows ...

[GPFault]

3) factorio is more tuned for over-producing than over-consuming, balancing happens when one consumer backs up and stops consuming

It is always over-consuming of something

[DarkShadow44]

How do you propose this should be implemented? Because round robin is not always desired either...

[Nidan]

You can force round robin behavior using the circuit network. Implement a counter or pass around some token to control which station releases the next train.

[MEOWMI]

It is nice for figuring out that you aren't producing enough, because it's easier to notice one station struggling to keep up with production than ALL stations struggling to. As others have pointed out, you almost always want to have an excess of supply trains.

Hard to say exactly which way is better, though I did sort of expect it to be round robin since inserters work this way and IIRC trains trying to reserve the same block works like that too.

[ssilk]

I have several unload and load stations. I found that some trains are never moving. I created a test setup to highlight the problem.
I placed three "A1" stations and one "B" station with a limit of 1 train. I placed three trains with the same schedule: move to "A1", wait 5 seconds, move "B", wait 20 seconds (time enough to move from B to A1).

I just would add a station “Z” which is in front of B and is able to hold at minimum 3 trains (nearly as much as are on all “A” stations).

The schedule for all trains is then A1, Z, B.

[Tricorius]

TL;DR

Trains (particularly with dynamic train limits) can soft-deadlock when both “ends” of a loop are “full” (in regard to Train Limit).

What ?

It’s not my solution, so I’ll point to Nilaus’ recent video: https://www.youtube.com/watch?v=i2DyxgQxxmA&t=245s.

Scenario: fairly standard logistics train “loop” consisting of load (e.g. iron ore mine) and unload (iron ore for smelting).

Assume two stations, load and unload; two trains; and both stations set with Train Limit: 1.

Trains that have fulfilled their “leave” criteria (fully unloaded / fully loaded / satisfied circuit condition) are still counted toward the train limit. In the stated situation, both stations are at Train Limit: 1/1 and soft-deadlock (neither station accepts either train). They both wait with “Destination Full”.

Removing “pending” trains, or whatever the condition is called, would set both limits to 0/1 and allow both trains to proceed to their next station, freeing the deadlock.

Why ?

It’s frustrating debugging clashing states and I think this soft-lock could pretty easily happen to someone without in-depth knowledge of the train system; therefore being nearly impossible to debug. I can’t think of a time when this would be an undesired behavior; so I don’t see any downsides.

[Kyralessa]

If you only have two stations (in this schedule) and two trains, why are you using train limits in the first place?

[Tricorius]

If you only have two stations (in this schedule) and two trains, why are you using train limits in the first place?

I don’t. This is a simplified example. I have blocks of sets of stations. But the math is the same for 2 or 2000. If you have the same number of trains as you have stations with train limits they soft-deadlock. (Ten loaded trains, 2 each in five blocks of loaders; ten unloaded trains, 1 or 2 of each block of unloaders. It’s not fun to debug. I, for one, wrongly assumed that only an active “at the station” train or one in transit had a reserved spot in the limit queue.)

Also, the train limits get set dynamically based on the number of loads available in the chests at the station.

If it has enough to load (or unload) one train, then the limit is set to 1. If it’s storing 4 train-loads it is set to 4. (Although there is a max override in case there are only two local waiting branches.

For the curious, here’s a megabase (Nilaus) showing the problem at a much grander scale: https://youtu.be/uyenBf4g9Zk

There could also be reasons you’ve set Limit: 1 on a simple “loop”. If you didn’t build a stacker, you don’t want a second train blocking the main railway (again, intermediate rail knowledge wouldn’t necessarily know they need waiting branches or off-network load / unload so a second train may clog their entire network). This solution would resolve these soft-deadlocks.

[disentius]

To clarify the "bug":
1. train wants to leave
2. there is no free station available. All destinations report x/x as trainlimit/train reservation.
3.The train has left the station (no "T" signal) yet the station counts the departed train toward the reserved trains in x/x

[Tricorius]

To clarify the "bug":
1. train wants to leave
2. there is no free station available. All destinations report x/x as trainlimit/train reservation.
3.The train has left the station (no "T" signal) yet the station counts the departed train toward the reserved trains in x/x

Exactly. The proposed solution: if a train has “left the station”, it should free it’s reservation from the limits queue / array.

[mmmPI]

Removing “pending” trains, or whatever the condition is called, would set both limits to 0/1 and allow both trains to proceed to their next station, freeing the deadlock.

Yes it would allow both train to move in the example.

But it is not something you can generalize (imo).

All the trains with "destination full" located at the top of example in the video would be removed from all their stations since they all have the condition for that particular station clear.

This means the game would see 4 or 5 station "open". But in this case sending 4 or 5 trains could cause problems.

In the example it's a particular case, the one where the soft-deadlock occur in a visible way but with only 2 different station name and also 1 train only in one of the 2 areas containing train stops.


If you where to have train schedule with A B C for example, and more trains, then some trains could be considered "pending" for C being stationnary at station B preventing some trains leaving A. The Soft deadlock could still occur if those trains waiting at A prevent train at C from completing the 3step loop.

But in this case, it could be that train at B are waiting for materials. trains at A are thus logically waiting for room in the next train stop. and preventing trains at C from piling up behind them despite train condition for train stop A being fulfilled. This is the acual behavior. Here the proposed change would allows all trains at C to complete the loop and you'd have more trains behind each A station than you set the limit to. ( potentially causing train waiting outside stackers or crash if you don't use signals ).

In other word, the proposed change wouldn't function in all soft-deadlock situations but would cause some existing system to break. ( at least from my understanding maybe i've missed someting).


On a side note the same thing can happen if you only have 2 block in your train network, and 2 trains. Then each train is on its own block and there is a soft deadlock. Or let say 5 block and 5 trains. The same logic apply if you have 20 trains and the sum of the limit allowed in your station is also exactly 20.

[Tricorius]

Removing “pending” trains, or whatever the condition is called, would set both limits to 0/1 and allow both trains to proceed to their next station, freeing the deadlock.

Yes it would allow both train to move in the example.

But it is not something you can generalize (imo).

All the trains with "destination full" located at the top of example in the video would be removed from all their stations since they all have the condition for that particular station clear.

This means the game would see 4 or 5 station "open". But in this case sending 4 or 5 trains could cause problems.

This is a good potential point and one reason I posted this in Ideas and Suggestions instead of bugs. There could be unforeseen side-effects.

And if I recall the whole Train Limits feature was added to help reduce the “18 trains race to a single station that just opened” issue.

However, in the video those stations are closed. When they were open the trains were running smoothly since there was a free station available. However, adding another train in the empty branch and enabling all the stations would likely trigger this same condition. And if all five of those trains were empty there could be a rush of trains.

Perhaps a better solution might be for the limits system to check if a train pair are both “waiting to leave” and therefore have valid paths to each other (or basically is there a receiving station for this train that wants to leave which *also* has a train wanting to leave) then, in very certain circumstances schedule an overflow limit (or whatever has to happen behind the scenes).

[asheiduk]

To clarify the "bug":
1. train wants to leave
2. there is no free station available. All destinations report x/x as trainlimit/train reservation.
3.The train has left the station (no "T" signal) yet the station counts the departed train toward the reserved trains in x/x

I'd like to extand 3) with: ("C" signal is still greater than zero)

The tooltip for the "C" signal is: "Number of trains going to this train stop. Includes the stopped train" So even the tooltip suggests that "Read stopped train" and "Read train count" should correlate -- but don't actually.

[Tricorius]

To clarify the "bug":
1. train wants to leave
2. there is no free station available. All destinations report x/x as trainlimit/train reservation.
3.The train has left the station (no "T" signal) yet the station counts the departed train toward the reserved trains in x/x

I'd like to extand 3) with: ("C" signal is still greater than zero)

The tooltip for the "C" signal is: "Number of trains going to this train stop. Includes the stopped train" So even the tooltip suggests that "Read stopped train" and "Read train count" should correlate -- but don't actually.

Excellent point. My understanding is that “C” is basically the first number (x) in the Train Limit: x/y label.