Train path finding in congested networks

Don't know how to use a machine? Looking for efficient setups? Stuck in a mission?
Post Reply
iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Train path finding in congested networks

Post by iceman_1212 »

On sonie2k's stream, he was doing some testing on trains on a congested network. Surprisingly, there were two crashes of trains that were on automated schedule (not manual control) on signalled track. What could have caused this? I've read that trains may sometimes pick paths with red signals if no clear paths are available but I have not heard of trains disobeying signals...

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Train path finding in congested networks

Post by Engimage »

Signalled network sounds too generic. One might place signals wrong or forget to place some. Also circuit networks effecting some signals might also effect this.
I have yet to see a train crash on a correctly set up networks.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

So I experienced something similar in my current save and I suspect that we may want to move this post to the bugs/technical section.

The situation was that I was laying track for final base - track which is totally independent of and ~2k tiles away from my starter base. I was experiencing the "track lag" which was addressed in one of the recent FFFs. I also received a message that one of my locos on the rail network for my starting base no longer had a path. Given I'm not playing with biters and the fact that I hadn't gone to that base in a few hours, I thought it was a fluke and ignored it.

A bit later I decided to investigate the starter base's trains after seeing in the production tab that nothing was being produced. It turns out that the loco (about which I received the no path error earlier) had in fact switched from automatic to manual at a particularly busy intersection and had lost a wagon as a train had crashed into it from behind.

It's a point of track where a lot of trains are merging onto one. I suspect that the loco in question went into manual mode on the busy track at a time when it should have respected the merging train's block reservation - and which it would have done had it not gotten switched from automatic to manual.

As reference, I'm using RSO to tweak the resources to be extremely rich and I've let the world run overnight for 4-5 nights now with no issues re: deadlock. It was only when I began aggressively expanding track (which was in fact unconnected) that I experienced this issue.

Am happy to provide a save and my list of mods if helpful for diagnosing purposes. Please let me know.

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Train path finding in congested networks

Post by Engimage »

Train switches to manual mode if it experiences a crash.
The question is why the crash has occured.
So if it happened during laying tracks which forces train to re-path it leads me to an assumption that trains could change their paths and others could not react to signal change in time. In theory this should not happen as it is as all signals work pretty predictively and new path could not be layed out through busy track. Not knowing how it works exactly I would theorize that TWO trains changed their paths at a time and signals were recalculated AFTER repath happened. In this situation trains could not break in time reacting to signal change and could crash.
If something like that has happened it is definitely a bug. To prevent it signals should be recalculated after EVERY individual train repath happens.
And this will happen even rarer in 0.15 looking at the new repath logic.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

Just happened again and this particular example should be easy to grasp as it's all on one screen and the crash took place at slow speeds so no wagons/locos were destroyed. As you can see, the entire area between the stacker and the unloading bays is one block. It was running fine for a few hours until I added a lot of track (to an unconnected network).

Image

Here is my save file as well as all of my mods, if helpful. My mods that affect trains are Fat Controller, FARL (which shouldn't be relevant for this situation), Ore color (changes loco color to match contents), Wagon color (changes wagon color to show % full).

https://www.dropbox.com/sh/h8fuyh4inuu5 ... IcZWa?dl=0
Last edited by iceman_1212 on Thu Apr 06, 2017 1:20 pm, edited 2 times in total.

Frightning
Filter Inserter
Filter Inserter
Posts: 807
Joined: Fri Apr 29, 2016 5:27 pm
Contact:

Re: Train path finding in congested networks

Post by Frightning »

iceman_1212 wrote:Just happened again and this particular example should be easy to grasp as it's all on one screen and the crash took place at slow speeds so now wagons/locos were destroyed. As you can see, the entire area between the stacker and the unloading bays is one block. It was running fine for a few hours until I added a lot of track (to an unconnected network).

Image
Nice catch, definitely shouldn't have happened, need to have Rseding91 and others at Wube take a look at this (would be especially handy if you have a save from before this happened as well as after the fact so they can test things and analyze what happened).

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Train path finding in congested networks

Post by Engimage »

iceman_1212 wrote:Just happened again and this particular example should be easy to grasp as it's all on one screen and the crash took place at slow speeds so no wagons/locos were destroyed. As you can see, the entire area between the stacker and the unloading bays is one block. It was running fine for a few hours until I added a lot of track (to an unconnected network).

Image
This definitely should not have happened.
However this might be an edge case. Take note of how close exit signal is from the train on the unloading station. It would clearly work more stable if you could make your station 1 tile longer to prevent this edge case from happening. I think in this case the block is accounted as busy but the exiting train can't actually react to that signal as it is say 1 pixel past it or something like that. On the picture we can see that southern train (moving from the stacker) has advanced like 6 wagons before crash and one going from the station has advanced 4 wagons which means it started moving when stacker one was already on the block which confirms my theory.

Also for future reference it is much better practice to exit the station from the opposite side from the entrance even for double headed trains. This would mean exiting trains will not block incoming ones effectively increasing station throughput by alot.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

PacifyerGrey wrote:It would clearly work more stable if you could make your station 1 tile longer to prevent this edge case from happening.
Can you elaborate pls? I see that the signal would encapsulate the entirety of the train if the station were 1-2 tiles longer, but I also see that the coal train (black locos) was pathing into a fully occupied station and not the one from which the exiting train had just left. Also, given signal logic is based on train presence in block, I don't see how my admittedly poor station sizing resulted in any kind of instability.

That said, agreed that station size could be bigger and re: exiting from the other end - I guess I can put two temp stations behind the unloading bays that the trains can use to reverse.

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Train path finding in congested networks

Post by Engimage »

iceman_1212 wrote:but I also see that the coal train (black locos) was pathing into a fully occupied station and not the one from which the exiting train had just left
I did not actually notice this.
However I did notice how many mods you have installed :) Not sure if this could actually impact your situation but it surely could. I see at least FAT Controller which is clearly train related.
Not sure if something could really break due to train re-pathing in this case.

As for where the train was heading - pathfinding could definitely lead a train to that station as you can't tell for sure if the station is busy or empty - as the train might be waiting for signal leaving the station and not actually unloading at station at this point - as the station length is actually too exact.

This should be definitely examined by devs cause something did actually force trains to ignore signals.

But I am almost sure that if the station would be at least 1 tile longer before the signal - it would work much smoother.

I would also suggest you attach this save to the thread so devs can actually look at the situation closer.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

PacifyerGrey wrote:I would also suggest you attach this save to the thread so devs can actually look at the situation closer.
Good idea. I've updated my previous post to include a link to a dropbox folder that has both my save file as well as my mod list.

aaargha
Filter Inserter
Filter Inserter
Posts: 329
Joined: Wed Dec 07, 2016 8:35 am
Contact:

Re: Train path finding in congested networks

Post by aaargha »

This might be related to some bugs that exists with signals/stations close to curved rails, there are a few different ones. If you want to help the devs with this you should probably file a report on the bug report forum (or add to an existing one) and try to make a vanilla reproduction. At the moment mod interference seems pretty likely to me at least.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

aaargha wrote:This might be related to some bugs that exists with signals/stations close to curved rails, there are a few different ones.
Can you elaborate pls? Is there a relevant thread that discusses these?
aaargha wrote:If you want to help the devs with this you should probably file a report on the bug report forum (or add to an existing one) and try to make a vanilla reproduction.
The moderators of this forum are generally proactive about moving threads to the appropriate topic and I made the suggestion early on in this thread. At this point, I'm going to wait to hear from either mods/devs re: appropriate next steps. This may well be an issue related to the recalc of all paths upon placing/removing a piece of track (which was addressed in the last FFF).

Re: mods exacerbating the issue - I doubt it as I experienced the same issue without FAT Controller and FARL;also, I believe sonie2k was running pure vanilla when he experienced it on his stream two weekends ago.

aaargha
Filter Inserter
Filter Inserter
Posts: 329
Joined: Wed Dec 07, 2016 8:35 am
Contact:

Re: Train path finding in congested networks

Post by aaargha »

I don't know of any thread dedicated to discussing these in particular, but I've not really looked for one either, just made sure I'm not reporting duplicates. The ones I know of are the ones I've found/been involved with: signals 1 2 and stations.

Yeah, from what you are describing, especially with sonie2ks assumed vanilla run, it sure sounds and looks like a bug. I'm just not sure how useful this save is, considering the mod list (there are more that definitely/sure sound like they change trains), which is why I'd encourage a vanilla reproduction (see mod replies to signals 1). It's also easier to debug without all the extra noise.

Also -if you haven't already- take a look in this thread if there is any additional info that could provide that could be helpful to the devs.

Loewchen
Global Moderator
Global Moderator
Posts: 8285
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: Train path finding in congested networks

Post by Loewchen »

The signal bugs already reported are (with one exemption) all only visually, meaning trains are still not able to collide with one another caused by those issues. The fact that you witnessed several collisions within hours makes me sceptical if this is actually a vanilla issue. The best way to go forward would be a save file with a collision soon afterwards.

aaargha
Filter Inserter
Filter Inserter
Posts: 329
Joined: Wed Dec 07, 2016 8:35 am
Contact:

Re: Train path finding in congested networks

Post by aaargha »

Found a vanilla reproduction. It seems to be caused by the global repath/disruption that triggers when building rails. To make it somewhat reliable we're using the "blueprint of death" to build lots of rails and kill UPS.

Version: 14.22

To use reproduction save:
Change the requester chests containing rails (marked with orange in the setup picture) to storage, this starts the construction.
Once UPS plummets turn off the constant combinator (purple) to launch the trains.
Enjoy the slow motion crash.

It's not 100% reliable and results may differ a bit from run to run, but it's clearly not working as it should.
Attachments
Setup explanation
Setup explanation
setup.png (4.56 MiB) Viewed 5869 times
Example result
Example result
train ignore signal.png (4.78 MiB) Viewed 5869 times
IGNORE SIGNAL TEST.zip
(3.25 MiB) Downloaded 59 times

Engimage
Smart Inserter
Smart Inserter
Posts: 1067
Joined: Wed Jun 29, 2016 10:02 am
Contact:

Re: Train path finding in congested networks

Post by Engimage »

aaargha wrote:Found a vanilla reproduction. It seems to be caused by the global repath/disruption that triggers when building rails. To make it somewhat reliable we're using the "blueprint of death" to build lots of rails and kill UPS.

Version: 14.22

To use reproduction save:
Change the requester chests containing rails (marked with orange in the setup picture) to storage, this starts the construction.
Once UPS plummets turn off the constant combinator (purple) to launch the trains.
Enjoy the slow motion crash.

It's not 100% reliable and results may differ a bit from run to run, but it's clearly not working as it should.
Good job providing a reproduction test! Please file a bug report in the dedicated subforum if you have not done this yet
viewforum.php?f=7

I do understand that this issue will be much less common in 0.15 due to changes in path recalculation triggers but it is still definitely a bug.

aaargha
Filter Inserter
Filter Inserter
Posts: 329
Joined: Wed Dec 07, 2016 8:35 am
Contact:

Re: Train path finding in congested networks

Post by aaargha »

Report: viewtopic.php?f=7&t=44108

I'm not sure if it's the recalculation that does it or if it's some other kind of disruption related to building rails. It does seem to be another version of the path-finding algorithm than the one usually used though, it does not, as far as I can tell, always reach the same conclusions as the regular (used at red signals) one. Or it may be that it does not affect all trains, haven't found a good way to test that.

iceman_1212
Filter Inserter
Filter Inserter
Posts: 256
Joined: Wed Aug 17, 2016 9:49 am
Contact:

Re: Train path finding in congested networks

Post by iceman_1212 »

Very nice, I see that the "blueprint of death" which I was using in my attempts at reproducing in vanilla were clearly not deadly enough :lol:

Aeternus
Filter Inserter
Filter Inserter
Posts: 835
Joined: Wed Mar 29, 2017 2:10 am
Contact:

Re: Train path finding in congested networks

Post by Aeternus »

I've seen something similar happen that might be related to this - I was doing some work with trying to make a round-robin switching for loading docks by using Combinators in a simple SR Latch configuration (I have an ore transfer dock and want all 4 tracks to be used evenly). This works beautifully, however the latch when first layed down is uninitialized, which means it starts to oscillate rapidly - activating and deactivating every tick. I had it accidentally set up to already close the track signal, so the signal to close the track cycled 60 times per second between on and off. A train didn't care and barreled right through the signal, and me, at full speed. I had to load a save :(

My guesstimate is that the open/closed signals are not recalculated immediately, and definately not every "tick", and this delay is the problem. So a very slow moving train that has permission to cross a red signal (just leaving the station) might take a short while to enter the block and occupy it. If not fast enough, the block occupation is recalculated (because of new rail being built?) and with no train found, the signal is set to green, so a second train enters the block. The first train, already having permission to enter the block, picks up some speed until the collision occurs.

aaargha
Filter Inserter
Filter Inserter
Posts: 329
Joined: Wed Dec 07, 2016 8:35 am
Contact:

Re: Train path finding in congested networks

Post by aaargha »

Aeternus wrote:I had it accidentally set up to already close the track signal, so the signal to close the track cycled 60 times per second between on and off. A train didn't care and barreled right through the signal, and me, at full speed. I had to load a save :(
Unfortunately that does sound like intended behaviour, unless I'm misunderstanding. If the signal turns green when the train is on approach, even for one tick, it is enough for the train to reserve the block (signal goes yellow). Once the signal has turned yellow the train does no longer care that you try to close it with the circuit network.
Aeternus wrote:My guesstimate is that the open/closed signals are not recalculated immediately, and definitely not every "tick", and this delay is the problem. So a very slow moving train that has permission to cross a red signal (just leaving the station) might take a short while to enter the block and occupy it. If not fast enough, the block occupation is recalculated (because of new rail being built?) and with no train found, the signal is set to green, so a second train enters the block. The first train, already having permission to enter the block, picks up some speed until the collision occurs.
Normally a block is occupied when there is a train in it or one of the input signals goes yellow, the block is reserved. As far as I can tell this is still the case even in my test save, it's just that the trains don't care any more. I tried placing a rail car by the station (signals will always be red) and they still blew right past them at 150+ km/h, even if they recognised that they should stop at the signals (stop point changed). I think it's the rail building that does it, otherwise these issues should be much more common. But I dunno, we might see?

Post Reply

Return to “Gameplay Help”