Dynamic high throughput train deliveries

[doxsroxs]

This is as much to tell you what Im building as it is to get some help since I am far from an expert on this.
I have used Siggboys system quite extensively and after messing around with it I now want to build my own system with slightly different design criteria.

Goals

• The main goal is for this system to be scalable to a really high throughput
• Secondary goals include minimal configuration when setting up and when expanding the system

Limitations and design issues

• Current design assumes a fixed size train
• Only full loads will be transported
• Each station is built to handle one type of material
• The system does not scale indefinetly, there is a limit in throughput for each material due to the matching system. In addition, this is somewhat dependent on the number of stations
• Will probably require a large number of combinators for each station (depending on how smart I am... or how much you guys help me )
• Currently need both green and red wire to function, if possible Id like to optimize this but I am not confident I can without reducing system throughput
• This is currently a paper design, I will need to tinker a lot with the details to get this fully functional

Overall system setup and their main functions
Depot and control panel

• Manages bit/adress list
• Stores system settings, train capacity etc.
• Main system clock
• Store, refuel and dispatch unused trains
• Monitor for system health
• System reset for when things are a mess

Provider station

• Broadcasts available resource
• Requests empty train for loading (either when material for 1 trainload is available, or using prefetch to improve throughput)

Requester station

• Match with and request available resource from provider station (including collision detection)
• Dispatch empty trains to Depot
• Possible feature not yet included - dynamic dispatch directly to next station without passing depot (dependant on dynamic refuel solution)

Green wire

• Station registration and removal from the station bit/adress list
• Registering new station IDs

Red Wire

• Match materials for transport

Matching and system throughput
To enable maximum throughput in the system I have decided on a decentralized peer to peer system. In practice this means the logic for matching and sending trains are part of each station.
To speed up matching and increase system throughput I have decided to assign each station a bit in a bit mask. My hope is that this can greatly reduce the selection and matching process.
The combinator uses a 32 bit signed integer. I reserve 1 bit for interrupt and collision detection. This way a large number of stations can signal if they have a material available each tick.
Ill have to experiment a bit with this part to see how many bits I can actually use, it depends on how fast I can decode/encode this. In a best case scenario I should be able to flag the status for 31 stations each tick. (unless Im getting this wrong...)
If more than 31 stations are to be used Ill simply have to alternate between the two or however many are needed. (this enumeration should be based on the system clock)

Here is an example:
Ill use iron and the station is assigned bit 3 (integer value of 4)
Starting in the default state the provider station signals 4 iron on the red wire once it has a full train ready for transport, all these values add up, but by looking at each bit we know the availability at each station.
A requester station that needs iron reads the positive iron signal on the red wire and initiates the decoding and selection process. I hope to keep this as short as possible by selecting the first or last available bit that is flagged.
Once selected the requester station will send an iron interrupt signal using bit 32 (-2147483648). If multiple bit masks are used this will have to be timed correctly using the system clock.
All iron stations in the system will thus see a negative signal on the system and stop transmitting. If two requester stations signal at the same time this will cause an overflow and the signal will not be negative. This will tell both requester stations that a collision has occured. Either dropping the attempt and inducing a delay based on station ID in each station, or handling the collision. Im not sure whats best yet.

Assuming no collision was detected the next tick the requester station will send the bit value for the selected provider station. In this case 4, this will ensure the provider station listens.
The tick after that the requester station will send its own station ID as an iron integer, at the same time the provider station will send its station ID as an iron integer.
The number on the bus is now a combination of both station IDs and the matching process is completed. All iron stations will now go back to the default state and the bus is now ready for the next matching.
My plan is to keep this matching process to a fixed number of ticks so that the bus can go back to normal without a second interrupt signal to start it up again.
In a maximum throughput scenario it should be possible for the next match to be initiated on the tick after this.

While that is going on both stations now have a combined number that they will need to decode. This is a simple process since they both know their own station ID and simply subtract that from the number they received on the bus.
If the number is higher than 0 they have a confirmed match. The provider can dispatch the train and the requester can store in a register that it is to receive a trainload of iron.

Ill just stop here for now, Im already out of time...

[doxsroxs]

Reserved for more stuff later

[Dimanper]

Hi, doxsroxs.
Although i don't play with mods and don't know anything about siggboy's setup, i thought that sharing my expirience of dealing with "Matching and system throughput" might be useful, though i don't know if that information actually fit into Smart Train mod realities.
As i said, i don't play with mods, but i use kind of "smart stations" that send trains only when some conditions are met (resources are available, resources are needed, reserved slot on the station for a train, etc) in my train network on the current map. And from time to time these stations need to decide between each other which one will send train in particular tick or so to that sweet available slot on "requester" station (ex. sharp demand of iron ore in iron smeltery). In fact, since i am using only 2 signals (0 and 1 signals of green wire) for all of my stations in the train network, these decisions happen every time any two (or more) stations happen to reserve slot for their train or stuff like that in the same tick. And here comes the part where i thougth you were trying to achieve something similar. Basically, the whole system works like that: (i replaced 0 and 1 signals, which i use, with A and B for clarity)

Code: Select all

> prerequisite conditions are met => something wants to transmit something into the system
>   check A signal (signal that indicates how many things want to use the system right now)
>   A != 0 => wait for it to be A = 0 (something is using the system right now, it will wait until that something ends)
>   A = 0 => transmit A(1) and B([ID value]) into the system (indication to others that it started to use the system and ID for next steps calculations)
>     A = 1 => do whatever we want to do with the system (only one thing transmitted A(1) last tick, therefore no matching required; we need to transmit A(1) during the time of interaction with the system to keep others from it)
>     A > 1 => ... (matching part) (more than 1 things attempted to use the system)
>       [ID value] > (B / A) => ... (transmitting and returning) (it's all actually done in one "tick-step" of combinators stuff in order to shorten matching time, but there is a lot of simplified for clarity steps here)
>         transmit A(1) and B([ID value]) into the system
>         return to step 3 ((A = 1)/(A >1))
>       [ID value] <= (B / A) =>
>         return to step 2 ((A != 0)/(A = 0))

So basically what we are doing here is rejecting half (on average) of things which tried to interact with the system in the same tick each cycle of comparing [ID value] with (B / A) (sum of all ID's transmitted and sum of all things matching, so B / A = mean of all ID's), placing them all in the "queue" for next matching cycles and picking thing with highest ID.
Ticks used by this system: 1 + ticks needed to interact with the system (for me it's usually +1 (just sending 1 value) - +4 (some calculations and interactions)) + 2 * roundup(log2(N)) (on average; N - amount of things matching)
- almost no tick losses when only 1 thing tries to interact with the system
- relatively fast matching process (for 100 things trying to interact with the system at the same time, their interaction time (on average) = 1440 ticks (24 sec) + 100 * (interaction with the system) ticks)
- use of only two signals per 1 "field of smart things" (dunno how to call it)
- almost unlimited amount of "smart" things per these 2 signals (only limits are: (2^31 - 1) ID's; situations in which a lot of things are trying to interact with the system at the same time transmitting their ID with B([ID value]) can lead to overflow with >2147 ID's with value of 1M on average. Don't use >2147 ID's with 1M values)
- ability to use more of these "fields" of two signals per system, which don't interfere with each other

Also you can use whatever you want instead of ID's in matching conditions. Just make sure to change values in time, don't reach 2^31 - 1 limit and work around situations when same values occur (perhaps replace them with ID's in those matching instances)

In my setups these matching attachments look like this

fts1c.png (560.71 KiB) Viewed 4554 times

fts1.png (547.29 KiB) Viewed 4554 times

fts1b.png (616.36 KiB) Viewed 4554 times

fts2.png (630.86 KiB) Viewed 4554 times

fts2b.png (667 KiB) Viewed 4554 times

fts3.png (479.11 KiB) Viewed 4554 times

fts3b.png (501.44 KiB) Viewed 4554 times

[cid0rz]

Hi, doxsroxs.
Although i don't play with mods and don't know anything about siggboy's setup, i thought that sharing my expirience of dealing with "Matching and system throughput" might be useful, though now if that information actually fit into Smart Train mod realities.
As i said, i don't play with mods, but i use kind of "smart stations" that send trains only when some conditions are met (resources are available, resources are needed, reserved slot on the station for a train, etc) in my train network on the current map. And from time to time these stations need to decide between each other which one will send train in particular tick or so to that sweet available slot on "requester" station (ex. sharp demand of iron ore in iron smeltery). In fact, since i am using only 2 signals (0 and 1 signals of green wire) for all of my stations in the train network, these decisions happen every time any two (or more) stations happen to reserve slot for their train or stuff like that in the same tick. And here comes the part where i thougth you were trying to achieve something similar. Basically, the whole system works like that: (i replaced 0 and 1 signals, which i use, with A and B for clarity)

Code: Select all

> prerequisite conditions are met => something wants to transmit something into the system
>   check A signal (signal that indicates how many things want to use the system right now)
>   A != 0 => wait for it to be A = 0 (something is using the system right now, it will wait until that something ends)
>   A = 0 => transmit A(1) and B([ID value]) into the system (indication to others that it started to use the system and ID for next steps calculations)
>     A = 1 => do whatever we want to do with the system (only one thing transmitted A(1) last tick, therefore no matching required; we need to transmit A(1) during the time of interaction with the system to keep others from it)
>     A > 1 => ... (matching part) (more than 1 things attempted to use the system)
>       [ID value] > (B / A) => ... (transmitting and returning) (it's all actually done in one "tick-step" of combinators stuff in order to shorten matching time, but there is a lot of simplified for clarity steps here)
>         transmit A(1) and B([ID value]) into the system
>         return to step 3 ((A = 1)/(A >1))
>       [ID value] <= (B / A) =>
>         return to step 2 ((A != 0)/(A = 0))

So basically what we are doing here is rejecting half (on average) of things which tried to interact with the system in the same tick each cycle of comparing [ID value] with (B / A) (sum of all ID's transmitted and sum of all things matching, so B / A = mean of all ID's), placing them all in the "queue" for next matching cycles and picking thing with highest ID.
Ticks used by this system: 1 + ticks needed to interact with the system (for me it's usually +1 (just sending 1 value) - +4 (some calculations and interactions)) + 2 * roundup(log2(N)) (on average; N - amount of things matching)
- almost no tick losses when only 1 thing tries to interact with the system
- relatively fast matching process (for 100 things trying to interact with the system at the same time, their interaction time (on average) = 1440 ticks (24 sec) + 100 * (interaction with the system) ticks)
- use of only two signals per 1 "field of smart things" (dunno how to call it)
- almost unlimited amount of "smart" things per these 2 signals (only limits are: (2^31 - 1) ID's; situations in which a lot of things are trying to interact with the system at the same time transmitting their ID with B([ID value]) can lead to overflow with >2147 ID's with value of 1M on average. Don't use >2147 ID's with 1M values)
- ability to use more of these "fields" of two signals per system, which don't interfere with each other

Also you can use whatever you want instead of ID's in matching conditions. Just make sure to change values in time, don't reach 2^31 - 1 limit and work around situations when same values occur (perhaps replace them with ID's in those matching instances)

In my setups these matching attachments look like this

fts1c.png

fts1.png

fts1b.png

fts2.png

fts2b.png

fts3.png

fts3b.png

Hello, I'm very interested to see what we can get here. I also developed some scheduling system but sequential like siggboys one. Regarding your system Dimanper, could you explain with an example how your system works? Without the blueprints only with your description I don't see the whole process

[doxsroxs]

Sorry for the late response, I have been away on vacation in the sun a bit.
I have also run into issues with my communications protocol that might force me to make major changes unless I can work it out.

The whole point of the system here is for it to be modular and expandable. This is to ensure its compatible with mods etc.
My current design forces all signals for requester stations requesting a certain material to use the material signal only.
For example, when requesting iron, use only the iron signal. This way I can get much higher throughput for the whole system.

My current issue is that using the first bit as an interrupt bit works fine except when I have collision where two or more stations try to send an interrupt signal on the same tick.
Both stations will know something is wrong and I can handle that. The problem is that the rest of the stations might not notice this and this might lead to faulty matches.

[cid0rz]

Hey, nice to hear form u, welcome back to your desert planet xD well biters welcome you for sure. I can imagine you get trouble with collisions, since the beginning was the part that made me feel more unsure that it could work. Now I'm programming some simulator for factorio circuits, It is taking a lot of time so I'm rarely in the game if it is not to make experiments or take blueprints for test, but if you want our advice or support just ask ^^

[doxsroxs]

Hey, nice to hear form u, welcome back to your desert planet xD well biters welcome you for sure. I can imagine you get trouble with collisions, since the beginning was the part that made me feel more unsure that it could work. Now I'm programming some simulator for factorio circuits, It is taking a lot of time so I'm rarely in the game if it is not to make experiments or take blueprints for test, but if you want our advice or support just ask ^^

Hahaha, thanks!

After some testing I found out adding bit 32 x 2 will simply discard the bit and yeild a 0 result.
I have been thinking that maybe I can make the system more robust by reserving more bits for collision detection.

For example by using bit 30 as the interrupt bit and using bits 31 and 32 for collision detection.
The problem with that is that I can never scale it enough to be 100% foolproof.

In a situation where you run out of a material and then make it available again you might have 20-30 requesters all trying to request the material at the same time.
If anyone has a good idea for how to solve this, please let me know. Id prefer not to resort to enumeration or something similar since I feel it will lower throughput in a really large system.
Then again, so will communication protocol collisions

[cid0rz]

But then, I know we are stubborn but.. if you start sequencing, maybe is a better idea to start sequencing from the beginning and then go for a sequential system, like my beautiful baby xDD, just kidding But I thought also for example like cabling all the stations for one resource together in green wire for example, then each time this resource is requested, if the station cannot send a train will hand the order to the next and so on, but as you see i ran away from simultaneity and then the throughput is compromised. Anyways for me having reasonable buffers and being able to send multiple trains from multiple providers to the same requester solves quite well the problem for middle throughput setups. The main problem here is, you want to transfer information to multiple entities and they have to comunicate between each other to know what to do and the wire is not able to identify the origin of a signal or separate things sent from different places so there has to be an order, you make it fixed (sequential) or you make it variable, but the complexity of the system is greatly increased and you have certain limitations because it is a game and some things you cannot change ^^.