Smart Train Deliveries with Combinator Magick [0.13/0.14]

[doxsroxs]

The number of wagons is set in the arithmetic combinator that does the multiplication, right next to the constant combinators that store all the amounts "per wagon". There's another arithmetic combinator that subtracts "zero", you do not need this at the moment, but it will become clear what it's for when I describe the setup for multi-providers. (Leave that combinator alone, let it subtract "zero", it's basically doing nothing at the moment in your setup.)

The demo was created in the latest Factorio release 0.13.17, update your game and it will work.

Thanks for a fast reply! I think I found it myself a few minutes after my post :p
Just to be certain we are talking about the same one, its this one, right?



The others where really easy to find, especially the depot lanes where you used a train signal.
To make it clearer with the number of wagons you could perhaps do the same there and have a constant combinator with wagon signals in it.

Thought I was on the newest build, didnt realize they released a new one yesterday, for some reason I thought I was on .19, when in fact I was on .16...

Just want to check another thing with you, it seems now that I get two trains sent to the same station straight after eachother before the first one is done unloading?

Situation is:
I have less than 50k lead ore in buffer, I also have it set to request when under 100k, so value in constant combinator is -100k.
First train gets sent and 48k is reserved. Its still below 100k in the one comparing in stock materials to constant combinator output, so its ok to send another train.
But before it would wait for station to become free before sending second train, now it sends it straight away.
Is this an intended change or did I find something strange?

I dont mind since it fills buffers faster

[siggboy]

To make it clearer with the number of wagons you could perhaps do the same there and have a constant combinator with wagon signals in it.

Yes, that would indeed be easier. I'll probably do a lot of these clean-up and quality-of-life improvements when everything is stable.

If you add bells and whistles like this then it does increase the complexity and combinator count, so I do not want to do this before we have a really good grasp on what's important and what's not.

First train gets sent and 48k is reserved. Its still below 100k in the one comparing in stock materials to constant combinator output, so its ok to send another train.
But before it would wait for station to become free before sending second train, now it sends it straight away.
Is this an intended change or did I find something strange?

The requester stations can get more than one train at a time. Only the provider stations are "locked down" while there's a train inbound to them.

If you have a requester that is consuming material fast enough, then the "reserved" amount won't be enough to cover the latency, and another train will be sent to that same requester.

This is intended, it's not even a change; it has always worked that way (even in the 0.12 version where I also had the resource accounting for requesters). I think it's a very good feature, and it's only possible because I do not count "trains", but I actually count the amount of material that is incoming to a requester.

[doxsroxs]

To make it clearer with the number of wagons you could perhaps do the same there and have a constant combinator with wagon signals in it.

Yes, that would indeed be easier. I'll probably do a lot of these clean-up and quality-of-life improvements when everything is stable.

If you add bells and whistles like this then it does increase the complexity and combinator count, so I do not want to do this before we have a really good grasp on what's important and what's not.

Seems like you are getting there, has been working really well for us now

First train gets sent and 48k is reserved. Its still below 100k in the one comparing in stock materials to constant combinator output, so its ok to send another train.
But before it would wait for station to become free before sending second train, now it sends it straight away.
Is this an intended change or did I find something strange?

The requester stations can get more than one train at a time. Only the provider stations are "locked down" while there's a train inbound to them.

If you have a requester that is consuming material fast enough, then the "reserved" amount won't be enough to cover the latency, and another train will be sent to that same requester.

This is intended, it's not even a change; it has always worked that way (even in the 0.12 version where I also had the resource accounting for requesters). I think it's a very good feature, and it's only possible because I do not count "trains", but I actually count the amount of material that is incoming to a requester.

Nice, I think its good as well, just didnt know it worked like that

[siggboy]

Seems like you are getting there, has been working really well for us now

Yes, for me, too. I'm having a lot of fun again on my current map (like in 0.12, it was the same feeling when finally everything worked).

I don't know what actually "fixed" the issues we were having before the depot update. Maybe it was just one little thing I've changed when I did the rework. I made a few changes to make it more robust, and just in general checked everything twice to make sure there are no hidden fuckups.

The map I'm playing on had absolutely no disruptions anymore today; the only "bad" things that happened were due to errors on my part (missing wire connections). So it's pretty safe to say that the current version is bug-free and should work as advertised.

The requester stations can get more than one train at a time. Only the provider stations are "locked down" while there's a train inbound to them.

Nice, I think its good as well, just didnt know it worked like that

Well, if you think about it, it would be pretty dumb if the requesters were locked down to one train at a time. Especially since it's not uncommon to have a requester demand a lot of different items (in my main base I'll soon have to add another combinator to the requester station because it's already requesting 10 different resources).

The reason why it's a lot easier to allow many inbound trains to a requester is because the next stop after the requester will always be the depot. Therefore it's not necessary to tell the requester where the trains have to go next. Also, the requester will always simply unload everything that's in the train, so it doesn't have to know which train brings which resources.

All this means that the requester can be pretty "dumb" -- the first version only had 4 combinators at the train station. Adding the "resource accounting" made it a little more complex, but it was an important feature to add.

In the old version without resource accounting, when a requester asked for a resource, all the available trains would jump at it simultaneously. For example, the requester would ask "please bring me iron ore", and then 4 trains with iron ore would arrive at the station 1 minute later. A lot of trains were queueing up, and it jammed the railway. It also made the deliveries very uneven -- on average it didn't matter, but the resource distribution was very unbalanced.

[doxsroxs]

Seems like you are getting there, has been working really well for us now

Yes, for me, too. I'm having a lot of fun again on my current map (like in 0.12, it was the same feeling when finally everything worked).

I don't know what actually "fixed" the issues we were having before the depot update. Maybe it was just one little thing I've changed when I did the rework. I made a few changes to make it more robust, and just in general checked everything twice to make sure there are no hidden fuckups.

The map I'm playing on had absolutely no disruptions anymore today; the only "bad" things that happened were due to errors on my part (missing wire connections). So it's pretty safe to say that the current version is bug-free and should work as advertised.

The requester stations can get more than one train at a time. Only the provider stations are "locked down" while there's a train inbound to them.

Nice, I think its good as well, just didnt know it worked like that

Well, if you think about it, it would be pretty dumb if the requesters were locked down to one train at a time. Especially since it's not uncommon to have a requester demand a lot of different items (in my main base I'll soon have to add another combinator to the requester station because it's already requesting 10 different resources).

The reason why it's a lot easier to allow many inbound trains to a requester is because the next stop after the requester will always be the depot. Therefore it's not necessary to tell the requester where the trains have to go next. Also, the requester will always simply unload everything that's in the train, so it doesn't have to know which train brings which resources.

All this means that the requester can be pretty "dumb" -- the first version only had 4 combinators at the train station. Adding the "resource accounting" made it a little more complex, but it was an important feature to add.

In the old version without resource accounting, when a requester asked for a resource, all the available trains would jump at it simultaneously. For example, the requester would ask "please bring me iron ore", and then 4 trains with iron ore would arrive at the station 1 minute later. A lot of trains were queueing up, and it jammed the railway. It also made the deliveries very uneven -- on average it didn't matter, but the resource distribution was very unbalanced.

I agree, much more fun when you finally get something to work so you can move forward with the plans for world domination!

However, i think I managed to break something again, this is probably not your fault, but just wanted to give you a heads up regarding stability and people like me doing their best to break your stuff.
I was experimenting with my central storage and accidentally connected my not entirely complete contraption (read, "epic mess of wires and combinators") to the stations while they in turn where connected to the red bus.
I actually did not output anything on the bus per se, but I switched my two stations on and off very fast for a few seconds.

I think it killed your depot logic or the other stations logic, everything on the red bus stopped responding and all I see is enumerating O and some other stuff I cant quite identify at the start of the O enumeration

[siggboy]

Can't really say what that is, it totally depends on what you've done before it happened.

I thought about adding some kind of global "reset" switch to the circuit many times, so it's easier to recover from situations like this. It's not that easy to make a really reliable "reset switch" however.

If you develop new things you should do it in a sandbox map for the purpose. All of this was developed in a sandbox map, it would have been way too painful in an actual game.

[doxsroxs]

Dont worry, I have frequent saves since I know Im messing about trying to make the system do something it was not built to do.
I just restored a save and continued working, Ill post a picture once I get it broken working the way I want

[doxsroxs]

EPIC WIN!!!

I have managed to hotwire two stations into doing what I want, I now have a central storage system!

The wiring is a mess and can probably be simplified, but here it is!



Left to do:
Add a check for capacity before requesting material so we dont queue up trains without anywhere to store stuff
Clean up the wiring
Simplify

2 and 3 will most likely never happen since I have tons of other stuff to do, we dont even have titanium yet and coal is running out while we are venting 14k hydrogen and 14k chlorine every minute to keep our aluminium production going.
Remember, we are running bobs mods, numbers are not comparable to vanilla, we currently have lvl 4 speed modules, there are 8 levels available in the mod. With T2 beacons I can place 23-24 beacons around a single oil well.
I assume that will rise even more with a T3 beacon... Basically, its nuts.

Anyway, back to the function, I compare own stations iron levels with whats advertised on the red bus. If higher than own and below 8M we know someone else has iron.
I start a clock with a latch to keep it going since signals on the bus will change when requesting. After 200 ticks I turn off the provider station by fooling it into having 0 material.
After 400 ticks I turn on the requester station that will ask for 48k iron. No matter if the request is matched or not a while later I turn off the request for iron again. (usually it is)
Then a bit later I turn on the provider station by making it see available material again. Finally the clock resets itself and stops after one cycle.

If there is still more material advertised on the red bus, the condition will be true again and the clock starts over. Once its started it just runs its cycle.
One potential issue might be when the network is very large and the depot might not enumerate stations fast enough to make matches. The solution in that case might be to just increase the ticks between each cycle.

As for storage, we currently have a capacity of just over 5 million, but it will be oveer 20 once I get the boxes in place. That should leave us plenty of buffer and no rush to start new mines when old ones are depleted.
Once we get tungsten storage boxes capacity will rise to 84,6 million

Let me know any of you guys are interested in a blueprint string of this mess, Im sure someone could improve it a lot

[siggboy]

If higher than own and below 8M we know someone else has iron.

I think I should explain this "8M" business again (because it does not mean that a resource is not available, it could very well be).

If a requester does want a resource it will output a binary "1" followed by 23 zeroes (= 8388608). So if two requesters request the same resource you'd get a 16777216 signal, and so on.

The providers simply output the precise amount, what they have buffered. These values are not shifted.

It all gets added together, the shifted flags from the requesters and the raw amounts from the providers.

So, the upper 9 bits in the resulting number is the space of the requesters, and the lower 23 bits is the space of the providers -- because requesters shift their output by 23 bits to the left, and providers just output the raw value.

If you think about the binary numbers it's very easy to understand. The large decimal numbers are not helpful at all.

If you look at the red wire bus (while the signal "S=1" is present), you can simply multiply and then divide everything by 512, and that will get rid off the highest 9 bits in the number, therefore it will eliminate the part that is output by the requesters, and the result is the total sum of resources that are provided.

So what you do is:

Red bus into "Green > 9; Output Everything" into "Each * 512" into "Each / 512" -> result is total sum of provided resources at this point.
This number will become zero for a short while when the depot is going through phase 2 and 3 (yellow and green lamp color). Instead of filtering by "Green > 9" you could also filter with "S=1", in fact that might be even better.

(The "Green" signal is a multiple of 10 if we're getting stage 1 polling responses, it will always be "Green = 1" for stage 2 polling responses.)

This value is only valid while the depot is in stage 1 of the polling, i.e. while it's outputting the "S=1" signal to poll all stations simultaneously.

Also, if you only want to know which resources are REQUESTED (disregarding what's provided), then you divide by 8388608, that will shift the requester flags to bit position 1, and the result will be the number of requesters that are currently requesting each resource. You'd put this combinator behind the "Green > 9" filter as well.

Maybe this will at least help you understand, or maybe make a better implementation of your "hack".


Also, let me point out that if you have more than 8,388,607 of any single resource in your providers (total amount buffered per resource), then the entire system will fail spectacularly. In your game this might actually happen at some point. (I guess in an actual game you'd never have 8 million of something actually BUFFERED in the provider stations, but of course if you make a "super provider" with lots and lots of chests connected to it it will be possible, especially with OP mods.)

This limit can be increased, but you'd have to change every requester station.

Let me know any of you guys are interested in a blueprint string of this mess, Im sure someone could improve it a lot

If you post a blueprint I can have a look, maybe suggest another implementation.

Well done anyway, what you did was not easy .

[siggboy]

I found a condition that the depot can get stuck in phase 2 for a while. It will select a resource for delivery for which there is no longer any demand. Then no trains will be sent until there is demand again for that resource. I think this happens when a train is unloading at a requester, and during that time this resource gets picked for the next delivery. Then when the depot actually starts looking for a requester, the demand has dropped below the threshold and it will get stuck, unless there's another requester for the same goods (which is kind of rare).

So it will recover eventually, but it's not good. I'll make a fix, but it could be difficult. I'm not entirely sure yet, as it's kind of a limitation of the current protocol.

Probably I'll need some kind of "retry" hack, which will restart phase 1 if no match can be made in phase 2.

[doxsroxs]

If higher than own and below 8M we know someone else has iron.

I think I should explain this "8M" business again (because it does not mean that a resource is not available, it could very well be).

If a requester does want a resource it will output a binary "1" followed by 23 zeroes (= 8388608). So if two requesters request the same resource you'd get a 16777216 signal, and so on.

The providers simply output the precise amount, what they have buffered. These values are not shifted.

It all gets added together, the shifted flags from the requesters and the raw amounts from the providers.

So, the upper 9 bits in the resulting number is the space of the requesters, and the lower 23 bits is the space of the providers -- because requesters shift their output by 23 bits to the left, and providers just output the raw value.

If you think about the binary numbers it's very easy to understand. The large decimal numbers are not helpful at all.

If you look at the red wire bus (while the signal "S=1" is present), you can simply multiply and then divide everything by 512, and that will get rid off the highest 9 bits in the number, therefore it will eliminate the part that is output by the requesters, and the result is the total sum of resources that are provided.

So what you do is:

Red bus into "Green > 9; Output Everything" into "Each * 512" into "Each / 512" -> result is total sum of provided resources at this point.
This number will become zero for a short while when the depot is going through phase 2 and 3 (yellow and green lamp color). Instead of filtering by "Green > 9" you could also filter with "S=1", in fact that might be even better.

(The "Green" signal is a multiple of 10 if we're getting stage 1 polling responses, it will always be "Green = 1" for stage 2 polling responses.)

This value is only valid while the depot is in stage 1 of the polling, i.e. while it's outputting the "S=1" signal to poll all stations simultaneously.

Also, if you only want to know which resources are REQUESTED (disregarding what's provided), then you divide by 8388608, that will shift the requester flags to bit position 1, and the result will be the number of requesters that are currently requesting each resource. You'd put this combinator behind the "Green > 9" filter as well.

Maybe this will at least help you understand, or maybe make a better implementation of your "hack".


Also, let me point out that if you have more than 8,388,607 of any single resource in your providers (total amount buffered per resource), then the entire system will fail spectacularly. In your game this might actually happen at some point. (I guess in an actual game you'd never have 8 million of something actually BUFFERED in the provider stations, but of course if you make a "super provider" with lots and lots of chests connected to it it will be possible, especially with OP mods.)

This limit can be increased, but you'd have to change every requester station.

Let me know any of you guys are interested in a blueprint string of this mess, Im sure someone could improve it a lot

If you post a blueprint I can have a look, maybe suggest another implementation.

Well done anyway, what you did was not easy .

Thanks, thats some really good and detailed info there. It helps me understand better.
However, the reason I check to make sure I dont have a signal over 8M is to make sure the central storage is never prioritized over a factory request for materials. Its better that it stays in provider mode in that case.
I also read about the 8M limit for provided/buffered resources so the total amount in central storage is never transmitted over the bus. The provider station part will only know the amount in its own chests.

When it comes to OP mods I agree Bob's mods are a bit extreme. We didnt know that when we started, we started this game with just hours testing in vanilla and then jumped in here.
We also agreed not to use god modules or raw modules since they are to OP with huge bonuses and no drawbacks. Again, we found out when we researched them... so its a gentlemens agreement.
We do use normal speed mods etc. So we have huge power requirements, alu production is around 30MW. New refinery will be 100+MW.

That said it does give unique challenges to play with high throughput builds. I am currently debating increasing the number of railcars to increase throughput since loading times at stations are to long when planning for extreme throughput factories and higher tech railcars will only increase loading times.
Belts are also a problem, latest coal mine is fed by 8 almost 100% saturated blue belts in turn fed by T2 miners with full speed module layout doing about 16k coal/min.

Have you done any throughput experiments? I know I can get high with multiple stations. Im sure it can be extreme, but question is, am I worrying for nothing? :p

Ill get a blueprint for you tonight, but be warned, I took a hammer approach to this and just made a crude contraption to get this working. Like modding a pc with an axe, not a balanced system like yours

[siggboy]

I think it killed your depot logic or the other stations logic, everything on the red bus stopped responding and all I see is enumerating O and some other stuff I cant quite identify at the start of the O enumeration

This might not have been your fault after all, it could have been an instance of that issue I mentioned one post earlier. When the depot gets stuck in phase 2, then you will see the polling on the bus, which then lasts a long time. Usually phase 2 is very short, so that was probably confusing you.

In any case, I will make a workaround for this, because it actually happens rather often on a busy train network and it can bring the entire system to a halt.

[siggboy]

Have you done any throughput experiments? I know I can get high with multiple stations. Im sure it can be extreme, but question is, am I worrying for nothing? :p

In 0.12 I made a distributed base, where all the products are shipped with trains (so no belting of plates directly into consumers, and no feeding of chips into rocket factory etc.).

In that base I did have trains with 4 wagons and 2 locomotives, and no mods that enhanced the Vanilla entities. The only mod that changed gameplay (except SmartTrains) was Nucular, for energy generation.

The final version of the map produced a little more than 1 rocket every 3 minutes. At that point I did have 20 trains on the map, total. I did have no buffers of anything, only the minimum amount that you need at trains stations.

The depot had 7 lanes, but that was complete overkill, probably 2 or 3 lanes would have been enough.

I'm pretty sure I could have scaled it to 1 rocket per minute. Maybe I'll even resurrect that map one day and try if it's possible, just to see how many trains are necessary.

In my current game I only have 2-wagon trains, so it might become a problem to scale that. I will see. It's also a constrained game because I build everything on artificial islands.

I will send you a save of that game so you can have a look. At the moment I'm not producing any rockets yet, didn't even have my first launch. When the rockets are flying I'll send you a save and you can play around with it, and see how I make things work (or fail :).

[doe]

Any chance someone could make a youtube video, setting up the depot with all combinators etc, then a provider depot and then requester depot, basically step by step how to get this working. Seems like i am too think as pig shit to understand how to get this to work or whats happening lol.

[ray4ever]

I like the idea of your concept! And even though it seems to work fine using standard combinators, I was thinking if it would be easier to use of it would be a mod.

So your blueprints would be reduced to a single entity which would be easy to produce and place. This would make it easier to use and would be less prone to errors.

[doe]

I like the idea of your concept! And even though it seems to work fine using standard combinators, I was thinking if it would be easier to use of it would be a mod.

So your blueprints would be reduced to a single entity which would be easy to produce and place. This would make it easier to use and would be less prone to errors.

Wandered if you could possibly help?

I got the depot smartstop blueprint, and placed it down with all the items needed and used robots to construct it, but i keep getting "missing item, the box on the smart station is missing, dont know why robots cant place it?

[siggboy]

Any chance someone could make a youtube video, setting up the depot with all combinators etc, then a provider depot and then requester depot, basically step by step how to get this working.

A video would be a great idea to show the project off to others, however I do not think that more people would actually use this if I made a video.

If you want to use this, you have to put in some work -- just look at how long my conversation with doxsroxs is now, it's the last 8 pages of this thread. This is the time we needed to figure out things for him and also work on eliminating problems from the setup.

The video would be a great demo, but it's not necessary to use this yourself if you want to. This is not a mod, it's not possible to make it easy to use.

I like the idea of your concept! And even though it seems to work fine using standard combinators, I was thinking if it would be easier to use of it would be a mod.

If this was a mod, then it would be a LUA program doing all those things that the combinator setup does right now. Which is not the plan here. This is a combinator project that I've started months ago because I knew it would be possible to do something like this with combinators, and I wanted to see if I can do it.

I know that I could more or less easily extend SmartTrains to do the same thing that my circuit does, but it would not be a lot of fun nor a good challenge. The result would be an overpowered mod that solves a critical problem that the player is supposed to solve.

Just look at all the complicated things for 0.12 to make a steam power plant a backup generator, if your primary power source is solar. This can easily be done with a mod that outputs the necessary information, but that's not interesting.

So your blueprints would be reduced to a single entity which would be easy to produce and place.

I agree with you 100% that these complex combinator circuits take too much space, and they are too difficult to debug, to understand and to refactor. Several players have asked the devs to make a combinator blackbox which would lift some of these limitations, but it's very unlikely that we'll actually get that in the game.

I got the depot smartstop blueprint, and placed it down with all the items needed and used robots to construct it, but i keep getting "missing item, the box on the smart station is missing, dont know why robots cant place it?

If you have a smart train stop, then this box (and the lamp) that belong to the train stop will appear automatically when the bot places the actual train stop.

If that doesn't work for you, then you have some other mod that breaks the placing of the train station, but it would have to be some cheating mod I guess.

Maybe you're just missing a constant combinator in your inventory. It looks the same as the "box" on the train stop, but it's actually a different item.

[doe]

Any chance someone could make a youtube video, setting up the depot with all combinators etc, then a provider depot and then requester depot, basically step by step how to get this working.

A video would be a great idea to show the project off to others, however I do not think that more people would actually use this if I made a video.

If you want to use this, you have to put in some work -- just look at how long my conversation with doxsroxs is now, it's the last 8 pages of this thread. This is the time we needed to figure out things for him and also work on eliminating problems from the setup.

The video would be a great demo, but it's not necessary to use this yourself if you want to. This is not a mod, it's not possible to make it easy to use.

I got the depot smartstop blueprint, and placed it down with all the items needed and used robots to construct it, but i keep getting "missing item, the box on the smart station is missing, dont know why robots cant place it?

If you have a smart train stop, then this box (and the lamp) that belong to the train stop will appear automatically when the bot places the actual train stop.

If that doesn't work for you, then you have some other mod that breaks the placing of the train station, but it would have to be some cheating mod I guess.

Maybe you're just missing a constant combinator in your inventory. It looks the same as the "box" on the train stop, but it's actually a different item.

Thanks for the response, much appreciated. Yeah you are right, i just need to take the time to learn understand it and stop being so lazy lol.
Will try out missing comb and re edit this post if that was indeed my problem.

thanks again, oh and love the work you put into this.

[siggboy]

Thanks for the response, much appreciated. Yeah you are right, i just need to take the time to learn understand it and stop being so lazy lol.
Will try out missing comb and re edit this post if that was indeed my problem.
thanks again, oh and love the work you put into this.

No problem. I love when others try out my build and give me feedback.

I'll be happy to answer any questions.

If you want to use it in a real game, don't make things too complicated right away. Start with easy configurations, such as delivering ore to smelting, before trying more advanced things, such as shipping intermediates or oil barrels.

[doe]

Thanks for the response, much appreciated. Yeah you are right, i just need to take the time to learn understand it and stop being so lazy lol.
Will try out missing comb and re edit this post if that was indeed my problem.
thanks again, oh and love the work you put into this.

No problem. I love when others try out my build and give me feedback.

I'll be happy to answer any questions.

If you want to use it in a real game, don't make things too complicated right away. Start with easy configurations, such as delivering ore to smelting, before trying more advanced things, such as shipping intermediates or oil barrels.

Thanks, will take the advice.

Regarding the smart stop, i think i sort of see what the problem is, with your smartstop depot blueprint, when i place it down with bots, i am missing combi at the train stop, as there is 2 lights also with it ( 1 is a combi but disguised as a light which i find weird), but when i build a smart stop or bot puts 1 down without blueprint, i get 1 light and 1 combi with smartstop and works fine. I thought it might of been the version of smarttrains but i got the latest version.

The smartstop on left is what came with your blueprint, it has 2 lights and a combi, but cant seem to place the combi, smart stop on right has 1 combi and 1 light bots were able to place it np, any idea?