Friday Facts #260 - New fluid system

[zukivi]

I'm a civil engineer, so this topic was really interesting to read. Since this new method has calculation on intersection and it's iterative, it made me think about the Cross Hardy method for solving a looped pipe network, it is similar in a way. I studied it in uni and it was really popular before the widespread use of computers and calculation, since it was a simple iterative algorithm that converged quickly and it was basically the only way you could design a water supply network "by hand" or with just a calculator.

Developers, have you considered this method before making this new one? Because it seems really similar. Of course you guys don't really care about the head loss that much and I guess the "blockiness" of factorio is not really the same.

[Dry Hairy Tree]

In real life, which is a simulator for Factorio, fluid movement is mostly controlled via gravity. I've run systems irrigating hundreds of acres from a central well, and dozens of grow beds from a central pond. In all instances valves make a huge difference so that the power used to shift water uphill is not wasted and so that distribution is evenly spread.

Shift the water above the system, let gravity do everything else. Flow speed/water pressure is created by funneling into smaller diameter pipes from larger. Gravity will do that too.

Pumps and valves are key, pipes are just the connections between working parts. Pump (1) -> junction (0.5 + 0.5).

The above math depicts a model of one way flow. I see no instance of the game where two way flow is required other than equalizing volume in one section of pipe for aesthetics sake (section = contiguous pipe with no valves and no junctions). If not for visuals, pipes could just carry the numbers from pump (1) to junctions (1/n) or outlets (1/n).

Not sure if that makes any sense. It's how I theoretically map distribution of water for a new system, adjusting valves is how I make the theory (1/n) fit reality.

[Dominik]

Naive question: why not use a very simple chip firing game model?

From a quick look I don't see how that is applicable. Can you elaborate?

[djnono]

While the new proposed system seems better, I feel it will only actually improve performance on edge cases. If you are at a point where you have ups issues, which is the case for nuclear on big setups for exemple, you rarely have long stretch of pipes, mostly lots of intersections.

The sanity would be great, but that's a very high cost.

Now if I could make a suggestion:

95% of the pipe fluid network never intersect with the main update loop entities. I.E., pumpjacks, pumps, pipes, tanks and refineries only impact the other entities because of the electrical grid or logic network (plus an edge case with coal liquefaction), the only entities that impact both networks are assemblers using fluids or chem factories using/producing items.

Why not drop the entire fluid network on its own thread, using the proposed system, and handle the impact on the main update thread with a 1 tick delay ?

Electric usage for brownout conditions should be trivial to handle.

And the impact of the fluid network on the main update would be limited to (if the assy is idle) check if there was enough lube/acid/whatever at the end of last tick fluid iteration to start a new cycle, and pass an array at the end of the update loop to the next iteration of fluid loop to correct values during the first pass of the calculations.

That would add the sanity of the new system with a performance boost, and would be much easier to maintain as it would be mostly isolated form the main update loop

[BlueTemplar]

So, about tainted plumbing systems. The HVAC-simulator Oxygen Not Included has an interesting mechanic for dealing with wrong fluids in a pipe/vent network. A fluid consumer just pulls any fluid/gas from the system, regardless of what it's supposed to receive. When a consumer receives the wrong substance, it takes a small amount of damage and annihilates the fluid. I think it's a neat method. You might consider it.

Oh yeah, and include this mod in vanilla too !
https://gfycat.com/UnacceptableEnchantingKingsnake

[Dominik]

Why not drop the entire fluid network on its own thread, using the proposed system, and handle the impact on the main update thread with a 1 tick delay ?

See, when writing it I totally forgot about the multi-threading. Yes, things like this should be possible and hopefully we will get to it too.

[Dominik]

I'm a civil engineer, so this topic was really interesting to read. Since this new method has calculation on intersection and it's iterative, it made me think about the Cross Hardy method for solving a looped pipe network, it is similar in a way. I studied it in uni and it was really popular before the widespread use of computers and calculation, since it was a simple iterative algorithm that converged quickly and it was basically the only way you could design a water supply network "by hand" or with just a calculator.

Developers, have you considered this method before making this new one? Because it seems really similar. Of course you guys don't really care about the head loss that much and I guess the "blockiness" of factorio is not really the same.

The reason I did not use it is that this is the first time I hear about this algorithm. I will have a closer look later and see if it would be useful but I already suspect that yes. Thanks for the suggestion

[Dominik]

How about using routing system that will be sending packets around fluid network, with contents like: Type of fluid, numerical value and destination ID? It could work like in Simcity or Cities Skylines, or pretty much any computer network but with packets that are traveling slow. I think that it would be possible to visualize it nicely (eg. pipe would fill up more if more packets would travel through that pipe in some period of time) and there is benefit that there are many routing algorithms so there could be one that would fit your needs .

Packets have a critical difference from a fluid - they are quantized. The fluid needs to be able to flow in any quantities, split and combine, be affected by pipe size.

[Illiander42]

One suggestion:

Replace "straight pipes" with "pipes with only one or two connections"

Otherwise diagonal pipelines will cost heaps more than straight ones.

[Lubricus]

I am not that happy with the proposed changes. The biggest problem I have with pipes and fluids is performance and bad information on what is happening.
With lot of junctions and undergrounds I think the knew system will be worse for performance as it have to make two loops per frame. It's rare with throughput issues, it only happens with large nuclear setups and mega-base scale oil processing. So the problem with flow thru junction is more theoretical bad than a problem for the game-play. And unequal distribution is only a problem with pipe networks with loops can hinder throughput, otherwise back-pressure will fix it (if you produce enough it will work and if you produce to little it should anyway stall).

I see throughput loss for longer pipes as an feature not a problem, but I think it can be sacrificed for performance.

flow.png (850.4 KiB) Viewed 8239 times

Some extra info about the fluids would be a much bigger improvement. The graphics with the widows on the pipes is so discrete and i never zoom in enough to see the speed in the animation.

[Dominik]

Maybe I’m taking a seriously simple and naive approach here, but hear me out.

1) Belts are uni-directional.
2) Entities such as Boilers, Refineries and Chem Plants have dedicated input/output locations.

Now, why coukdn’t straight pipes act like belt segments and flow one direction. Underground pipes could be like underground belts with the flow direction changable by rotation hotkey. Pumps would still be useful to increase flow speed or circuit controls.

Since straight pipes would now be uni-directional, they would determine flow direction at junctions much like side loading a belt. I know what you’ll say, “What about the T-intersections everyone uses to input Refineries or Chem Plants?” Simple... a new technology is introduced to make T and 4-way pipes, definitely the T pipes. It would give 3 versions, 1 with inputs both ends and output center (like a Boiler), and the other 2 would input one end and center while outputting other end. There wouldn’t be one that input center and output both ends as it just brings us back to the current split flow issues. The 4-way woukd be 3 inputs and 1 output, but that could be done just by uni-directional pipe line and in a sense “side loading”.

(this also reacts to some other posts) We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc. Having pipes with a direction does not really work though - when you use tanks, you want fluids to go to them at one time and back later.

[Dominik]

One suggestion:

Replace "straight pipes" with "pipes with only one or two connections"

Otherwise diagonal pipelines will cost heaps more than straight ones.

It is so, of course

[Charidan]

Don't wanna sound negative, but thinking about actual setups used in game (see pictures) and seeing how many junction there are and relatively few straight pipe sections, I'm not sure it would really be that much of an optimization to be honest.

Even in the images you link there are pipe segments of about 2 pipes between most junctions, and several longer segments. The segment system would probably provide a x1.5 UPS change, maybe closer to x2.

However, refining isn't the primary case for segment-based pipe reduction. Consider the pumping station around the pumpjacks with several longer segments condensing into a small number of pipes (probably one) that is then piped overland. Currently you have to use underground pipes so there are only two entity updates per ten tiles. In the new system, the entire transit pipe from a pumping station to a distant refinery is a single update. The UPS savings are huge.

Turning pipes into segments is basically zero cost with positive gain. You technically need to spend some time calculating the segments, but you only do that when placing or removing pipes, so during the update tick there is no logic cost at all; you just iterate over the already existing segments the same way you would iterate over the set of all pipes in the old system. It might not do much for nuclear power, but it at least helps for refining a little and has great return on transit pipes.

[Dominik]

A couple thoughts come to mind regarding nuclear as mentioned.
Which pipe system was the bigger performance hog? (fluids vs heat)
If something like this gets implemented for heat pipes, will probably need to have reactors combine for calculations due to the grid layout.

Right now I am not sure about the heat pipes, but I believe that they use a separate logic and are quite optimized. I might be wrong though.

[Dominik]

how does/will it split, like - from top to bottom

1. 1/6
2. 1/6
3. 2/6
4. 1/6
5. 1/6

or 1/5 each?

1. 1/6
2. 1/6
3. 2/6
4. 1/6
5. 1/6

Yes, it should be the 1/3 in the middle and 1/6 for the rest. That is for empty / same low fill pipes though. If there already is some fluid inside some pipe it will affect the splitting appropriately.

In case you are interested a bit deeper in the junction mechanics, there is a section of the FFF that we decided not to publish in the end as it went too deep for most people. It is an example of how the thinking about the junction mechanics work - called "Try it yourself". So, you can still give it a go
https://docs.google.com/document/d/1ocv ... 24WJ5YvCeg

[Stlyau]

Maybe I’m taking a seriously simple and naive approach here, but hear me out.

1) Belts are uni-directional.
2) Entities such as Boilers, Refineries and Chem Plants have dedicated input/output locations.

Now, why coukdn’t straight pipes act like belt segments and flow one direction. Underground pipes could be like underground belts with the flow direction changable by rotation hotkey. Pumps would still be useful to increase flow speed or circuit controls.

Since straight pipes would now be uni-directional, they would determine flow direction at junctions much like side loading a belt. I know what you’ll say, “What about the T-intersections everyone uses to input Refineries or Chem Plants?” Simple... a new technology is introduced to make T and 4-way pipes, definitely the T pipes. It would give 3 versions, 1 with inputs both ends and output center (like a Boiler), and the other 2 would input one end and center while outputting other end. There wouldn’t be one that input center and output both ends as it just brings us back to the current split flow issues. The 4-way woukd be 3 inputs and 1 output, but that could be done just by uni-directional pipe line and in a sense “side loading”.

(this also reacts to some other posts) We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc. Having pipes with a direction does not really work though - when you use tanks, you want fluids to go to them at one time and back later.

But Dominik... can’t the same be said of belts to buffer storage? Wanting the items to flow one direction then later the other? What do we do? We reverse (rotate) the belt. By reversing flow right to left (much like rotating a pump in the middle of a pipeline) we send it back to the tank you mentioned. Yes this may involve rotating multiple pipes but could just as easy make a loop back to the tank, since I assume your example is for when tearing up a build before new design is laid in it’s place.

[Ghoulish]

Interesting FFF! Great write up.

We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc.

Please do add this feature, flow control is a mod which adds these and it truly does improve fluid handling, especially in the later part of the game where the pipework turns in to the proverbial spaghetti. We've all been there - you need to sneak a pipe through a gap but can't because it'll connect to the pipe next to it and different fluids would mix..

[aequitas]

It could also lead to more UPS efficient designs, trying to minimize the number of splits in your pipe network, we know some players really enjoy trying to optimize for different metrics.

Nice mod idea, instead of pollution, making the UPS the driving factor for biter growth. But it would probably be hard to attribute the UPS amount to certain regions as it is now with pollution.

[Garlik30]

While the new proposed system seems better, I feel it will only actually improve performance on edge cases. If you are at a point where you have ups issues, which is the case for nuclear on big setups for exemple, you rarely have long stretch of pipes, mostly lots of intersections.

Hi @Dominik, thanks for your 1st FFF. And obviously, thanks for this rework you have done.
I'm also afraid, as djnono is, that UPS issues are mainly arising in megabases, where intersections are everywhere, even in nicely thought up builds.

The solution you have is I think the best one you have without making the fluid pipes overpowered (electric system solution).

But I'dd like to suggest another aproach to the UPS problem of fluids in the late game. Objects early game are moved mainly with belts, but in megabases/late game, we all know, whatever optimisations you can do, belts will always have more drain on UPS than trains + bots combo. Fluid can be moved by pipes and trains but do not have a usable bot equivalent (lets not talk about barrels ).

Why cant we have a late game solution to moving fluids without pipes? Like bots replace belts? Barrels are not a real solution, as assemblers cant use barrels directly, the add the whole barreling/un-barreling problem, you have to move empty barrels arround,... Nobody likes barrels. Sorry

Why not have a new placable mini fluid tank: one square meter. They would act as requester chests for fluids. This object would be filled by bots (maybe a new kind of bot, specially for moving fluids without needing to use barels). And assemblers could directly pump from them if they are in there fluid inbox position. The new bots would just take the fluids from the big fluid tanks, and move them to the mini fluid tanks. Really, 'just' a requester/provider combo for late game fluid transportation over small distances.

Helicopters can move water IRL: https://i.stack.imgur.com/5oMG8.jpg , Factorio bots should be allowed to do as well

Wether you like my proposition, thanks for reading till here. Love yall, thanks for the great game you give us. Chears!

[hoylemd]

edit: Much better ideas have been proposed. Just cleaning up after myself