Friday Facts #260 - New fluid system

[mrvn]

Next none of the sources or drains are constant. They produce or consume a set volume in a single tick and then pause. So there never is balance point to reach.

I'm not sure that assumption is necessarily valid. Assembler progress each tick is already dependent on the amount of power satisfaction they get; fluid input could probably be adapted to do the same. I don't know if that'd be worth it for realism's sake, but I do think it'd be worth it if it helps fluid system performance.

I do think interdependencies like that should have a tick delay if at all possible to decouple the various aspects of the simulation.

doubtful that would make sense.

[meganothing]

But fluid input > output should ideally not be enough to ensure all outputs are filled. And as bad as the current system is, it does that right at least.

This is not the point.

You are right - putting more in than you need is not enough to ensure satisfaction at the output.

My point is - fixing that is very simple (add more pipes), so every time you will end up with pipes 100% full and all outputs satisfied.

I don't see why do we need realistic (and possibly buggy) fluid handling just to be occasionally prompted to add more pipes.

You could say the same about the belt network. If a single belt can't deliver enough upgrade it or add a parallel belt, i.e. use resources or space for more throughput. And if you don't have enough space and can't upgrade, you have to redesign. Or if the distance is too long you change plans and put the stuff on trains instead.

So yes, you can have fun with a more complex pipe simulation. And especially the throughput loss over long distances is something that neither the belt nor the electricity network have and so distinguishes the pipe network from the others and makes handling a pipe network different.

[Matthias_Wlkp]

My point is - fixing that is very simple (add more pipes), so every time you will end up with pipes 100% full and all outputs satisfied.

You could say the same about the belt network. If a single belt can't deliver enough upgrade it or add a parallel belt, i.e. use resources or space for more throughput. And if you don't have enough space and can't upgrade, you have to redesign. Or if the distance is too long you change plans and put the stuff on trains instead.

You kind of prove my point: more pipes is way less fun than what you describe with the challenges you face with belts and trains (I would also add bots to that).

Pipes add enough complexity as they are and don't need to be complicated.

It would be fun if more realistic pipes added more choice in building pipe systems - but it won't. Regardless of how sophisticated the pipe realism is, the overall solution will look almost the same (quantity of pipes might differ).

[meganothing]

My point is - fixing that is very simple (add more pipes), so every time you will end up with pipes 100% full and all outputs satisfied.

You could say the same about the belt network. If a single belt can't deliver enough upgrade it or add a parallel belt, i.e. use resources or space for more throughput. And if you don't have enough space and can't upgrade, you have to redesign. Or if the distance is too long you change plans and put the stuff on trains instead.

You kind of prove my point: more pipes is way less fun than what you describe with the challenges you face with belts and trains (I would also add bots to that).

Pipes add enough complexity as they are and don't need to be complicated.

It would be fun if more realistic pipes added more choice in building pipe systems - but it won't. Regardless of how sophisticated the pipe realism is, the overall solution will look almost the same (quantity of pipes might differ).

Wait a moment. If I say it is the same with belts you first have to show me where they differ before you can conclude belts are more fun . Well, their ability to transport different things in parallel is a huge fun-creator but the szenarios I was talking about above would happen with belts AND pipes

In my momentary factory I actually not only transport raw oil with trains but also sulphuric acid. And the sulphuric acid is even transported in barrels (although I must admit I did it just for the fun ). With the loss over distance and the pipes being obstacles you need trains even more for fluids than the solid stuff.

[adam_bise]

I sure am glad the community has people like TheYeast and Pyroathiest to help the devs out in understanding pipes and pressure realistically. As for Pyroathiest's model, If doing it right means we lose the little windows on pipes, big deal! I'm sure everyone just wants the system to work in a predictable and efficient manner. Besides, the windows don't necessarily need to represent level, just showing what's inside is fine.

[raidho36]

https://www.factorio.com/blog/post/fff-260

I'm sort of baffled that you didn't get it right in the first place. About 15 years ago I was making a gamemaker project entirely centered around that very same fluid transmission mechanic - equalizing levels in neighbouring vessels. My game wasnt grid based and you could lay pipes at any angle, so you could've built extremely complex meshes with 20-way intersections (there was no limit for that), and all of that worked flawlessly. If i remember it correctly, I made it work right on the first try, on account of how simple the math and logic were.

1) compute average level for all connected vessels, central vessel including
2) for each connected vessel, compute maximum level change towards the average, taking throughput limit into account
3) using above values, compute total flow though the central vessel, change its level by that amount, change connected vessels' levels by their own flow values

Using a technique alike to double buffering makes processing order irrelevant, too.

The new approach is said to do directional pipe sorting. But wouldn't simply considering directly adjacent pipes in any order be sufficient?

[mrudat]

What is described here is the difference between an implicit (solve it all with a big equation system) and an explicit (small local updates) solution of the system. Implicit sounds good, because you get the whole solution at once, and its globally correct - but i believe its not the right decision here.

First: Implicit it superior for steady state solution. Thats why all those who propose it like to pretend that factorio pipes are mostly steady state. I do not believe that.
Second: The big drawback of explicit is the small timestep size. This does not really matter here, because you need to do small timesteps anyway.
As already estimated: By using any implicit algorithm, equation systems of more than 10000 unknowns can easily arise. This does not sound a lot, but if you try so solve it every step, it really hurts.

Finally: Whatever system gets chosen, one of my main concerns is again: Better opportunity for modding improved pipes. Several have mentioned Bobs/Angels and how the fluid system gets a nightmare there. This is mainly due to the fact, that none of the dozens of new pipes introduced there is really good. Given the fact that bob is an excellend modder I conclude that there are just no good parameters available for tuning right now. Please have a look at this.

I believe that, based on comments by people who know their stuff IRL, it is implied (by the fact that we're using pipes) that factorio's pipes are steady-state (in the sense that pressurised pipes are always full of liquid when in use, as opposed to aqueducts which are open to the air, and thus fluid levels can change).

I suspect that fluid usage is steady-state enough for accurate simulation in bulk, rather than needing to track instantaneous pressure in each pipe segment; especially if we also retain the built-in buffer as part of each crafting machine (I suspect that this can't be removed without major rework); fluid flows into/out of the pipe system into these buffers; fluid flow into/out of the buffer is easily calculable as an average flow.

It appears to be feasible (if perhaps somewhat computationally expensive) to perform a static analysis to determine maximum flow rates through an arbitrary pipe system; provided you don't exceed maximum flow rate, logically incompressible fluids teleport from source -> sink; similar to how a fully compressed belt might operate; there's no room to stash extra items, and the pipe systems doesn't allow for gaps (it stops feeding you fluid in that case).

As for modding better pipes; given that discussion has been involving IRL fluid mechanics, that's more-or-less trivial. Vanilla pipes are (visually) a 1m diameter pipe; no idea about if in-game fluid flow is anywhere in the right ballpark for what can be transported down a 1m OD cast-iron pipe IRL. All other things being equal, a bigger pipe allows for more fluid flow. A stronger pipe (eg. steel/titanium/...?) allows for higher pressure, allowing more fluid flow, though you might need to also introduce a higher pressure pump to make use of that.


As an aside, how about petroleum gas? Is that not a compressible fluid, and thus behaves distinctly differently, I believe?

[daemonworks]

I might be missing something, but why wouldn't working it kind of similarly to the electrical network do what you want?

So, a cude oil pipe, produces X amount which is distributed to Y amount of refineries. just do an even split... but you cap the X based on the desired pipe capacity. If you want a pipe to carry only 100, then even if you're producing 200 then at most each refinery will get 20. Could have some logic that checks if pipes are doubled to improve throughput, or a larger type of pipe could be added that can't be connected directly to almost anything else, requiring a pump or something to move fluids between the small pipes and the large pipes, with the large pipes being their own separate "network" that has a larger limit.

Also, just as there are multiple speeds of belts, there /should/ be better ways to move fluids around as you tech up.

[Sirveri]

I might be missing something, but why wouldn't working it kind of similarly to the electrical network do what you want?

So, a cude oil pipe, produces X amount which is distributed to Y amount of refineries. just do an even split... but you cap the X based on the desired pipe capacity. If you want a pipe to carry only 100, then even if you're producing 200 then at most each refinery will get 20. Could have some logic that checks if pipes are doubled to improve throughput, or a larger type of pipe could be added that can't be connected directly to almost anything else, requiring a pump or something to move fluids between the small pipes and the large pipes, with the large pipes being their own separate "network" that has a larger limit.

Also, just as there are multiple speeds of belts, there /should/ be better ways to move fluids around as you tech up.

I was just going to suggest this. Take your volumetric flow rate, divide that across all end users, if excess fill the pipe network volume (% indicates level height displayed in the section).

If we wanted to get complicated we'd start talking about pump head, output curves, and a bunch of other fun stuff. But for this game that's actually close enough. Though they really should add check valves... Especially for tanks, since those typically have segregated input and output lines. Lord knows I had to pump forward to aft enough times and then blow it all overboard.

[meganothing]

Could have some logic that checks if pipes are doubled to improve throughput,

That is easily written but hard to implement correctly. In a complex pipe network there are hundreds of inputs and outputs and any input could be connected to any output by a single, double, triple.. lines of pipes and such lines of pipe do not need to be directly parallel side by side.

[rkfg]

I somehow missed this FFF so only replying now. 12 pages is a lot to digest, maybe my idea has already been posted and discussed but anyway I'll describe it.

The pipe system consists of sources and sinks, basically. Fluids come from somewhere and get consumed anywhere else. The pipe system rarely changes so it can be effectively cached and only recalculated on topology change. We can iterate all sinks and sources and make a Dijkstra pathfinding from each of the nodes, but instead of path cost we'll calculate "influx" (as positives, decreasing on each intersection) and "suction" (as negatives, increasing on intersectons) values, the sum of them is assigned to the blocks. After that it'll be obvious where the liquid moves and how strong is the flow in each block (it always moves to the lesser value and the difference defines the volume). All of this is only required on system change and you don't need to evaluate any complex formulas afterwards.

The downside is pretty obvious: on huge pipe systems there would be zeroes somewhere in the middle and the flow would just stop there. It would require additional pumps to compensate but hey, that's quite realistic too. However, it solves the circular and stall liquids problems. The fair distribution is also covered as we only change the flow power on intersections and not on straight pipes.

It's not obvious from the article if the update is the bottleneck or calculations of that update. If the former then my approach is no better than the current one, of course.

[algogenetienne]

I somehow missed this FFF so only replying now. 12 pages is a lot to digest, maybe my idea has already been posted and discussed but anyway I'll describe it.

The pipe system consists of sources and sinks, basically. Fluids come from somewhere and get consumed anywhere else. The pipe system rarely changes so it can be effectively cached and only recalculated on topology change. We can iterate all sinks and sources and make a Dijkstra pathfinding from each of the nodes, but instead of path cost we'll calculate "influx" (as positives, decreasing on each intersection) and "suction" (as negatives, increasing on intersectons) values, the sum of them is assigned to the blocks. After that it'll be obvious where the liquid moves and how strong is the flow in each block (it always moves to the lesser value and the difference defines the volume). All of this is only required on system change and you don't need to evaluate any complex formulas afterwards.

The downside is pretty obvious: on huge pipe systems there would be zeroes somewhere in the middle and the flow would just stop there. It would require additional pumps to compensate but hey, that's quite realistic too. However, it solves the circular and stall liquids problems. The fair distribution is also covered as we only change the flow power on intersections and not on straight pipes.

It's not obvious from the article if the update is the bottleneck or calculations of that update. If the former then my approach is no better than the current one, of course.

I agree on the model, but Dijkstra algorithm can not be used to solve this problem. Each algorithm can solve a specific problem, if you try to take one algorithm and put it on another problem, you have no guarantee it will work and chances are, you are missing on an algorithm that has been crafted specifically for the problem you are addressing. And then the researchers who work on the latter are sad.

If you are curious, you can check out the first post on mathematical optimization (middle of page 11). That's how one simulates state-wide electrical networks with loss of voltage, and it works to simulate pressure and flow too.

Then, there is the problem that it is an implicit method, which are not often used in video games, due to the fact that there is a limited time to compute next iteration, but that's a problem you will have to face in any attempt to consider the pipe network as a whole.

[rkfg]

I agree on the model, but Dijkstra algorithm can not be used to solve this problem. Each algorithm can solve a specific problem, if you try to take one algorithm and put it on another problem, you have no guarantee it will work and chances are, you are missing on an algorithm that has been crafted specifically for the problem you are addressing. And then the researchers who work on the latter are sad.

I know that this algorithm is not for simulating fluids, I just think it's appropriate in this case. It guarantees traversing the entire graph and builds a path that can easily be backtracked so for me it seems to fit.

If you are curious, you can check out the first post on mathematical optimization (middle of page 11). That's how one simulates state-wide electrical networks with loss of voltage, and it works to simulate pressure and flow too.

Then, there is the problem that it is an implicit method, which are not often used in video games, due to the fact that there is a limited time to compute next iteration, but that's a problem you will have to face in any attempt to consider the pipe network as a whole.

I tried to read this but didn't understand it, unfortunately. Maybe a picture (or better, code) would help. Somehow pure math doesn't work for me, it's like a foreign language. I might understand the words but don't understand the sentences. And I totally know it's a common thing among developers and engineers (like, a lot of cases). Of course, if you are fluent in both math and coding it's a big fat plus but I'm not so I'd prefer good code (with well named, not single-character, variables) over math any time.

[eradicator]

I surely hope that the new system will be great :). But i share the concern that @Dominik is overestimating the amount of straight pipes in large refineries and power plants. So i'd like to contribute two pictures of builds i use that hopefully demonstrate just how many junctions there are, or are not used in real builds.

First the refinery. As you can see there is almost as many junctions as straight pipes, and due to all the different fluids there isn't really any way to optimize this for long straight pipes. I suspect that most mega-factory stage refineries will have a similar fluid system that consists mainly of underground pipes and junctions.

refinery.jpg (444.96 KiB) Viewed 7777 times

Second the power plant. This is probably somewhat atypical, but in my build i don't even have fluid-pipes anymore, but the heatpipes still cause significant UPS costs. And reading that heatpipes are not target of the optimization made me very sad indeed. Also i was hoping that the "fluid optimization" would eventually include optimizing rows of equal buildings (heat exchangers, turbines. i.e. by treating them as one input tank?) but as it sounds this also won't be the case.

powerplant.jpg (658.43 KiB) Viewed 7777 times

(Forum pictures attachment system is still broken, so i can't [inline] the pictures, and the power plant is shown twice... Edit: fixed post now that forum works again...)

[Ars Techne]

Saw this on steam, just passing by, haven't read any replies of this topic, sorry if repeat thoughts.
My Input:
1. Make Tanks large "Junctions". They need to be calculated individually anyways, this would make it more integrated

2. Leave the float, but make an in game device that "purges" fluid to the end. (players should be having a pump or facility somewhere that should prevent back flow). Or have fluid sorters? Easy fix for an odd problem.

3. Flow direction is easy. Judge it on the object. Pumps are always the start, Factories are always the end, and Tanks are judged based on which they are closer too. A Tank "next" to a Factory is a start point and a Tank "next" to a Offshore Pump is an end. Regular Pumps act as both Start and End. That should eliminate some of the processes and leave only balance calculations between junctions.

In Addition:
- I would keep "pressure" for calculating speed of flow. A full tank/pipe would transfer fluid faster than an empty one. A Tank X% full has a fluid transfer speed of X/s. Offshore Pumps just have a consistent X/s as their pressure should always be the same while on. Regular Pumps would act more like a diode in that it would be one way and X/s out = X/s in. But having that % full to determine that speed.

- For even balancing at junctions you need a "big picture" function. I'm pretty sure you talked about this but didn't go into detail. You have it examine neighboring pieces (pipe straights or junctions)(even up stream), reserve an amount that would equally distribute given the junction is still above the neighbors level after distribution, and push that to neighbors that need fluid. Some graphics required...
-------10-------|
-------||-------| In this case Left is > and so wont be considered. We know that after distribution down would be > and so isnt considered.
100==50==10-| So only up and right require water. So (50+10+10)/3 = 23.3333 => 50-23.3333 = 26.6666 => 26.6666/2 = 13.3333 =>
-------||-------| 10 + 13.3333 = 23.3333 (for both up and right). And viola, balanced. Bottom and left then flow into the junction...
-------40-------| (There are issues with this equation, it's just for simplicity of explanation)
Which brings up the point that junctions should be doing all of the calculations... but then junctions right next to each other...
Junctions needing fluid from another junction would "request" a quantity. If the Quantity is not met, Then that junction is re evaluated for refilling. If all directions for refilling were junctions, then refill amount given, else pull more from straight pipe to equalize if possible.

I think that's everything... Maybe I should apply to help work on this, I do have a mind for it and a bachelors in Information Systems

[McDuff]

I have a question: what behaviour do you think users expect, and what makes sense to simulate?

For eg, I would say that it makes sense that in a 2 or 3 way junction that the liquid gets split 2 or 3 ways evenly

Code: Select all

          OUT
           ¦¦
i.e. IN === J  ===OUT
           ¦¦
           OUT

You'd expect all the outs to get 33%.

What behaviour would you expect on a system like this?

Code: Select all

IN =======J=====J========J=======J=====OUT
          ¦¦    ¦¦       ¦¦      ¦¦
          OUT   OUT      OUT     OUT

Would you expect the first output to get fully satisfied before letting fluid pass, or for it to evenly split 50% of whatever was passing through it?

[meganothing]

I have a question: what behaviour do you think users expect, and what makes sense to simulate?

For eg, I would say that it makes sense that in a 2 or 3 way junction that the liquid gets split 2 or 3 ways evenly

Code: Select all

          OUT
           ¦¦
i.e. IN === J  ===OUT
           ¦¦
           OUT

You'd expect all the outs to get 33%.

That is probably too simple. Assume the north and south "OUT"s to be dead ends (or outputs that need only minimal water, or full tanks). You would get only 33% from the right "OUT" even though nobody else needs that other water. Add another pair of north/south dead ends and you would only get 11%. Wouldn't really be intuitive

[Alice3173]

What behaviour would you expect on a system like this?

Code: Select all

IN =======J=====J========J=======J=====OUT
          ¦¦    ¦¦       ¦¦      ¦¦
          OUT   OUT      OUT     OUT

Would you expect the first output to get fully satisfied before letting fluid pass, or for it to evenly split 50% of whatever was passing through it?

One like that should be 50%, 25%, 12.5%, 6.25%, 6.25% for each output going from left to right, not taking into account max throughput for the outputs. (Ie: If the pipe's throughput is much greater than what each output uses then the values would be much different.) If the algorithm goes on a per-junction basis then it'd just split the output evenly at each junction. So the first junction sends 50% to the output and 50% on to the next junction. The second junction then sends half of what remains to the next output (25%) and the other half to the next junction and so on.

[pleegwat]

That's the graph theory approach. Based on full physics, I'd expect resistance and inertia to at least put them closer together, though I'm being too lazy to do the math right now.

[McDuff]

I have a question: what behaviour do you think users expect, and what makes sense to simulate?

For eg, I would say that it makes sense that in a 2 or 3 way junction that the liquid gets split 2 or 3 ways evenly

Code: Select all

          OUT
           ¦¦
i.e. IN === J  ===OUT
           ¦¦
           OUT

You'd expect all the outs to get 33%.

That is probably too simple. Assume the north and south "OUT"s to be dead ends (or outputs that need only minimal water, or full tanks). You would get only 33% from the right "OUT" even though nobody else needs that other water. Add another pair of north/south dead ends and you would only get 11%. Wouldn't really be intuitive

Assume here that all outputs are unsatisfied and that any "satisfied" output backs up and becomes "closed" according to the system. I'm talking about a situation in which all outputs/sinks are demanding fluid - say 3 boilers arranged like that.

In a situation where supply vastly outpaces demand it wouldn't really matter because the pipes would be "full" and each output would get 100% of what it needs. The question is how shortages should manifest so that the player can intuitively work out what is going wrong and fix it.