Friday Facts #260 - New fluid system

[Avezo]

Make sure to read comments to FFF on steam too, some people seem to have good ideas too.

[Claudius1729]

This sounds kind of fun to me, I want to try my hand at it.

I'm not some kind of fluid engineer and I don't have any expectations with coming up with something good, just want to have fun with code.

[Oktokolo]

Does nuclear fuel reprocessing exist? No!

Actually, nuclear fuel reprocessing does (still) exist and was one of the first sources of plutonium for atomic bombs.

[FasterJump]

I think that using the electric network is a good idea. I propose a model based on real life water piping, the one you have in your houses. This allows to get rid of water "level" and use only pressure as variable.

Each source (ex offshore pump) works as a power source that has a fixed tension delivery (pressure).
Each drain (ex boilers) works as a power drain that demand a specific intensity consumption (flow rate).
Intermediate circuitry behave like electric resistance (1Ω for 1 tile).

Example: S=source, D=drain, electric potentials values in blue.

pipe_circuit_1.png (8.32 KiB) Viewed 6875 times

We have intensity values in every tiles, so it is possible to have the animation visible behind the pipes windows!

Example with more nodes in the circuit

attachment=1]pipe_circuit_2.png[/attachment]

[FasterJump]

But what if the demand exceeds the production? No worries, we can calculate how much fluid each drain gets.
In this example, each drain have a high demand, but the supply network doesn't provide enough.

pipe_circuit_3.png (13.11 KiB) Viewed 6875 times

The "faucet valve" of the drain is wide open, pressure is null.

Sorry for making a double post, but i couldn't attach more than 1 image per post

[Zulan]

I think that using the electric network is a good idea. I propose a model based on real life water piping, the one you have in your houses. This allows to get rid of water "level" and use only pressure as variable.

Each source (ex offshore pump) works as a power source that has a fixed tension delivery (pressure).
Each drain (ex boilers) works as a power drain that demand a specific intensity consumption (flow rate).
Intermediate circuitry behave like electric resistance (1Ω for 1 tile).

That's somewhat similar than my suggestion. But I believe if you go with a fixed source *current*, you will do not respect the pipe throughput limitations. You could basically connect any number of sources to a single chain of pipes. The voltage drop would go nuts, but I don't see anything in your model to limit throughput.

If instead of the current, you specify only source and drain voltage, you get a limited current scaled with resistance / pipe length.

[FasterJump]

I may have taken inspiration in previous posts, including yours

You're right, I ignored the pipe throughput limitations. But:
- I think pipes that have a diameter of 1 meter should have a very high throughput
- Maybe a limitation could be added, if this gameplay mechanic is judged essential in the game

My model was not meant to be a throughput limiter, it's main purpose is to know the flowrate in each tile, and in case of insufficient sources, decide how much fluid each drain can obtain.

[Dooces]

First and foremost, a huge thanks to Dominik for this wonderfully detailed FFF. The community involvement from the dev team is unparalleled - not just a PR guy regurgitating scripted answers, but real people engaging their devoted fans. FFFs themselves are above and beyond - the reason I originally purchased the game some 8000 hours ago - but the dev team doesnt stop there. Engaging the forums, answering questions, maintaining composure amidst a plethora of redundant and likely ignorant suggestions (one more further down this post), even streaming the new map editor's near-final tests (thanks rsed!), all warrants a sincere and grateful THANK YOU!

And now for my ignorant suggestion for pipes:

My initial thought to this FFF was that the proposed approach would probably make fairly big, but not very optimized bases more ups efficient, but I am unclear how it would impact most of my layouts. Pushing for the most I can get out of the cpu, I have spent many hours getting pipes as short as possible, and most, if not all pipes are [source -> straight pipe -> UG pipe -> straight pipe -> asm & UG -> straight pipe -> asm & UG] - an almost 1:1 ratio of pipe segments:pipe junctions under the proposed solution.

When supply> demand, would it be possible to keep all pipes at 100 fluid (or potentially progressively less, but hopefully "inactive"), and treating that pipe "network", junctions included, as an single segment, feeding the destination directly from the source, and simply emulate the flow in the pipes for visual reference?

To limit "infinite flow", the total pipe segments between each (source or pump) and destination could be evaluated whenever the network changes, and limited either at the destination fluid box or as a lower fluid flow rate at progressively further distances. The fluid flow would be entirely virtual for (perhaps discrete levels of) graphical representation.

Storage tanks would either act as supply or drain, depending on if they're full or not, and if supply is adequate without draining them, and pumps would act as both.

The malformed idea being is that typically (in my case), once a fluid system is setup, it pretty much runs at steady state 99% of the time, to iterate the details of how exactly it happens each tick feels like an incredible amount of overhead. A UPS optimal system would simply be one where supply>demand and long distances accounted for. The mechanics of the flow evaluated perhaps at intervals, whereby elaborate algorithms could be instanced if supply<demand or pipe distance exceeds required flow- giving the player a call to action to rectify the situation only if they are concerned about UPS or the minutiae of optimizations. While supply > demand, and distance between source and destination is within limits, how the fluid flows through junctions becomes irrelevant outside of graphical representations. The UPS tax of an elaborate algorithm would be levied only on those who could easily afford it and/or indifferent, or those with an clear avenue for exemption.

It would be great if pipe planning becomes only a problem of adequate supply and distance - both of which can be addressed by the player with more pumps or more fluid.

I doubt I could suggest anything that hasnt been already, and likely dismissed long ago by devs more competent than I, but even the illusion of being able to help develop a game many of us are passionate about makes me FEEL included in the process.

And for that, 1000 THANK YOUs would inadequately reflect my gratitude.

[Elok]

About the pipe calculation.

Why don't you do the calculation per group instead of one calculation per piece of pipe?

Example,

To make thing simpler, let's say each pipe can hold a total number of 100 'unit' of water

Group A have 22 pipe total, 2200 unit total.

You fill the first pipe with 100 unit of water. The group A have 100/2200 unit with water. So each pipe need 1/2 unit each, in other words ~4.5/100 unit of water.

Of course, if you do that instantly, you will lose all the animation, but you can always update pipe value smoothly. In the end, it's 1 calculation per group and 1 value update per pipe.

And you'll lose the "wave" effect that we see on the FF. But, honestly, who noticed such wave feature existed? It's a gimmick that doesn't bring much to the game and is very harsh on the CPU.

That's very similar to what they ended up doing.

A little similar, but I'm a big step further. The only similar thing is that straight pipe are calculated are one. But there's a complex calculation for each group.

Here's a few differences :
- For me, the whole pipe system (junction included) is treated as one group. Not just straith pipe.
- In his calculation, each 'group' evaluate how much fluid it have, how much fluid each neighbors have and if it should send fluid to them or not. In my system, the whole group evaluate it's capacity in percent. Them each individual pipe update to reach that target. Since, most of the time a pipe system should be either completely full or empty, a group could know that every pipe doesn't need to be updated and you could skip the individual calculation and just transfer the fluid for the tank to it's destination (there 5 tank and 20 factory connected to the group? Send the fluid from the tank to the factory. The tank are still full enough? The group is still full and no calculation update needed).

[Dominik]

Hi, I will reply in one post. I am sorry if I missed someone. Some posts were not really related or were/are answered somewhere already.
I would like to thank everybody for all the thoughts and work you are putting into this and for nice words and encouragement. This discussion is very productive and gives me useful ideas I will incorporate.

@Garlik30: For me, the default model is the realistic one and I try to follow realism (which automatically makes stuff intuitive), unless there is a reason not to. Pipes are the most efficient way to move fluids you can possibly have, while using bots for it is ridiculous. It is also ridiculous to use bots for moving steel and furnaces and I never liked bot factories at all. So sorry, bot fluid system will not be coming from me.

@hoylemd: Having flow in just one direction would indeed simplify things a lot. It is easy to imagine pipe setups though where two directions totally make sense. E.g. imagine a pipe in this shape |_|. At the top are two consumers, at the bottom two suppliers that each produce 1/2 of what one consumer use. Now the consumers alternate in being on and off. Then what you want to see is the middle pipe being used in different direction each time if you want to always get all production to the active consumer.
Also it is not true that all you need to do is production and consumption. That way you end up with power network option. The throughput also matters.

@Vxsote: I did not really mention gravity in the FFF, forgot to. I started my simulator by modelling gravity. Right now it is there, but I might just cut it out as it does not contribute anything useful, except for a bit of realism in edge cases for imo too much computation. Similar for friction - too complex and just annoying. About the GPU - how would you use it? I don't know very much about GPU computation, I recon it is good for parallel algebra, matrix operation and such. I don't know how to apply it here. PM?

@toxinate: Resistance only works proportionally, can't give you a hard limit. The Kirchoff's law computation works nicely cos all can be modelled by simple linear functions and a hard throughput limit breaks that. At least that is my understanding of not-an-electrical-engineer. @Zulan if you know how to make the equations work with the throughput limit, let me know.
@FasterJump more and more electricity proposals I am having this bad feeling that I might have made one mistake - being stuck with the assumption that a pipe must have a hard constant throughput limit. But without that, having just a soft limit that is modelable by the resistance could be good enough. Ugh.

@pinga Yes, you are right. I will need to do better. I am already thinking about some shortcuts for most common cases of saturated pipes etc. that should help. But also it seems to me that people are forgetting important part of the nuclear setup which is the many long pipes bringing water that can easily account for majority of pipes in the system.
About the machine starvation - look at the gifs at the top of FFF that Klonan made - shows a case of a symmetrical system where some machines are starved. Problem of the game alone, not the player.

@smurphy1 "where the output is split between A and B." ... splitting means a junction. You are not saving anything here. Also you can't do such a simple calculation once there is any fluid in the pipes that breaks the symmetry.

@Phillip017 Actually the other day I had an idea for solving the fluid mixing which imo solves it very easily. Once a fluid is run through an empty pipe, a fluid type is assigned to it's whole system and no other fluid can get in until the system is drained first to restart it. lol, now see @Rhamphoryncus proposing that.
Yes, I will do speed testing.

@Elok this is again the electric network system. You lose throughput limit and all else too. Also @Matthias_Wlkp

@Homepage I see you have thought about it a lot, but you need to go deeper still. The graph search and drain:source ratios need to be recalculated +- every tick as the production and consumption changes +- every tick. Everything can change by that. Now if you do that and address all the details to make it work right (e.g. in your case if you have a straight saturated pipe and and a side dead branch from it, no fluid will ever reach it), you end up with the same thing I have. The topology sort I do is linear, it is really not very different.

@Tankh: Good question, and I don't know. Truth is, I wanted to understand what I am doing so I have not looked at the old code yet I don't know what the complexity difference will be in the end.

@majesty2450 Sounds like a good idea, will think about it. I was thinking about a light version of that, just check if some part of the pipe system did not change from the last tick and if not, just skip all computation inside.

@bobingabout Sorry, I know nothing about this. Ask Klonan, I suppose.

@mrudat I think that the whole thing kinda stands on the simple statement "maximum possible flow through the pipe network". Problem is that there are maximum flow limits of any subset of the network. Unfortunately it can't be solved very simply.

@TheDagmaar I think that it is just a different view on the same thing. My system evaluates the pipes from drains to sources and moves fluids towards the drains.

@Dooces as I already wrote earlier I am considering such optimisations, although they have, so far, been low in the list. Reason is that I am afraid that it would explode to many issues of how to define the networks etc. Also, if I do it too well, imagine a huge smooth running pipe system, then due to something the refineries do not satisfy 100% demand anymore and suddenly booom, ups drop cos the optimisation does not work anymore. So I am still not sure about it.

Again, thanks everyone

[Dooces]

@Dooces as I already wrote earlier I am considering such optimisations, although they have, so far, been low in the list. Reason is that I am afraid that it would explode to many issues of how to define the networks etc. Also, if I do it too well, imagine a huge smooth running pipe system, then due to something the refineries do not satisfy 100% demand anymore and suddenly booom, ups drop cos the optimisation does not work anymore. So I am still not sure about it.

Again, thanks everyone

Thank you!


Wouldnt the networks just be each entire pipe?

Well the UPS would drop to what it would be currently - ~5-10% of overall CPU time (from my naive profiles), but the player can address it, if he chooses to do so... UPS problems from pipes just come down to adequate supply... Reverting to the current algorithm when supply < demand would be temporary, but exciting to know the player has to power to fix it!

In the end - however you do it, will be for the better. God speed!

[Jap2.0]

Do producers/consumers count as junctions? i.e. if you have an unbarreling assembler, a pipe, and a refinery, would that count and 1, 2, or 3 sections? What about offshore pumps, regular pumps, and tanks?

[Tricorius]

It seems to me one good way to “fix” the problem with UPS drains in complex pipe setups is to fix the need for complex pipe setups.

There should be a better way (with hopefully better realism) of storing the heat capacity of a nuclear reactor array than tossing it in a huge mess of pipes and steam tanks.

Personally, I’d like to be able to unlock better battery tech with yellow and purple science. I’d love to have a smaller footprint to store that energy, though I don’t know if increased accumulator tech would unbalance other stuff (like the overall cheapness of solar...I know space requirements are suppose to be an offset to that).


(I realize this is likely to unbalanced things, and I know you’re pushing to a 1.0 so you don’t want to get bogged down in implementing new tech milestones with additional craftables, but also the fluid optimization has taken a lot of time as well.)

That said, this is all super awesome. I think it is probably implied, but are straight pipes and undergrounds considered part of the same segment. So 100 “chunks” with a fully overground single tile width pipe traversing it would be computationally the same as a single underground pipe chain running the same length?

[Oktokolo]

I am sorry if I missed someone.

No problem. I just repeat my proposal:
Regarding the issue making laying unrelated pipes next to eachother impossible, i would like to have the same mechincs like for laying rails (but without curves). Draw straight pipes and only where the new pipe overlaps an old one, connections are made.

[Vxsote]

<lots of snipping>

@Vxsote: I did not really mention gravity in the FFF, forgot to. I started my simulator by modelling gravity. Right now it is there, but I might just cut it out as it does not contribute anything useful, except for a bit of realism in edge cases for imo too much computation. Similar for friction - too complex and just annoying. About the GPU - how would you use it? I don't know very much about GPU computation, I recon it is good for parallel algebra, matrix operation and such. I don't know how to apply it here. PM?

@toxinate: Resistance only works proportionally, can't give you a hard limit. The Kirchoff's law computation works nicely cos all can be modelled by simple linear functions and a hard throughput limit breaks that.

@FasterJump more and more electricity proposals I am having this bad feeling that I might have made one mistake - being stuck with the assumption that a pipe must have a hard constant throughput limit. But without that, having just a soft limit that is modelable by the resistance could be good enough. Ugh.

Will PM regarding GPU.

I also wanted to reply generally about the comparisons to electrical networks and the maximum throughput concepts. Electrical networks and fluid flow do actually have a lot of similarities. People already have mentioned voltage is similar to pressure, resistance is similar to friction, and pipe volume is sort of like capacitance. I would also add that the fluid's momentum is similar to inductance, in that it resists change in flow/current. I think they are probably close enough that an approximation of one that seems realistic in the game could be easily adapted to the other. But I also think one of the things that is being overlooked here is that we're not really modeling steady state flow. Producers and consumers start and stop, and the realism - things like the fluid sloshing around, is the result of the dynamic response. Then again, fluid visibly sloshing around probably isn't that realistic (but I like the effect).

Anyhow, the only reason that fluid throughout is limited at all is due to friction in the pipes. If it was frictionless, you could accelerate the fluid to whatever speed you want, and then you could have whatever throughput you want. (A side note: everything discussed so far is really about non-compressible fluids, which is a good approximation in some cases, and not so good in others. But let's just stick to non-compressible). Modeling friction directly? Probably not necessary, but please do not abandon the concept of loss entirely.

Perhaps the hard limit that some might be thinking of is really the volume of the pipe segment. Another simplification, but it's reasonable to say that you can't put 1.01L of fluid into a 1.00L container. That doesn't mean that you can't have 25L of fluid move through that container over a given period of time. However, if you are only simulating flow at each tick by moving some of the volume of the fluid in each segment to its neighbors, and limiting the calculations to (0 - max volume) per segment, then you have effectively put a cap on throughput as a function of the simulation tick rate. That is undesirable, and in general you don't want there to be a relationship like that.

Really the throughput is limited by the maximum pressure, but even that is tricky because the pressure gradients when the pipe fills up, or if someone starts instantly consuming, are extremely steep. But ignoring that, it's really all about pressure drop per pipe segment (the result of friction, without modeling friction directly), and maximum output pressure of the pumps. And the output pressure of the pumps would drop as they reach their maximum output capacity. Can this be modeled without using the concept of pressure? I guess that's the question we're all speculating about and Dominik is trying to answer.

Feature request: pipe bursts due to fluid hammer effect if you switch too many consumers/producers at the same time

[Avezo]

This reminds me of something from few electrical engineering classes I've had at uni, which could probably applied with great UPS benefits here - No matter how complicated electrical circuit(s) were, there were methods of reducing it into simple circuit with barely few pieces that work the same - imagine you could reduce all calculations for entire pipe network like that.

[H8UL]

Nobody likes barrels. Sorry

Objection!

Interesting FFF! Great write up.

We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc.

Please do add this feature, flow control is a mod which adds these and it truly does improve fluid handling, especially in the later part of the game where the pipework turns in to the proverbial spaghetti. We've all been there - you need to sneak a pipe through a gap but can't because it'll connect to the pipe next to it and different fluids would mix..

The autoconnecting pipes separate pipe building from belt building. If pipes work largely like belts you don't have to think differently, don't have other topological problems to solve.

I'm inclined to agree. Whereas belts have an alternative in bots, pipes don't, and are tricky to optimize due to the way they connect. Good! I like the puzzle.

Which is why I also don't like the idea of bots trivialising fluid handling, and causing the same gameplay conundrum as bots do in replacing belts in the late game. I understand that this impacts UPS for megafactories, but imho bots-only megafactories aren't where the fun is -- for me, and I think a lot of people. It's not the numbers that make a large factory fun, but how it hangs together. That's subjective of course, but I dislike having to play suboptimally in order to make a fun base.

Better if belts and pipes are optimised UPS and have gameplay merit instead or being replaced late game imho

[pleegwat]

@vxsote, isn't flow speed limited by the speed of sound in the fluid in question?

[Homepage]

@Homepage I see you have thought about it a lot, but you need to go deeper still. The graph search and drain:source ratios need to be recalculated +- every tick as the production and consumption changes +- every tick. Everything can change by that. Now if you do that and address all the details to make it work right (e.g. in your case if you have a straight saturated pipe and and a side dead branch from it, no fluid will ever reach it), you end up with the same thing I have. The topology sort I do is linear, it is really not very different.

Yeah, I know the ratios need to be recalculated with every tick and nearly every traversal of the graph would be different, however, that does not change the directions of edges in the graph, it only changes which segments would need to be included in the simulation at that particular tick. However, if you do have a counter-example (and you are willing to spare the time and effort to share it) I would like to see it, to see where exactly I went wrong in my assumptions.

My algorithm has actually covered dead-end pipes - I mark them as low-priority drains.

I guess that our approaches are pretty much the same, all I do is cache the directions of edges in the graph and try to do some early optimizations. I admit I forgot that topology sort is actually linear, silly me .

Anyway, as I said, if you could spare the time and show me a counter-example showing that I cannot cache the directions of edges, I would really appreciate it . Graph theory is indeed intriguing.

[Ohz]

Interesting FFF! Great write up.

We are still considering using different pipes for different shapes, which would allow two parallel separate pipes etc.

Please do add this feature, flow control is a mod which adds these and it truly does improve fluid handling, especially in the later part of the game where the pipework turns in to the proverbial spaghetti. We've all been there - you need to sneak a pipe through a gap but can't because it'll connect to the pipe next to it and different fluids would mix..

Disagree. They made rails able to go straight or turn for a reason: reduce number of different type of stuff in your inventory. Or you drop a pipe and then hit R several times.