Friday Facts #260 - New fluid system

[Hermit]

Good 1st post Dominik!

I think some points you raised lead to an obvious question... if the in-game electrical model is too simplified for fluids, is in-game electrical also too simple for electrical? Should Factorio model amps, volts, ohms, transmission losses, etc? If not, why is that okay for electrical but not fluids?

This is not meant as a flippant question. I sincerely think taking a step back and defining these things is important to shape development.

[Philip017]

@Phillip017 Actually the other day I had an idea for solving the fluid mixing which imo solves it very easily. Once a fluid is run through an empty pipe, a fluid type is assigned to it's whole system and no other fluid can get in until the system is drained first to restart it. lol, now see @Rhamphoryncus proposing that.
Yes, I will do speed testing.

Again, thanks everyone

Thank you very much for your reply. I really appreciate all your hard work you all have done on this wonderful game and how the developers interact with the community around their game.

I am very happy to hear about the whole pipeline becoming assigned to a fluid unless drained first. but now i will have to see how easily it drains. i usually build the whole thing before connecting any fluids, so if i miss a pipe somewhere i usually end up having to dig up the entire pipe line to clean it, if connecting a storage tank and a pump will clean it, all the better, if just a storage tank will clean the line that will be awesome.

with the entire pipeline being assigned a specific fluid type, i have to wonder why we cant simply have the entire thing calculated as one entity, junctions and other pipes only serve as connectors, i don't usually like to have to deal with through put on the pipes, but have run into occasions when the pipe is full at the top and empty at the exit because although i am making enough of the resource i can't get it to where it needs to go with out more pipes. which usually ends up with a total redesign to get it to work again. but hey that's part of the game, something else to figure out.

and for performance, i really hope that you can push things (like pipe mechanics) that hog cpu cycles onto another thread, cpu's these days are looking to be capped on single core performance, no more ghz, 5 is max, at least till they switch from silicon. and instead we have more cores, great for programs that spawn many threads, not so much for those that run on only one. keep in mind not everyone has a 5ghz cpu either, mine is only 4ghz, and some people have laptops that are still ~ 2ghz.

[Rebmes]

I hope this all amounts to a system that is, indeed, more performance-friendly than the previous system. I don't tend to use the logic gates, so the pipes' other shortcomings aren't visible to me.

[Quarnozian]

I just want the machines that create/utilize the fluids to flow at a steady rate.

So let's say a machine makes 60 fluid per second... while it's operating it outputs 1 fluid every tick. Same idea for the machines using the fluids. If they're crafting something that needs 20 fluid, and takes 0.5 seconds to craft, then they'd be using 1 fluid per tick, skipping every 3rd tick. ( 40 fluid per second = 0.66666~ fluid per tick)
This alone could potentially fix a lot of sloshing problems.
_________________________________________________________

I would also prefer if full pipes switched to a PSI model, where the pipes act like a hydraulic system instead of an aqueduct. However this would essentially act like instant-teleporation of fluids, so I don't see it happening.

i.e., in a hydraulic system pulling fluid out wouldn't mean empty space in the pipe. Instead it would suck in fluid from the next pipe, which would suck in fluid from the pipe after that, all the way until it sucked fluid from a tank or machine that was generating the fluid.

This would require a change in the way tanks work though. Since they take up 9 tiles, let's say the first 900 fluid corresponds to the fluid levels in the pipes. when the pipes reach 100% full, the tank has 900 fluid in it, and any more fluid added to the pipe system immediately increases the tank's fluid level. Any fluid pulled from the pipe system would be immediately subtracted from the tank's fluid level.

[BlueTemplar]

I try to follow realism (which automatically makes stuff intuitive)

Well, the problem that it sometimes doesn't. Case in point : Liquids slowing down over long pipe distances is realistic, but not that intuitive.

However, if you are only simulating flow at each tick by moving some of the volume of the fluid in each segment to its neighbors, and limiting the calculations to (0 - max volume) per segment, then you have effectively put a cap on throughput as a function of the simulation tick rate. That is undesirable, and in general you don't want there to be a relationship like that.

It doesn't seem to be a big issue though? AFAIK, this is exactly how electric pump's maximum throughput is currently determined : it's 200 L (the pump's capacity) x 60 ticks/s = 12kL/s !
(electric pumps ignore fluid mechanics on their input - meanwhile offshore pumps don't even have a fluidbox!)

Some people here seemed to wish for pressure, height, viscosity to be simulated by the game's fluid mechanics... but they already are !
As a reminder, here's what a Factorio fluid system currently looks like :

(pump disabled)

And the formula is :
Flow = (Pressure_A - Pressure_B) * 0.4 + Limited[Previous_Flow * 0.59, Target_Capacity * 0.1]
Where 0.4 is the pressure_to_speed_ratio and 0.59 is the flow_to_energy_ratio, which seem to account for things like the fluid's viscosity, and which, while are the same for all the fluids in vanilla, have had mods experiment with changing these values (at least Bob's mods).

Anyway, IMHO, fluids just suffer from a very bad image problem.
(People have an issue, ask about it, get answered "Oh yeah, fluids are too complicated to understand anyway!")
But might completely redoing their mechanics be perhaps an overkill ?

What are the current issues that people seem to have with fluids ?
From most common to least common (IMHO) :

1.) Production stopping because of one of the fluids backing up in the refinery/output fluid network.
Not even a fluid problem, but an assembler-with-multiple-outputs problem!
(see Angel's ore crushing/sorting for a similar issue with items)
Solution : Not sure, but probably not relevant to the discussion at hand anyway.

2.) Fluids not flowing because 0.001 L of a different fluid is blocking the flow somewhere.
Solution : Seems like moving fluids from floats to integers might help ? (As well with UPS?)
(Would also solve a plethora of, less common but still annoying, signal-related problems !)
As well as making sure that the Alt-view clearly shows where the offending "intruder" is ?
I don't understand why fluids have to be so granular? Especially after their unit having been divided by 10 ?
(BTW, is the Offshore Pump producing 1/10 of Electric Pump's max flow by design, or an oversight after that change?)
If they do need a lot of granularity, maybe dividing the units by 10 again would be enough ?

3.) Machines not working well because of limited fluid throughput due to overlong pipes.
Honestly, I have a bit of a hard time in believing this one*
- considering that you need to reach more than 18 pipes between pumps for the throughput to drop under offshore pump's 1.2kL/s one,
- that the first thing that you get told when asking for help is to use underground pipes instead (which increase that distance to a pretty comfortable 99 tiles (> 3 chunks),
- and that you need to reach a good 350-400 pipes (~1900-2200 tiles ~60-69 chunks!) for the flow rate to drop to half that (and that's after the capacitive/flow breakpoint!),
So I'd very much like to see a pre-rocket-launch base that has run into these issues !
(No really, I insist! Otherwise I will just consider this as a perception problem. Blueprints / Save files please ! (and of course only QoL mods!))
Solution :

Why is debug's mode show-fluid-box-fluid-info doing a so much better job of showing what happens than the normal game view ?!?
- First, it seems that there's some bug where you have sprites that still show quite a bit of fluid, while there's no (or almost none) actual fluid in the pipe left. Maybe fix that first?
(Or is that a fancy non-linear display? Integer fluids might solve that issue then?)
- Second, I see that you used the great idea to represent the flow speed by bubbles. Maybe make them even more apparent? (Would a good non-linear scaler for fluid speed / bubble speed do the job ?)
Maybe try adding more fluid viewpoints?
(One issue is that on a high-throughput-optimized fluid network, you'll hardly have any viewpoints left, as you're only going to have straight pipe segments if there's only 1-2 tiles left to connect, and the undergrounds are not long enough.
How to display fluid/flow on them?
Then maybe it's less of an issue, as by that point the player will hopefully be experienced enough with fluids to debug the problem ?
BTW, what are the current ways to measure fluid flow speeds by the circuit network? There needs to be one !)

*One exception : Fluid Tanks - If I understand the issue correctly, their very high capacity (250 pipes!) for the same pressure means that, once their level has equalized, it becomes very hard to make them vary their pressure, which also severely limits their flow!
Solution : it might look weird, but maybe they should show the flow in the same way as pipes? (via bubbles)

4.) Fluid not being equally distributed at junctions (and not "randomly", but in a specific cardinal-directional order) :
Again, concrete examples of the issue In Real Factorio Spaghetti please !
Sure, you might have some assemblers working faster than others, but that will only happen if your flow is insufficient to start with, otherwise, the fluid will just back up and the problem will be solved in a short order.
I mean, IIRC inserters have similar issues, but only when placed in separate chunks, which is an issue even worse to debug, because good luck finding the difference if you don't play with debug's show-tile-grid ON !

5.) Megabase UPS issues.
First, this is only a concern for a tiny minority of players that get to that point.
Second, the FFF mentions nuclear reactors (and by extension, boilers using itemized fuel), but I don't see how any solution could have any chance to compete with the O(2) complexity of solar+accumulator !
(Unless of course Solar is reworked for have so high of a tile footprint impact, that megabases start running into RAM issues... which would probably just turn into a who-can-afford-the-most-RAM race!)
That leaves petrochem... which AFAIK has much lower flows... so what's the issue with using barrels as much as possible? ("Cold" Steam barelling for Coal Liquifaction and/or barrel temperature support, please?)

-.) If I forgot any other problems with fluids, please add them?

Now, about possible optimizations :
As been pointed out in the FFF (and on Reddit), moving from an "every pipe is simulated in that pipe network" to "only simulate inputs/outputs of that network" idea runs into the snag that fluid producers/consumers do not work in flows, but in pulses.
Well, might we be so bold as to propose that the fluid producers/consumers should work in flows instead ?
For instance :
- The input items are consumed all at the start of the production cycle.
- The input fluids are consumed during the whole cycle.
- The output items are generated all at the end of the cycle.
- The output fluids are generated during the whole cycle.
EDIT : Damn, ninjaed by Quarnozian ! That will teach me to proofread for an hour !

What would be great, would be to have two different Fluid Systems accounting for the different states :
During the Transient State, the Fluid Network would work roughly as now, with every pipe being simulated at each tick, with the fancy fluid level waves seen above, etc.
After a while, the Fluid Network would settle into a Steady State, with only the input/output flows from producers/consumers being calculated. This would be probably even more efficient than the FFF's newly proposed system (no need to calculate the junctions every tick!).
Of course, any change in that network's inputs/outputs would make it switch to the Transient State again, until it stabilizes again into a new Steady state.
And the Megabase owners will make their damned sure that this happens as rarely as possible !
If I'm not mistaken, this would be similar to how belts work after their recent(?) rework ? (Though the fluids would not even necessarily need to be "compressed"(=pipes full) in this case!)
I would like to point out that what seems to be the difference between my proposition, and the other similar ones
(though there are so many of them that I might have missed it),
is that it's the Transient State "settling down" that "calculates" what the Steady State is going to be.
(Determining when to "freeze" it might be the tricky part...)
(As a reminder, producers/consumers working with flows rather than in pulses, should solve the commonly raised "but the network would never be steady!" complaint.)

A final note/question about multithreading/GPU : if I'm not mistaken, multithreading the different fluid networks is currently very hard because they have to "check back" with the global simulation each tick?

[TheYeast]

Physicist here. I happened to bring up the wonky fluid behaviour about a year ago, but it didn’t seem to be a high priority at the time. Back then I came up with some alternative algorithms and ran some simulations in an Android app I hacked together for this purpose (can send APK+source if interested).

Based on what I came up with, the proposed model seems needlessly complicated to me. So I think my findings are worth sharing.

In my simulation I used the Euler equations (assumes zero viscosity) because they are computationally cheap and good enough for our purpose.

First off all, I eliminated the behavioural dependence on update order by making sure that the simulation of a pipe only depends on the previous state of the game. That is, a pipe must ignore updates in neighbouring pipes already performed for the same frame.
This fixes the issue of differing flow speeds that were induced by different update order.
It also fixes the fluid preferring a particular direction in junctions (first updated neighbour steals most of the fluid).
Because the simulation is now independent of update order, pipes might as well be updated in order of memory layout, which should be far superior to tracing a particular flow direction.

The model I used is very simple:
Every pipe stores the amount of fluid inside it (pipe.content). The amount of fluid determines the fluid level inside the pipe:

Code: Select all

float Pipe::getLevel() { return content / area; }

Different objects can have different surface areas, so if a pipe would have area=1, at level=100 it would contain 100 fluid, while a storage tank with area=250 would contain 25000 fluid at the same level.
The level is also used to determine the pressure inside the pipe. Higher level yields more pressure.

Every connection between two joined pipes stores the flow speed between these two pipes. A connection is best visualised as a thin line separating two joined pipes. For a straight line of N pipe pieces, there are N-1 connections between them.
I think that this is the key insight that makes this approach work where others fail: the flow speed should not concern the fluid inside a piece of pipe, but the fluid flowing between two neighbouring pieces.

To avoid indirection it is best to store the connection objects inside pipe objects, by letting each pipe internally store two (possibly unused) connections, for example left and top. For the connections to its right/bottom, the neighbouring pipes must be accessed, but that was going to happen anyway during an update.

In the first pass, the flow speeds are updated by comparing the pressure on both sides of each connection:

Code: Select all

connection.flow += (connection.pipe1.getPressure() - connection.pipe0.getPressure()) * dt / liquid.mass;

This way, flow increases depending on the pressure difference between two pipes. Flow speed needs more time to change for heavier liquids (momentum).
There are several alternatives for calculating the pressure inside a pipe. The simplest variant is just using the level of fluid inside the pipe:

Code: Select all

float Pipe::getPressure() { return getLevel(); } 

To prevent fluids from flowing in circles forever, and sloshing back and forth, a small amount of friction needs to be applied to steer flow towards zero in the absence of external forces.

Code: Select all

connection.flow *= (1.0 – liquid.friction);

Where 0 < liquid.friction << 1.0

To prevent draining more from the pipe than it actually contains, I limited the flow based on the direction:

Code: Select all

if (connection.flow > 0) {
	limit = connection.pipe0.content * 0.25;
	if (connection.flow > limit) connection.flow = limit;
} else {
	limit = connection.pipe1.content * -0.25;
	if (connection.flow < limit) connection.flow = limit;
}

The 0.25 factor is applied so that when all four sides are connected, each connection can at most take a quarter of the current content out of a pipe, therefore never lowering it below 0. Of course, in practise taking more out of a pipe would be possible if it wasn't drained by three other pipes at the same time, of even refilled by them, but we cannot know that during this pass. Trying to would break update order independence.

In the second pass connection flow speeds are used to transfer that exact amount between two pipes.

Code: Select all

connection.pipe0.content -= connection.flow;
connection.pipe1.content += connection.flow;

Note that in the first pass pipes are only read (through getPressure()) while connections are updated, and in the second pass pipes are updated while connections are only read. This property makes the simulation independent of update order, and that eliminates most of the wonky artefacts.

It is also worth noting that there is no enforced limit on how much fluid a pipe can contain at any time. This however is not a problem. If no source (pump, oil well, factory outlet) supplies a level above say 100, pressure differences will discourage the level rising above that value. Only conditions where two high waves travel in opposite direction may cause the level to temporarily rise to up to 200. This fact can of course be hidden from the player by clamping the displayed value.
Alternatively, it would be possible (as I limit the outflow) to also limit the inflow to a quarter of the remaining space inside the destination, but I think that needlessly limits the throughput.

I also ran simulations with a different way to calculate pressure:

Code: Select all

float Pipe::getPressure() {
	float lvl = getLevel();
	if (lvl <= maxLevel) return lvl;
	return maxLevel + 20 * (lvl - maxLevel);
}

This models a linear rise in pressure as the fluid level rises, but once the pipe is filled, any additional increase is considered a compression of the fluid, and results in a much sharper rise in pressure.

This approach mitigates the absence of an enforced limit on pipe content, and also allows for more natural shapes of storage tanks if desired. Say a storage tank allows a maxLevel=1000. This would result in a pressure of 1000 when filled. A neighbouring pipe with maxLevel=100 would also have a pressure of 1000 but at a level of just 145. The additional 45 above its maxLevel is a representation of the compression of the fluid inside it.

By tuning liquid.mass and liquid.friction, any desirable pipe throughput limit can be achieved, and behaviour can be different for say crude oil and steam.
When merging pipes into segments for efficiency, the connection.flow update pass can be modified to account for the segment length, and thus reduce the behavioural changes caused by the optimization.

This model works very well in all the simulations I ran, as it should, because its physics model is rather accurate. It makes me a bit sad to read that in the new Factorio fluid physics “speed kinda models the pressure”

Anyway, I’d love to hear what you think, and once again, I can send the source of my model if you guys are interested.

TheYeast

[BlueTemplar]

@TheYeast :
Maybe I'm mistaken, but your model sounds very similar to Factorio's current model ?
(Except that you fixed the update order issue, and want to potentially introduce a more complex pressure system / compression of fluids.)

[DaveMcW]

Since one-way pipes are so much easier to optimize, I suggest aggressively converting everything possible to one-way pipes. The player does not need to place directional pipes, but internally the game should calculate it.

For example, a segment connected to a fluid output should only flow in one direction. Even if the output machine is turned off.

one-way-fluid.jpg (150.65 KiB) Viewed 8180 times

If a junction has N one-way input segments and 1 unknown segment, we can deduce that the unknown segment must be a 1-way output.

If all the segments connected to a junction are one-way, and the total input/output flow rate of machines connected to it does not exceed the limit, we can merge everything into a single segment.

[mrudat]

I don't know if it's been suggested on Steam or not already, but are networking-related (router, switch, etc) algorithms useful?

In the general case, they account for links with fixed maximum throughput, and links that may have flow going in either (or simultaneously opposite) direction at any point in time. As a generality they tend to pre-compute reasoning about how to decide what to do based on network topology, given that they need to make the same decision for large numbers of packets going from various values of a to various values of b.

https://en.wikipedia.org/wiki/Routing would probably be a good starting point to dig from.

Most practical routing algorithms have the property that they operate with information that is local to the machine making the decision, which may or may not be useful when applied to pipe networks in Factorio, given the reason for local information is that sharing a global view reliably is hard, while we have complete knowledge.

[csdt]

I just want to add something about the analogy fluid-electricity:
You naturally get a maximum throughput in your network if your sources have a maximum pressure:
If you look at this wikipedia article: https://en.wikipedia.org/wiki/Hagen%E2% ... n#Equation
You can see that the flow through a pipe can be expressed like this:
Q = ΔP*C/L
where Q is the flow, ΔP is the pressure difference, L is the length of the pipe and C depends only on the pipe diameter and the fluid viscosity.
It is the exact equivalent to the Ohm law: I = U*C/L where L is the length of the wire, and C is the linear conductivity.

Pipes:
So a bunch of pipe would just be a very long resistance. And the maximum throughput would decrease with the length of the pipe because of the viscosity of the fluid (like in real life).
To be noted that it is equivalent to put multiple resistors in series.
Also, if you have a part of the network that contains only pipes, are connected to the rest of the network in only 2 places, then this part can be virtually replaced by only a big "resistance". This works even if the subnetwork contains intersections.
I don't know if this is extensible to subnetworks with more entry points.

Pumps:
A pump can be simulated either as a diode + negative resistance (complex), or as both a drain and a source (less correct, but simpler).

Sources and drains:
A source would just impose a positive pressure, and a drain would be a point of zero pressure.
To simplify the user experience, I think every source should give the same pressure no matter what, and just "disconnect" (pressure is not fixed) when they run out of fluid.
To make the transition smooth, such a source would probably need an internal resistance to make sure that no more fluid can be extracted from the source than available.
The same with drains: as long as they need incoming fluids, they "impose" a pressure of zero. And when they're full, they "disconnect" (pressure is not fixed).
A drain could also have an internal resistance when they're almost full in order to not be able to take more than they can store.

Tanks:
A tank would be equivalent to a capacitor. To make that work, you would need to say that the tank is full when its pressure is equal to the pressure of sources. As the pressure of every sources is the same, you can compute the capacitance of the tank to ensure a certain volume of fluid at this pressure.
Or you could transform the capacitor temporarily into a source when the maximum allowed pressure in a tank is achieved (the tank is full).
I think the first is more accurate, and maybe simpler to implement also.

Maximum throughput:
This design supports a maximum throughput per pipe: As every sources have the same and fixed pressure and pipes have a "resistance", the maximum throughput will be given by the Ohm's law.
If you want more throughput, you need more pipes in parallel.

Misc:
As someone said, if you want to simulate the momentum of the fluid, it would be the equivalent to an inductance (a coil). And every pipe would have an inductance part.
Every pipe would also have a capacitance part (they can store fluid).
Even going that far, it would be possible to simulate locally everything (a pipe only needs the info from its neighbour).
I don't think that's really necessary to go that far, though.

I think it would be fine if pipes did not store any fluid at all, and transmission is instantaneous. This would require sources to have a fluid buffer, and also to forbid the connection between a network of a fluid A with a network of a fluid B.
With that, it would still be possible to show fluid movement inside pipes by just showing the "intensity" going through the pipe.

TL;DR

• Sources have a fluid buffer. As long as this buffer is not empty, the source impose a fixed pressure that does not depends on the source (an offshore pump would make the same pressure as an oil pump or the output of a raffinery).
  If the buffer is empty, the source is "disconnected" from the network, and no pressure is imposed.
  A source might have an intern "resistance" to accomodate a buffer partially full.
• Drains are the opposite of a source: when they need a fluid income, they imposed a fixed pressure of zero. When they're full, they are "disconnected" from the network and no pressure is imposed.
  A drain might have an intern "resistance" to accomodate a small need in fluid.
• pipes are just resistances and can be combined together to create bigger resistances. They don't contain any fluid per se, but you could still see the fluid moving inside but just showing the intensity going through the pipe. A subnetwork containing only pipes (even with intersections) and connected to the rest of the network in at most two places can be reduced into a single edge (a really big pipe).
• Regular pumps are just a combo drain+source, or more complicated: diode + negative resistance
• Tanks are discrete capacitors. When they are full, they have the same pressure as the output pressure of a source.
• A fluid network can contain only one fluid (all the sources/tanks must have the same fluid).

Final note
I think that electricity is so similar to fluid in pipes, that it is the way to go. You can simulate everything you want with this, even the most complicated stuff like fluid momentum.
On the most complex designs (with inductive and capacitive pipes), all the equations are local to entities.
With simpler designs (pipes are only resistives), the overall computation should be quite light.

[FasterJump]

Sources and drains:
A source would just impose a positive pressure, and a drain would be a point of zero pressure.

Although I like your model a lot, I think I disagree with this part.
Let's say you have a high pressure in your water pipes in your house. Open your faucet valve in your kitchen. The pressure inside the faucet will not drop to 0.
I think that the drains have a specific flow-rate limit, ie maximum intensity drain. This is what happens in factorio where assemblers have a maximum consumption speed.
But; let's say you have an assembler with an infinite speed, i would agree with you: the infinite sucking power would drop the pressure to 0.


In this image, we have 2 cases:
-The source production is highter than the drain demand
-The source production is lower than the drain demand

pressures_4.png (9.88 KiB) Viewed 8147 times

Note that this is a "graph" of the water pressure, NOT the water level: in all cases pipes are full of water

My 2 cents

[LucidMoses]

Here is a very fast system that accomplishes all your liquid goals.

Divide the pipes into types.

Flow pipes are straight, turns and underground.
Juntion pipes are any pipe that has more then two connections.
Devices (pumps, assemblers, refineries, etc)

Forget individual pipes volumes and direction for flow pipes. Instead put all that detail into the intrisic nature of the system by creating a table of pipe networks table.
When someone places a flow pipe by itself create a new item in the table with a key new key (index values would be a ton faster), Say ‘KeyA’.
When someone places a flow pipe beside another flow pipe that doesn’t cause a junction Give the new pipe the same KeyA and add the volume of the pipe to the total volume of the network table for KeyA.

Junctions are there own item in the pipe networks table that connect to multiple other networks.

For devices it simply needs to know the pipe network key for each connection.

Creating this would be crazy fast and do at the speed of pipe/device placement and removal. This pushing a big chunk of the problem to player placement speed and not ups. Removal could cause pipe network splits when not removing the end one. A simple pathing algorithm would give you the split data. Performance of this isn’t that important as the data is so small (<500ish) and it doesn’t happen that much. Say 10% of pipe removals by the player and 33% by bots.

Now in each tic all you do is run down the pipe network table and remove appropriate quantity from each network based on device inputs.

Run through the network table again balancing the junctions. For each Pipe network in the junction you have the volume of the pipe network and the amount of fluid in it. So make all pipe network’s unit level the same across all networks in the junction. For better results I would take into consideration a maximum flow rate of the junction and a running average (say 4 tics deep) for each pipe network connection and offset the balance by that amount.

Then go rough all supply devices and add the appropriate amount of liquid to the pipe network. I would allow every device attached to the pipe network to dump all its liquid as long as the capacity is not full. Let the liquid go over capacity (hidden for game play reasons).

On the graphics display side. Each pipe has the pipe network id and can easily tell what to render.

There are a bunch of edge condition you will have to take into account but none that affect performance. This would all be supper quick as you would only have to calculate junctions and devices and really how many T-Pipes to people have on their maps vs individual pipes.

So let’s check your requirements.

Fluids get where they should - Yup
They should act in a predictable manner, with reasonable splitting/joining in junctions – Yup and intuitive and instantly responsive to change without crazy things happening.
Fluids can travel instantly, if need be. - Initially, One tic delay on first cycle as usage is before production plus 1/2 tic delay per junction from source to destination. Player wise that’s sub second and pretty instant.
Respect the pipe throughput limitations – Yup.
Flow can be viewed on the pipes – Yup, type and amount.
Don’t do f**** up stuff like running in a circle indefinitely, sloshing back and forth endlessly etc. - Yup. Even with cheat code/commands flooding/emptying pipes the system would be stable and just trend it back to normal.
Should be faster/more UPS efficient. YES, One simple(ish) calc per junction. Even Production/Usage devices would also be simplified.

[GenBOOM]

I don't get why flow simulation is so hard?

treat the fluid source as the opposite of a black hole or think of the pipe network as a drain.
Pipes coming from source, 0, have a negative value -1, -2, -3 etc the farther you get from source the faster the liquid movement speed will increase to a set max value.
-This is fluid momentum based.
Source pipe pressure could be a value added to increase initial fluid flow in pipes.
-This imagines that pipes farther from source have more pressure added to them by the fluid behind them.
Pipe elevation is not a factor either, but could also be a pressure value from one section of pipe to another.
-This effectively makes fluid flow a 1-way system because the momentum of the flow would prevent it from going backwards, unless the pipe section has equalized or a new pipe section has been added spontaneously, in which case a new junction would have to be added but the direction the fluid is flowing would remain the same due to the momentum. (If you cut your garden hose while the water is running, the water does not run backwards in the hose to the new output, the input is split between the hose and the new hole!)

Pipes at the same distance from source should be updated at the same tick and pipes at a junction would be split evenly on that tick.
Pipes have a capacity.

Pipes that have a dead end would begin to fill up until the input at a junction equalizes.
-The first junction of that pipe section with the dead end eventually neutralizes that section of the junction, because 1 tick after the junction outputs are no longer balanced the unbalanced section would flow backwards into the junction.
-There is no need to look ahead and waste time calculating, as long as you know its a dead end / not in use. so keep in memory the outputs/end of pipes junctions and their input/outputs, no straight sections at all they can be skipped entirely. Only the input / output / junction sections of pipe matter, everything else is just drawn to screen based on those factors.
-The fluid flow calculation should realize that a section of the junction will be a higher value than the other 1 or 2 sections of junctions connected and the flow would instead be divided into the active sections of pipe. So you will get a slight output increase in the active sections of pipe once the inactive section equalizes.

So imagine that all active inputs are on top and all active outputs are drains on the bottom, this means that dead ends in a pipe or inactive outputs are equal in elevation to the relevant junction so the fluid can only move with momentum downwards towards a drain. Hopefully this all makes sense.

------[Ji1]
[JO1][J1][J02][_][O]
-----[J03]
------[_]
------[O]

-----[0]
[-2][-1][-2][-3][-4]
---[-2]
---[-3]
---[-4]

---[S]
[_][-][_][_][O]
---[_]
---[_]
---[O]

---[S]
[-][-][-][_][O]
---[-]
---[_]
---[O]

---[S]
[=][+][-][-][O]
---[-]
---[-]
---[O]

---[S]
[=][+][+][-][O]
---[+]
---[-]
---[O]

---[S]
[=][+][+][+][O]
---[+]
---[+]
---[O]

[TheVeteraNoob]

Nobody ever reads my posts anyways. But I just had an epiphany.

What if we only had one variable (fluid in pipe) that also encompasses pressure. It sounds similar to what we are doing currently. But what I'm proposing is a pretty substantial change in function.

Biggest Change: every producer of fluid has a maximum ammount of fluid inside before it stops producing. So offshore pumps, have a value of say... 50, assemblers and chemical plants at 100, storage tank, 0-100 based on fluid level, and pumps, 200. While this does treat fluids as highly compressible (which ruins any chance of realism in that one regard) this leads me to...

The second half: (this one doesn't have to be exactly done this method) Base rate of flow on Delta between pipes. Say... .4x Delta. Say you have 1 pipe with 200 and another with 100. In one tick it would transfer 40 fluid leaving 160 in one and 140 in the other, next tick it would be 8 fluid with 152 in one and 148 in the other, then the last tick is 2 is transferred leaving 150 (or round down and then the losses in pressure can be considered just frictional losses)

As for the mixed fluids problem. I think that the fluid containing the larger volume should be allowed to push the smaller volume until it is either; pushed into a singular entity, or it's volume if compressed into the next pipe would be higher. While that is sorta computationally heavy factories will never really get to megabase point with hundreds of mixed fluid networks.

One last point: I am +1 ing everybody that says to make fluids an integer.

[Jap2.0]

Nobody ever reads my posts anyways.

Objection.

[TheVeteraNoob]

Nobody ever reads my posts anyways.

Objection.

Sustained. You clearly proved me wrong.

[Omnifarious]

The maximum flow problem in a graph is a fairly well studied computer science problem.

I would suggest you do this:

Every time someone alters a pipe network, start a thread to recalculate the network. Have the thread check for an abort signal so that it can be aborted if the network changes again before it's finished. While this thread is running, flow through the network is halted.

Build a graph of the network. Junctions are nodes. Sinks and sources are nodes. Storage tanks are really tricky here. If, on any tick, production exceeds consumption, they are sinks. Otherwise, they are sources. Any group of directly connected storage tanks should be considered one storage tank in the fluid network it participates in.

You can then treat this network very much like the electrical network. But since you've defined a maximum flow for each pipe, one of the maximum flow algorithms described there on Wikipedia can be used to limit how much fluid can be delivered over the network. They can do this every time the network is reconfigured. This should also give you information on the direction of fluid flow through any pipe segment.

I see a few problems...

Sources are discreet. The algorithms are designed to deal with continuous sources. Because of this, the flow between the currently active source (or sources) and the available sinks may be less then the maximum flow from all sources to all sinks.

The same issue exists (to some extent) with sinks. A factory full up on fluid will become suddenly not full up when it starts manufacturing a new item.

One approach to solve that issue and retain accuracy is to make a different calculation for every combination of sources and sinks (though all storage tanks can be considered one source or one sink) and then cache the results.

Of course, it may turn out that there is a maximal flow algorithm that's fast enough to run on every tick with the size pipe networks that exist in most bases.

[Zavian]

Personally I agree with most of what Blue Templar posted. Almost all of them are caused by Factorio not showing players visually what is wrong whilst walking around. If you know where to look the information is available in the GUI, but for new players what is wrong and which bit of the GUI to look at to confirm that isn't obvious whilst walking around.

1.) Production stopping because of one of the fluids backing up in the refinery/output fluid network.
Not even a fluid problem, but an assembler-with-multiple-outputs problem!

I'll note that refineries are probably the first time new players will have to deal with multiple outputs that can backup. It's also the first time they need to deal with anything more complicated than a single linear pipe. Better visibility of information might help, but not a pipe/fluid mechanics problem.

3.) Machines not working well because of limited fluid throughput due to overlong pipes.

For me the related problem of people trying to run too many machines from one pipe/fluid source seems more common. The most common example seems to be trying to run too many boilers from a single offshore pump and pipe. If the player realises (or someone else points out) that one offshore pump doesn't provide enough water for that many boilers, people try to fix it by simply adding another pump, and then might immediately run into the pipe flow limit. (I've also seen people hit the pipe flow limit when trying to increase refinery output either by just extending their existing refinery layout or by adding beacons and when building nuclear reactors). Again not a pipe/fluid mechanics problem, but a information display issue. (A pipe tutorial that mentions that pipes have a max flow rate, and that the effective flow rate decreases the longer the pipe might help).

Fluid not being equally distributed at junctions

If supply exceeds demand, then typically that isn't a problem. For cases where supply doesn't exceed demand (eg chem plant making lube, plus a chem plant cracking heavy oil to light oil), then you can add a storage tank and a circuit controlled pump to shut off some of that demand when supply is low (eg a circuit controlled pump that only feeds heavy oil to cracking when there is more than 15k heavy oil in the tank).

Better distribution at junctions might help, but I consider a pump with a circuit connection to be a better way to fix this, which makes this a factory design issue. (A tutorial on setting up something similar might help new players).

For megabases then UPS is always a consideration, since the ultimate limits to base size are UPS and player patience/perseverance.

Lastly some of these problems and their solutions could be illustrated in the tutorial campaign eg an overlong pipe that is restricting supply to a refinery that is solved by adding pumps periodically, then later by replacing the pipe with trains. You could fix a shortage of lube caused by too much cracking in a similar manner.

[IronCartographer]

Too bad it's probably not practical to use the GPU for all the fluid simulation, due to the need to maintain a copy in RAM for all the interactions with entities and scripts. Even if you could streamline the memory transfer and sort all the entities along with their fluid boxes for cache locality, IIRC the GPU wouldn't have the floating point consistency required.

Sidenote: Dominik, you didn't actually address the question of straight-pipe-equivalent entities with pass-through that Bobingabout mentioned. Would steam engines/turbines and other linear dual-connection fluid boxes be easily merged? The result would be efficiency gains even within steam columns and flamethrower walls, at no more realism cost than the proposed straight-pipe merge.

[Omnivore]

To me, from both real-world time management and in-game performance perspectives, it seems as if realistic fluid mechanics are far more expensive than they're worth.

Yes, some things would be lost in simplification, but I suspect that what would be lost is far less important than the other things that both the thinky-meat and silicon should be working on, instead.

tl;dr: The nuclear option sounds good.