Friday Facts #260 - New fluid system

[Lubricus]

why dont you try a suction / propulsion system and a size 4 vector in a network of nodes with neighbouring relationship?

end points produce a fixed amount of s/p (end points are factories and pumps whereas a pump acts as 2 end points), pipe nodes will inherit the values of their neighbours.
the resulting underlying suction and propulsion in each node will then be calculated into a net throughput(that is not passed on to their neighbour! if we do that we get markov chains which we dont want to solve)

this has two clear advantages: when a pipe node is added to the network only that one node needs to be updated. if an end point is added only parts of the network will need to be updated and only once. We avoid complex calculations all together through a simplified model

optimisations can be done by using only one directional edges. propulsion would be updated only in the direction of edges, where as suction would only need to be updated in opposite direction of flow and only for the new nodes.

nodes with max 2 edges can still easily be found and clustered together (you called it a section) and treated as one node using the throughput of the bottleneck (lowest value)
clustering can be turned off in smaller bases in favor of realism but turned on if ups drops.

TLDR:
2 node types: pipes, end points
end points have fixed directional value
pipes inherit neighbouring values (friction values my be added during this step)
only directional values are exchanged between nodes and then translated into individual throughput.
now fluids will be passed on according to throughput


heres an example: the value in the nodes represents the total throughput. I have added a fixed friction value of 0.1
Edit: in this example at step 3 adding a new joint I made the mistake and manipulated the directional values on a junction. it would be better to do that at the throughput level. that way we wouldnt need to update the directional values of the already existing joint at all after attaching the new one

Fluid production and consumption varies over time and even worse comes in pulses at discreet ticks when recipes start/finishes.

[Bauer]

I would love to see that fluid production and consumption is done continuously. That would be much more intuitive.

[TheYeast]

- As expected, tanks need pumps. They seem to sometimes act way too greedy - blocking any fluid from reaching the consumers.

Yes, I didn't think the storage tanks through A tank can only 'decide' whether to take in fluid or not based on the level in the adjacent pipe, and that is not a really good criterium. That way upstream tanks will be more greedy than downstream tanks. Factorio would really benefit from a way to read pipe level into the circuit network, so the player can drive pumps to fill and empty tanks when he thinks it appropriate.

- If i enable min/max content clamping, the pipe system responses become too slow.

True: 11 producers in default config produce 1100 fluid each second. Theoretical maximum transfer is 100 (pipe content) * 60 (ups) = 6000. First clamp quarters that amount to 1500, second clamp halves it again to 750, so your 1100 exceeds that. I think the 6000 is possible using the clamp method quinor proposed (and then also not clamping the upper limit).

I updated the simulator to maintain 60 ups but render at lower fps if the browser cannot pull it off (instead of dropping ups as well).

Also, the Fixed acceleration model proposed by quinor inspired me to make an adaption to my original proposal. Quinors goal was to have a friction-less model so pipe-length is irrelevant for large endgame builds. In my original proposal friction can be set to zero to accomplish that, but that would mean waves would bounce back and forth endlessly. Quinors model quickly equalises fluid levels so you don't have that. Unfortunately the fixed acceleration creates a lot of noise in the fluid levels and flow direction as well. This is what I came up with:

Wave equation with damping - Model that can quickly eliminate waves without relying on friction.

It works like the normal wave equation, but the flow update step is a bit different:
Original flow update step:

Code: Select all

    dp = pressure0 - pressure1;
    flow += dp * cSquared;
 

Flow update step with dampening:

Code: Select all

    dp = pressure0 - pressure1;
    if (sign(dp) == sign(flow)) c = cSquared;
    else c = cSquaredDamper;
    flow += dp * c;
 

That is, if the pressure difference matches the previous flow direction, the normal cSquared is used, but if the previous flow direction is now against the pressure gradient, a higher value cSquaredDamper is used instead.
In other words, if there is a congestion ahead, the flow will be forced to decelerate faster.

If cSquaredDamper is chosen to be equal to cSquared, the model of course behaves the same as the normal wave equation.

This model quickly eliminates waves even when friction = 0, though I personally remain in favour of also keeping some friction in the system. In fact, this model is far better at eliminating waves than friction alone can accomplish, so I think this model would be the closest to what most players expect from their pipes systems.

Try and let me know what you think of it! Go to https://jsfiddle.net/TheYeast/c9d2azjo/show and select Wave equation with damping

[bsless]

I got excited and jumped ahead from page 6 to make my first forum post on a very interesting topic

I want to start off by seconding @Vxsote's suggestion. It should be completely feasible to take physical flow equations, simplify them down to the level of the problem, convert them to time difference representation from continuous, and use iterative methods to update the state of all the pipe network segments in one update.

I also wanted to chime on on the electrical network ideas proposed, especially by @FasterJump, and give my 2 cents as an electrical engineer, and my experience with using SPICE, and my sketchy memory from writing numeric differential equation solvers in school:

I think treating each pipe segment as having a resistance of 1 unit is a good idea, however, I would also give each pipe a capacitance unit. each pipe segment will be a small RC network of 0.5C --- 1R --- 0.5C. This will accurately model time domain effects when solving in each node of the network Kirchhoff laws and approximating the current through the capacitor by:
Q = CV => I = dQ/dt = CdV/dt => I =~ C ( V[t+Dt] - V[t] ) / Dt
Model pumps as current sources
This leaves you with a set of simple difference equations you can solve each update to update the state of all the pipes.
The amount you're actually interested in, in this case, is Q (the amount of liquid in the capacitor / tank)
You can also model fluid viscosity by scaling R

Pros:
Parallelism, time domain effects, can run on different threads, coheres with modelling tanks as capacitors

Cons:
Could be extremely prohibitive for a high number of junctions, but can't really tell how bad it is since the calculations are pretty simple

Possible workarounds:
Circuit "libraries" for typical pipe networks. Simulate once, extract plenty of behaviors based on inputs, and plop inside the simulation as a "black box"

Very interesting read. physical simulations are fun

[Oktokolo]

Wave equation with damping - Model that can quickly eliminate waves without relying on friction.
...
This model quickly eliminates waves even when friction = 0, though I personally remain in favour of also keeping some friction in the system. In fact, this model is far better at eliminating waves than friction alone can accomplish, so I think this model would be the closest to what most players expect from their pipes systems.

Try and let me know what you think of it! Go to https://jsfiddle.net/TheYeast/c9d2azjo/show and select Wave equation with damping

I like the Wave equation with damping. I still need to disable max content enforcement (not a bad thing - i see extra content as increased pressure and would see negative content as vacuum) and the lovely hammer waves get indeed dampened too much with default settings. But it behaves a lot like i would expect it to. I too am in the pro friction camp.
I would say: Keep the erratic waves and let players have hammer/shock arresters in their pipework like we have in real pipeworks. Hammers are no problem for small factories and extensive pipeworks should offer more complexity anyway.

[TheYeast]

I still need to disable max content enforcement (not a bad thing - i see extra content as increased pressure and would see negative content as vacuum) and the lovely hammer waves get indeed dampened too much with default settings.

I changed the defaults to a bit more conservative settings, so the hammers are back

Also, I added an improved clamping model: Overcommit Clamping

As Oktokolo noted, for short pipes and with low friction, the clamp function is the main constraint on flow rate.

Ideally, you would like to move all 100 fluid from one pipe to the next each frame. With 60 UPS you get a flow rate of 6000/s. However, to maintain update order Independence, you cannot assume the destination pipe has enough outflow to become empty in time. Therefore, the best you can do is having the whole pipe at level 50, so each segment in a pipe has 50 fluid available to pass on, as well as 50 space available to receive. So we move 50 per frame, or 3000/s.

This can be achieved by analysing how fluid and space are contested on all sides of a segment, and if not contested, pass the full 50 fluid along. This method was proposed by Quinor. It is, however, a bit more expensive to calculate.

The conservative approach ( Quarter Limit ) is to assume the worst case: four connected pipes all want to take out liquid, so they all get to take at most 1/4 of the available liquid. The same goes for available space. The maximal flow is achieved again by half filled pipes, such that of 50 a quarter (12.5) can be taken out and passed on. This results in 12.5*60 = 750/s

If there are less then four connections, we can of course just use that fraction, so for a straight pipe we can pass on half of 50 each frame and get 1500/s.
This is what the Connection Count Limit does. However, in hindsight, this is flawed: as soon as there is one junction in a straight pipe, it instantly becomes a bottleneck because it has more than two connections.

The new clamp method I added, Overcommit Clamping, will optimistically allow a connection to fill more than a quarter of the remaining space, hoping that not all connections are trying to do the same, or that others will drain the surplus. When this fails, the pipe will overflow, but unlike turning clamping off completely, this will only occur once: when overflowed, there is no more remaining space, so all inflow stops until the segment is drained below its capacity once again.
The same can be done with the outflow of liquid, which may result in underflow (negative content), stopping all outflow until refilled.

(Obviously, the rare occurrence of overflow and underflow should be hidden from the player)

Default dividers are divFluid = 4 and divSpace = 1. This still prohibits underflow, but does allow overflow if fluid is coming in from multiple directions.
Dividers of divFluid = 4 and divSpace = 4 have the same effect as the Quarter Limit method.

The maximal flow rate is calculated as

Code: Select all

maxFlow = 6000 / ( divFluid + divSpace )

This confirms that maxFlow for the Quarter Limit method is 750/s, and maxFlow for the default Overcommit Clamping settings is 1200/s.

If more is needed, using divFluid = 1 and divSpace = 1 can give us maxFlow = 3000/s. This will allow some underflow, but in practice this doesn't happen that often; mostly when adding or removing pipes.

If 1000/s is enough, using divFluid = 4 and divSpace = 2 can be used, which will prohibit overflow in pipes without junctions, as well as all underflow.

To study the effects, I added a Total overflow and Total underflow counter in the simulator:

https://jsfiddle.net/TheYeast/c9d2azjo/show

I timed some setups in current Factorio, and the max flow rate I can get through a pipe of 1 tile is around 2000/s, but this quickly drops to ~1000/s as the pipe gets longer (~15+ tiles).

So my question is: how much do really big end game factories currently need? And most importantly: do they rely on the very high flow rates that can currently be achieved over very short pipe lengths?

[quinor]

You need ~3k/s of oil for a 1rpm factory if my math is not off. It being ~3 pipes over long distances is OK in my opinion.

[Oktokolo]

I changed the defaults to a bit more conservative settings, so the hammers are back
Also, I added an improved clamping model: Overcommit Clamping

The defaults work fine now.

We have a nice (but rather soft, wich is fine for gameplay purposes) hammer effect when connecting/disconnecting stuff, wich might get relevant in huge circuit-controlled refinery setups (they look fine in the simulation), but can easily be countered by the common pump-tank-pump (or a shock arrester, if someone mods one in).
The flow looks right too (i did not do any math - just looked at pipes in the simulator).
Changes still propagate a bit slower than expected - but not by much, so it should not be an issue ingame.

[McDuff]

Is the idea of a pipe containing more than it can contain, or less than zero, even for a tick, acceptable?

[Oktokolo]

Is the idea of a pipe containing more than it can contain, or less than zero, even for a tick, acceptable?

Sure.
Overfilling can be seen as increasing the pressure. Negative content could be seen as underpressure. You could have both for as long as the causing conditions last.

Allowing temporary overfilling is cheap and TheYeast's current simulation profits a lot from it. Getting the fast and real-looking reaction without using that shortcut would probably need more simulation logic - wich obviously would eat up more CPU cycles.

[TheYeast]

Is the idea of a pipe containing more than it can contain, or less than zero, even for a tick, acceptable?

Sure.
Overfilling can be seen as increasing the pressure. Negative content could be seen as underpressure. You could have both for as long as the causing conditions last.

Exactly. Overflow is completely harmless, an it is strictly limited to a worst case of filling a pipe up to 400: completely empty and filled from all sides in a single tick. In practice however, I don't see how you could ever pull that off. Under extreme conditions I got 113 at most.

Underflow however has a subtle problem (therefore I picked the default divFluid=4 to prohibit it): when the player manages to underflow a pipe and remove it before it is refilled, he actually created some fluid from nothing, in the same way you lose fluid when you remove a non-empty pipe. This seems very hard to accomplish though, with minimal gain. However, if a setup can be created that takes in fluid, then empties itself, and ends up with underflowed components, building and destroying it repeatedly by blueprint would allow a player to cheat. I'm not sure if the created fluid would even be worth the power used by the construction robots, but if this does turn out to be a way to cheat, and divFluid < 4 is desirable for higher max flow rate, then the game could just remove the created fluid form a tank or pipe elsewhere in the factory to negate the gain.

[TheYeast]

[...] hammer effect when connecting/disconnecting stuff, wich might get relevant in huge circuit-controlled refinery setups (they look fine in the simulation), but can easily be countered by the common pump-tank-pump (or a shock arrester, if someone mods one in).

I can't really imagine a circuit-controlled setup that would have trouble with hammers.. do you have something in mind?

Changes still propagate a bit slower than expected - but not by much, so it should not be an issue ingame.

I picked cSquared = 0.03, so the wave speed is sqrt(0.03)*60 = 10.4 tiles per second. I find it aesthetically pleasing to see fluid rush through the pipes when I connect a new empty pipe to a source .

This is also the speed at which changes in supply/demand travel, so if you have a dormant consumer that suddenly wakes up, it might take some time for the flow to speed up to fully satisfy the new demand, especially with long pipes. But I consider this realistic: if you have a long pipe filled with a heavy stationary liquid, it will take some effort to get it moving. Given the limited fixed power that a pump can effectively utilise, it can only accelerate the fluid by so much. The longer the pipe, the heavier the weight that needs to be accelerated, the lower the acceleration. Players who want to use long pipes and have fast response to changes in demand should use a storage tank at the demand side as a local buffer.

[Oktokolo]

Underflow however has a subtle problem (therefore I picked the default divFluid=4 to prohibit it): when the player manages to underflow a pipe and remove it before it is refilled, he actually created some fluid from nothing, in the same way you lose fluid when you remove a non-empty pipe. This seems very hard to accomplish though, with minimal gain. However, if a setup can be created that takes in fluid, then empties itself, and ends up with underflowed components, building and destroying it repeatedly by blueprint would allow a player to cheat.

I would expect underflow abuse only to be feasible when playing with mods that allow automation of planning the construction and deconstruction (Recursive Blueprints). But such mods are not common in PvP and in PvE cheating some fluid isn't a problem at all.
So i don't think, it would be an issue worth fixing.
And your algorithm works fine without allowing underflow anyway (would still be nice to allow mods to enable it though)...

[...] hammer effect when connecting/disconnecting stuff, wich might get relevant in huge circuit-controlled refinery setups (they look fine in the simulation), but can easily be countered by the common pump-tank-pump (or a shock arrester, if someone mods one in).

I can't really imagine a circuit-controlled setup that would have trouble with hammers.. do you have something in mind?

I had large setups switched on and off by circuits in mind (cracking and fuel production in vanilla). But now that i think about it, apart from starving some machinery for a split-second, a hammer currently can't have negative effects in vanilla. Maybe someone should write a mod that makes pipes and machinery explode if hammered too hard...

[TheYeast]

- As expected, tanks need pumps. They seem to sometimes act way too greedy - blocking any fluid from reaching the consumers.

I reverted the behaviour of storage tanks to what Factorio does currently: storage tanks will become filled to the same percentage as the pipes they are connected to.
The build-in pump I gave them will not behave the way you would want them to depending on the placement in the pipe system. For creating setups that respond perfectly to sudden changes in demand over long distances (as discussed in a previous post), the player could configure their own pump+tank combinations. Because we cannot measure the fluid level in pipes, you would need an additional tank to measure the level though.

Also, I cleaned up the code a bit and input values in the UI are now validated:

https://jsfiddle.net/TheYeast/c9d2azjo/show

[lacika2000]

Also, I cleaned up the code a bit and input values in the UI are now validated:
https://jsfiddle.net/TheYeast/c9d2azjo/show

It is actually quite neat, well done!
What can we do so that the dev team takes notice?

[Koub]

What can we do so that the dev team takes notice?

Nothing : the dev team reads most, if not all what's posted in the forum .

[mrvn]

I think "behaves like a belt with splitters" is the intuitive result people think they're going to get without knowing anything about it. I guess the question is, is that the right thing they should be expecting?

It is loosely what happens in reality, it's intuitive and it is what all even halfway realistic models will do. It even holds true with outputs that only need a fraction of the throughput. The "belt with splitter" model would have items back up till it hits the splitter and then more goes the other way.

What the "belt with splitter" model doesn't account for is inertia. Water going down the pipe at speeds will be less likely to take a turn. It also doesn't account for trubulence. A junction disrupts the smooth flow and has more resistance than a straight pipe. But if you go into that much detail then you would have to think about pressure waves traveling down the pipes from pumps going on and off. Maybe even causing a standing wave or resonance in the pipe and the pipes exploding from too much pressure.

[TheYeast]

It is actually quite neat, well done!

Thanks!

In other news: because some concern was raised about the impact of the clamp method on the realism of the simulation, I decided that it would be useful to measure how much of the flow is actually being clamped, so I added a counter for that.

It only counts the inhibited flow due to the clamp method: for example, when a pipe tile holds 20 fluid, and the out-flow is 37, the flow would reasonably have to be limited to 20 anyway. When a clamp method would further reduce the flow to, say, 5, then only 15 is counted, not the full reduction of 32.

The amount of fluid clamped turns out to be very low, only a few units of fluid per frame on average. All counters ( overflow, underflow, flow clamped ) now display the amount per frame, and the average of that value over time. Clicking the counter will reset the time average, if you want to measure a new setup.

https://jsfiddle.net/TheYeast/c9d2azjo/show

[fishycat]

Here's an interesting article: https://arstechnica.com/science/2018/10 ... n-physics/

[wwu343]

Hi, i love the game but the fluid mechanics have always bothered me, mainly because I legitimately work in a factory that produces pulp, controlling tanks, flow rates and pumps. There are fundamental flaws with how fluids are handled in game.

Directional pressure and flow should be the name of the game... period. No fluid system would rely on a level equalization method to provide flow through a pipe. In the real world liquid is collected in a sump (hole in the ground) then pumped into a pipe or collected in a tank then pumped out or gravity fed out, which provides the pressure (think water tower).

One of the bigger things that has always stuck out is how tanks and pipes work; it makes no sense physically. In game if a pipe connected to a tank is half full, the system will try and reach an equilibrium so the tank is half full... the only problem is that the pipe would have to be the same height as the tank for that to be true. In reality a gravity fed pipe connected at the top of the tank would never fill the tank and one connected to the bottom of the tank would only fill it to the top of the pipe.

All tanks in the game should have inlets and outlets; you should need to fill the tank using back pressure (tanks are filled from the top) and then pumped out or then gravity fed out through the bottom of the tank.

All pipe systems should require a pump... make them cheaper to make, require them more. They should cause a directional pressure on the contents of the pipe (which could be shown graphically ie. "-> -> -> -> ->"), which influences both speed and direction of the flow.

This should make the game more realistic and more predictable on how liquids behave.

Thank you,
JE

How about making life easier on us and introducing an underground view so you can run all this piping under the factory, no need to have pipes go up and down out of the earth. (Think sim city style pipes).