Friday Facts #260 - New fluid system

[TheYeast]

Only one more post coming up!

We can now calculate flow rate, but the concern raised by Dominik was the slow response to changes in supply and demand.

The main issue I see is a very low reaction speed. For example - have a straight pipe from source to drain, source is pumping, drain is closed, pipe is full. The drain opens. [...]

I tried to make plausible that this is not the case by showing the solution to the wave equation, but to be honest, this was not entirely convincing: physicists like to reason about spherically symmetrical chickens in a vacuum. Or in this case an infinitely long pipe without sources or sinks messing everything up. A closed drain opening is horrific to otherwise simple differential equations. For the intermediate pipe between source and sink, the solution would still hold... mathematically.. but only if we allow negative amounts of fluid... we don't, adding more mess to the problem...

At this point, it has thus become easier to just demonstrate that there is no problem with sudden changes in supply/demand. I present to you a simple simulation of a straight length of pipe (I had no time to implement junctions).
https://jsfiddle.net/TheYeast/ypwsbftg/

Source and sink can be turned on and off. At the top of the javascript file, constants can be tinkered with.

Note that:

* The wave front travels at exactly 6 tiles per second, as I derived in my earlier post about the wave equation.
* The throughput settles at 1200/s for n=10, as calculated in my previous post about flow rate.
* Closing the sink will send a reflected wave back to the source, stopping fluid motion as it passes through.
* Opening the sink again will rapidly restore sink throughput. Initially this is provided by the reserve in the pipe...
* ..it takes about 1.5 second for the source to 'realise' it can start supplying again.
* The sink throughput will oscillate a bit before it settles, but this is nowhere near a slow reaction speed.

Feel free to toy around by changing pipe length, the physical constants cSquared and friction, and the amount of fluid a source/sink can handle per second.

Make a 100 segment pipe and see that opening a drain on a filled pipe does not need any time to react. The opposite in fact: the local reserve in the fully saturated pipe near the sink can provide fluid faster than the source can because of all the friction between them over the long distance. The intake will decline as the pipe empties, then rise again as the new flow from the source reaches the sink. And even during the temporary decline, the sink intake is quite substantial.
For extremely long pipes, if this does become a problem, the player may want to pump the fluid up into a storage tank to have a larger local reserve for sudden, large changes in demand.

That's it for me, as I'm on vacation the coming week. I will check the thread when I return.

@ Dominik, silkroadalice, pleegwat, PacifyerGrey, just_dont, Nova, Oktokolo and others I forgot: thanks for your appreciation of my proposal and posts!

TheYeast

[Oktokolo]

I really hope, that TheYeast's solution makes it into the game.
It looks like beeing more correct and less computationally expensive than the other solutions at the same time.

[Unrelentingloki]

Im not very familiar to fluid mechanics or flow of fluids in factorio, when i do pipes, i just make things and hope it works. but from reading your post on FFF. I wonder, if the Flow of liquid could work like blocks in a rail network. that would reduce the number of fluid boxes you would need to calculate. again im not too sure how it works, but instead of working it like.... the electric network and instant transmission, we are factorians not Z warriors. so perhaps working it like blocks in a rail network could solve the theories you seek?

[McDuff]

TheYeast's proposals seem like they would work, and like I can understand them, which is definitely a step up from the current pipe implementation.

I also think the idea of segmenting pipe networks seems like a good idea, and it appears that would also work.

Rather than calculating every junction pipe, would it be feasible to "chunk" junction blocks and treat several nearby junctions as the same pipe? This might also help with things like doubling up pipes for throughput? I don't know the right answer to that.

As far as not adding extra pieces I do absolutely agree with that, but the suggestion someone made to have a "straight" pipe that could only be connected 2 ways and a "junction" segment that could be connected up to 4 seems like it would be a good compromise? The junction bit would just be the existing pipe, in fact, with the straight pipe being a new variant.

[Bauer]

Another physicist here.

@Dominik:
I envy you for having to solve this problem! I can also imagine that TheYeast's approach is difficult for any non-physicist.

@TheYeast:
Thanks for the pleasure of reading your posts. This makes a lot of sense and seems to be computational feasible. This has my full support (if that helps to make it into the game).

I'm only a little bit worried though that the "unphysical" behaviour of liquid production and consumption (delta-functions) will lead to artifacts in complex networks that will be difficult to understand by the player. Assume a ring-tube with non-symetrical producers and consumers. The flow direction will be quite random and confusing because the speed waves following the delta-peak will travel in both directions of the ring and will not cancel out due to the missing symmetry. Or am I wrong?

[McDuff]

Bauer, that sounds like the sort of thing that would be easier to answer in a simulation than with the maths tbh. With TheYeast gone, I wonder if anyone else is able to work on his sim?

[FasterJump]

I think we can classify the models into 2 categories:

• Level based: in these models, the surface pressure is always the atmospheric pressure.
  -Reals aqueducts and rivers
  -Current factorio model
  -Dominik model
  -TheYeast model
   
• Pressure based: the "roof" of the pipe allow to have highter pressures.
  -Real petroleum pipelines
  -Real water piping
  -Pressure models
  -Electric model

pressures_5.png (6.28 KiB) Viewed 7273 times

[RocketManChronicles]

Would it really be a problem for storage tanks to have designated inlets and outlets? My thoughts are that these can act like check valves, that once you have a fluid flow into a tank, it cannot flow back. But more importantly, the pipes leading to the inlet will always empty into the storage tank. Then likewise, the pipes following a storage tank are 'full' when the fluid level in the tank is above 100 units (amount in pipe sections). Just an idea.

Also, I like the idea of a "suction" based system where the demand determines the flow from upstream. But I have been reading the thread so far and like the discussions.

However, this is complicated and I like what you have here in FFF. There is a lot of thought going on and I appreciate the listing of what was looked at and rejected along with what is being looked at and considered.

[Roodvlees]

@DaveMcW
I like your idea of single-directional pipes.
Although I do think it would make sense to change how they're built.
To how belts are built as well.

And maybe adding height as well.
Height would accentuate the directionality of pipes, since fluids naturally flow to lower levels faster.
This could work well with having underground pipes as well.
We'd need pumping stations to pump the fluids to higher levels to keep flow speed up though.

[Selvek]

Yay! Sounds like great improvements! Now we just need some method to have adjacent pipes not auto-connect to each other.

[Pyroatheist]

Hi there!

Before we begin: disclaimer. I've never played this game, but a friend of mine pointed me towards this blog post, and I had quite a lot to say about it. I've discussed the game mechanics pretty in-depth with him as it pertains to these pipes, so I think I have a decent enough handle on how this works to proceed.

Basic Concept

In real life, flow is driven by pressure. If you try to simulate flow without using pressure as your driving parameter (like using volume and velocity as noted in the blog post), you'll run into all sorts of issues down the road due to how those parameters don't inherently describe the behavior of the flow. Therefor, this implementation suggestion is based around using pressure as your primary driving parameter.

Basic Assumptions

To get started, we'll need to make a couple of assumptions. These cover some basic details about how this simulation gets implemented, and allow us to really simplify a lot of the calculation steps that are necessary.

Pipes start empty, but once pressurized they become and remain full for the duration of the use of the pipes.

This is a pretty important assumption because it's essential to doing reasonable calculations for flow in pipes and also is how pipes actually work. In real life, a pipe filled partially with water and partially with air isn't moving fluid: it's either unused or is a pressure bomb waiting to explode. Once you've turned the pressure on in a piping system, all the air will be forced out of the system and remain that way. No pipe system allows for backflow of air because it is both inefficient and extremely dangerous.

Minor and Major fluid losses will not be accounted for

Minor and Major losses are the names for what happens as fluid flows through a pipe. As it rushes around corners, passes down long tubes, or does spins while going through a T-bend, it loses pressure, and that's what drives a lot of pipe design. However, in a system like what factorio has, simulating these doesn't make a lot of sense. It would feel really bad to put a bunch of elbows on a pipe system to make it go where it needs to go and end up with a dribble at the end because of the minor losses. Therefor, this concept will assume that all minor and major losses in the pipe system are zero.

The Bernoulli Equation describes the flow

This probably doesn't mean a whole lot yet, but the implication is that the flow in these pipes can be described fully by a particular equation that is very well-behaved and easy to solve. This is a very reasonable assumption to make as it is true for the vast majority of pipe flow that any of us encounter regularly.

All pipes are flat *

It was explained to me that there is no elevation change in factorio. Therefor, all pipes operate at the same height, meaning that there will never be a difference in height between the inlet and the outlet of the pipe. I've included an asterisk here because once we discuss the implementation of tanks, a limited form of height may come in handy when applied specifically to them.

All outlets are ambient pressure **

Ok, so this might not mean a whole lot yet, but it will end up mattering. We're going to assume that all of our pipe outlets are at ambient pressure for this analysis. I've included a couple asterisks here because this is something that can be tweaked later on for some more complex simulation if we want to.

Ok, so that's it for our starting assumptions. Now, a more detailed dive on how fluid mechanics in pipes work, and how this helps us for implementation.

Pipe Implementation Details

Flow Velocity

In a pipe, flow is driven by just 2 parameters: pressure and density of the fluid. Normally there's more, but our previous assumptions cut it down to just those 2. The way you calculate flow through a pipe is by taking the starting and ending points of the pipe, finding out those 2 parameters, and calculating from there. Density of the fluid never changes, and that means that the only important parameter is pressure at the starting point and pressure at the ending point. Remember that we assumed earlier that pressure at the ending point is ambient, and so we already know one of those. This makes a very easy equation to solve, so I'll show it here.

Start: velocity = 0, pressure = PumpPressure, density = rho
Exit: velocity = unknown, pressure = ambient, density = rho

Velocity at exit = sqrt( 2 * (PumpPressure-ambient)/rho)

Ok, so that's pretty easy. There's really only 2 things that can change here: density and pump pressure. Density is the term you can use to play around with how different liquids act. A more dense liquid will flow slower and a less dense liquid will flow faster, so you can implement that to make different liquids feel different. Pump pressure is the other part of this. As you increase pressure, you increase the velocity of the flow. This is also where we reach our first pipe system physical limitation!

Maximum pressure is a limiting parameter that cannot be exceeded.

A pipe has a maximum pressure that it cannot increase past. This puts a hard limit on the velocity of the flow and allows you to implement a gameplay limit to how much liquid can be moved. It also allows you to do things like implement different kinds of pipes with higher pressure limits to allow for fast fluid flow.

Flow Rate

Right, so that tells us how fast the liquid is flowing. Now we need to turn it into what's called "volumetric flow rate", which is a description of how much liquid is being moved. This looks like "cubic feet per second", or something similar. For this we'll need another parameter: Pipe area.

Volumetric flow rate = Velocity * Pipe area

Alright, so that's really easy. However, there's a bit more to think about here. If you put junctions on your pipe, you're effectively increasing the amount of pipe area that you have, but you won't be getting a flow rate increase from that. The pipe area in this calculation is the smallest area in the system, making it our second pipe system physical limitation!

Maximum pipe area is limited by the smallest point in the system.

As you can see in the equation, the bigger the pipe, the more flow you get. This allows you, if you choose, to implement different size pipes that put different limits on the flow rate of the liquid. However, there's another hidden system limiting parameter here!

Maximum flow rate is limited by the source generation rate.

You very well might be able to build a pipe system that is capable of flowing faster than your source generates fluid. This is necessarily impossible, so the maximum flow rate simply needs to be limited by what the source can actually produce. In certain conditions, it could go over that (once we discuss tanks), but over the long term, this parameter will be a strict limiter, and will be applied to the system to cap the flow rate just like the pipe pressure limit caps the pressure in the system.

Junctions and Outlets

Next, lets talk about junctions and outlets. I saw from your post earlier that you want to basically divide the flow proportionally based on how many splits it's seen. The way this can be implemented is that each outlet will be given a specific proportion based on the upstream junctures. You can do a running tally of junctures once for each outlet to get this number. For each 2-way split, apply 1/2 multiplier. For each 3-way split, apply 1/3 multiplier, etc. Do this for each junction it hits passing upstream, and get the total proportion. This calculation only needs to be done once for each outlet for any given configuration of pipe, so once the system is built, you only have to redo that calculation if the upstream junctions change. Here's what the calculation looks like:

Outlet Flow Rate = Volumetric flow rate * Calculated Proportion

Again, pretty easy. You perform this calculation for each outlet point, and the total flow rate will be guaranteed to match the initial total flow rate. You don't have to worry specifically about how the upstream junctures are behaving because that's already accounted for by the proportion calculation.

Dead Ends and System Startup

First, dead ends. A dead end in a pipe is interesting. It's filled with fluid, it's pressurized, but the velocity of the flow is zero. This means that once the system is fully up and running, a dead end effectively stops existing, but it does matter for how long the system takes to start up. However, since it has no actual outlet, a dead end has absolutely zero relevance on any of the fluid flow calculations that we've gone through already, so that keeps it nice and clean.

Next, system startup. In order to start up the system in a fairly reasonable manner, the pipes first have to fill up before the flow gets going properly. There's a pretty simple way to simulate this though, and it only takes a couple of parameters. We need to know the total length of pipe in the system, and from this we calculate the volume.

Total system volume = Pipe Area * total system length

Earlier we calculated the volumetric flow rate, so now we apply that to the total system volume to figure out how long it takes the system to fill up.

System Fill Time = Total System Volume / Volumetric flow rate

Now that you've got the fill time, you can simply use all the pre-calculated outlet flow rates and scale them linearly from 0 to full across the duration of the startup time. Alternatively, you could do something like keep them zero for the first 50%, then scale them twice as fast for the 2nd 50% to kind of mock-up how you'll get more of a rush of liquid out of the pipe towards the end of the system pressurization. That implementation would be up to you, but both would be simple. The important thing to remember here is that this is a calculation that happens once, at system startup. Once the pipe system is fully pressurized, this transient simulation drops out, leaving us with just the previous steady calculations.

Pumps and Pressure

Lastly, we come to pumps and pressure. The purpose of a pump is to increase the pressure of a fluid. This drives flow through a system, and allows your whole scheme to run. There's really 2 main ways you could choose to implement this, and both of them would be appropriate. You could have the source machines generate an inherent pressure that drives the system or you could requires a pump to be placed on the pipe to drive the system. Alternatively, you could do both: source machines generate an inherent pressure, and inline pumps add pressure for faster flow. Regardless of how you implement it, this portion will generate that starting pressure value that drives this whole simulation. The strength of your pumps will allow you to control the strength that the players are able to start off their pipe system with.

Summary

Right, here's a quick summary of what I just discussed. First, the steady behavior of the system can be described with 3 basic equations.

• Velocity at exit = sqrt( 2 * (PumpPressure-ambient)/rho)
• Volumetric flow rate = Velocity * Pipe area
• Outlet Flow Rate = Volumetric flow rate * Calculated Proportion

These equations are quite simple and easy to calculate. Additionally, they describe the entire system without needing any granularity along the pipes and junctures past the initial setup of the calculations. Lastly, they only need to be recalculated when there is a change in one of the driving parameters, meaning that once you've done the initial system calculation, you can simply keep the output values as parameters for your larger factory without needing to recalculate them each cycle.

Next, the startup behavior of the system can be described with 2 more equations.

• Total system volume = Pipe Area * total system length
• System Fill Time = Total System Volume / Volumetric flow rate

Just like the others, these are both easy to calculate and only need to be performed once, at system startup.

As a whole, this is an extremely light-weight simulation to run. It describes the whole system with one set of equations, is easily adjusted to account for flow-limiting characteristics in the game mechanics, and properly describes the way a real-world system behaves (mostly anyway). While I'm sure your current implementation does work, I'm fairly confident that this implementation or a variant of it will be far more usable both from an in-game perspective and an actual development standpoint.

Final Comments

So some things in particular I haven't addressed yet are tanks and the 2 asterisked assumptions at the start. All 3 of those things basically go together, but they get us into a bit more of a complex realm. One of the nice things about this implementation, though, is that it naturally accounts for tanks and how they function if you choose to implement that. I'm not going to address that right now, but I can easily discuss that further if you want an explanation of how that could be done.

Feel free to question me or point out how I've brutally missed some basic mechanics of factorio here. I'm pretty confident that this is a good way to go about it, but this is far from a perfect implementation, so good suggestions always make improvements.

[Oktokolo]

Before we begin: disclaimer. I've never played this game, but a friend of mine pointed me towards this blog post, and I had quite a lot to say about it. I've discussed the game mechanics pretty in-depth with him as it pertains to these pipes, so I think I have a decent enough handle on how this works to proceed.

Sometimes i really admire the self-confidence of other people. I would certainly not have posted a proposal under such conditions. Especially not after reading that of the physicist who actually played the game, already talked about fluid dynamics with the devs a year ago, described a clever way for parallelizing his algorithm while also making the results predictable and provided a nice graphical simulation in JavaScript wich lets you test the algorithm with different fluid properties...

[Pyroatheist]

Before we begin: disclaimer. I've never played this game, but a friend of mine pointed me towards this blog post, and I had quite a lot to say about it. I've discussed the game mechanics pretty in-depth with him as it pertains to these pipes, so I think I have a decent enough handle on how this works to proceed.

Sometimes i really admire the self-confidence of other people. I would certainly not have posted a proposal under such conditions. Especially not after reading that of the physicist who actually played the game, already talked about fluid dynamics with the devs a year ago, described a clever way for parallelizing his algorithm while also making the results predictable and provided a nice graphical simulation in JavaScript wich lets you test the algorithm with different fluid properties...

So I read through that discussion, and I think it's a really interesting implementation. However, I think it actually goes a bit too deep in the sense of getting bogged down in things such as the viscosity of the liquid and the way pressure waves propagate. Here's why:

Pressure wave propagation isn't particularly important because pumps are not responsive to changes in flow rate. While it will take some amount of time for a valve opening to cause an actual movement effect upstream, the pump keeps doing its thing of providing a pressure gradient regardless of whether the downstream valve is open or shut. This lets you push the flow rate issue purely to a matter of whether you have enough flow rate throughout the entire system to supply the downstream demand instead of attempting to calculate how the upstream is going to provide it granularly as it flows through the pipe system.

The viscosity issue/friction issue is a bit knottier, but if we're making the assumption that Bernoulli holds true, as the physicist from earlier did, it heavily restricts the kind of wacky peanut-butter like substances that you can force through the pipes. While it's possible to do a simulation where that isn't true, it rapidly either leaves the realms of realism or starts requiring server farms to calculate. That being said, implementing Major losses into the simulation to display how fluids do slow down as they travel down a pipe would be a pretty minor adjustment, and could be brought into the overall calculation without disrupting the concept of a pressure-driven system at all.

[bobingabout]

Would it really be a problem for storage tanks to have designated inlets and outlets?

you can mod it easily enough. the problem however is that it's all one fluid box, so where as on an assembling machine where you set a base level of -1 on an input to force fluids to go inside it, and a base level of 1 on an output to force fluids to come out of it, I don't think you can do that with a single fluid box. So you'll probably want to end up using pumps anyway, just to make sure it fills up.

[McDuff]

I think the problem with Pyroatheist's suggestion here isn't that the model itself isn't functional or elegant, but that it doesn't seem to fit in with the way Factorio, specifically, does its calculations.

There's a requirement for each pipe to update, or at least for a pipe segment to update. And of course there are lots of fast demolish and rebuild stages. Factorio factories don't get built and then hit "run," there's no "start up" sequence per se, they're always on and they're reconfigured on the move.

[Zavian]

To be honest I stopped reading after

Pipes start empty, but once pressurized they become and remain full for the duration of the use of the pipes.

This is a pretty important assumption because it's essential to doing reasonable calculations for flow in pipes and also is how pipes actually work.

That assumption just doesn't apply to factorio. Far too often demand will exceed supply at least temporarily. When that happens pipes empty out. Even when demand equals supply, pipes are often only partially full.

[dee-]

At this point, it has thus become easier to just demonstrate that there is no problem with sudden changes in supply/demand. I present to you a simple simulation of a straight length of pipe (I had no time to implement junctions).

https://jsfiddle.net/TheYeast/ypwsbftg/

This is sweet!

As you already said it would benefit from junctions, tho.

[Engimage]

snip

Your assumptions are solid enough but there is one major failure in them
You have to be able to model every single pipe segment independently. Also consumers do not provide constant drain but do so in discrete cycles. You can see it as there are small tanks on the ends which are full most of time and then get halfed instantly allowing liquid to flow in until they get full again.
I do agree on a concept of making pipes full all time after being initially filled. Factorio concept of pipes being half full is generally bad idea driving people away from educational standpoint and realism overall. This should be the difference point in tanks vs pipes where tanks can be partially filled while pipes can not. And there should be no output at all while pipes are getting full.

After that you can operate with pressure and flow speed using any math complexity you want.

[murgo]

To be honest I stopped reading after

Pipes start empty, but once pressurized they become and remain full for the duration of the use of the pipes.

This is a pretty important assumption because it's essential to doing reasonable calculations for flow in pipes and also is how pipes actually work.

That assumption just doesn't apply to factorio. Far too often demand will exceed supply at least temporarily. When that happens pipes empty out. Even when demand equals supply, pipes are often only partially full.

Thing is, in real life the pipes will be full of water even if demand exceeds supply. If your apartment building has all taps running at max flow the pipes still will be full of water, the pressure just will drop. The pipe won't be "empty", and "empty" actually is air that needs to get into the pipe, not just absence of water.

Still, I like TheYeast's model even though I don't fully understand it, the JsFiddle was fun enough to play with and I think that kind of system would be nice for Factorio. It would be great to see the junctions in work in the proof of concept.

[Pyroatheist]

To be honest I stopped reading after

Pipes start empty, but once pressurized they become and remain full for the duration of the use of the pipes.

This is a pretty important assumption because it's essential to doing reasonable calculations for flow in pipes and also is how pipes actually work.

That assumption just doesn't apply to factorio. Far too often demand will exceed supply at least temporarily. When that happens pipes empty out. Even when demand equals supply, pipes are often only partially full.

So I understand how it's implemented now, but that's why I made that assumption. The system right now is absolutely not a system of pipes. It's an oddball implementation that is somewhere between a series of linked tanks and an open aqueduct. It doesn't even remotely resemble a pipe system, and that's why the implementation here has been so difficult to figure out. A series of linked tanks that is also an aqueduct does not behave in a fashion that's predictable at all, and so these extremely granular and kinda hacky calculations have to make up the difference.

As someone else mentioned, when demand exceeds supply, you don't empty out a pipe. It simply loses flow rate and stops flowing, but it doesn't backfill the pipes with air. That's why that assumption was stated.

snip

Your assumptions are solid enough but there is one major failure in them
You have to be able to model every single pipe segment independently. Also consumers do not provide constant drain but do so in discrete cycles. You can see it as there are small tanks on the ends which are full most of time and then get halfed instantly allowing liquid to flow in until they get full again.
I do agree on a concept of making pipes full all time after being initially filled. Factorio concept of pipes being half full is generally bad idea driving people away from educational standpoint and realism overall. This should be the difference point in tanks vs pipes where tanks can be partially filled while pipes can not. And there should be no output at all while pipes are getting full.

After that you can operate with pressure and flow speed using any math complexity you want.

So this makes sense, but is easily solve by slapping an appropriately small tank on both the inlet and all outlets of the system. Let the tank be filled up by a discrete quantity transferred per tick, and let the machines withdraw a discrete quantity once the tank has enough in it, emptying the tank. Everything in-between can still be cleanly described as a pressurized pipe system, and this system does contain the ability to model tanks in-line with the system as well, if you go a bit more complex with it.