Friday Facts #271 - Fluid optimisations & GUI Style inspector

[T-A-R]

And add those posters to the box;)

edit: What are current dev thoughts about mixing fluids? I like the idea of the second fluid disappears while damaging your pipes, so fluid-mixing stop themselves. Preventing mixing of fluids is a fundamental part of fluid logististics. Multi fluid stations are cool to, so for example, water could be used to rinse your pipes between different fluid-flows

[OBXandos]

I for one really like the new logo. Have you thought about adding some color to it though? I would also be interested in buying those PAX rollups as posters.

[featherwinglove]

...before actually merging straight pipe sections into one fluid box which should improve things again.

I thought this was a long time ago. I maybe should boot up Factorio on the Conroe/Cedar (a system named after its CPU and GPU die codenames respectively) to have a weak enough system to test older versions against the technique, but I tend to push my pipes underground to reduce the number of fluid boxes, then often using more advanced materials (i.e. I hardly ever play without Bob's) to improve performance further. I don't do this for UPS reasons, but for factory performance reasons. I think that when this is done seriously, the number of adjacent fluid boxes which are merged are not farther apart from each other than the reach of the corresponding underground pipe for the pipe material. In this manner, should one make sufficiently long above ground pipelines (which is unlikely), the performance will scale in the same way as it currently does for underground pipes.

[Philip017]

How many fluid systems there are when you mix two different fluids in one pipe by misplacement?

Actually the other day I had an idea for solving the fluid mixing which imo solves it very easily. Once a fluid is run through an empty pipe, a fluid type is assigned to it's whole system and no other fluid can get in until the system is drained first to restart it.

so once a fluid is in that system, no other fluid can enter it.

[Avezo]

How many fluid systems there are when you mix two different fluids in one pipe by misplacement?

Actually the other day I had an idea for solving the fluid mixing which imo solves it very easily. Once a fluid is run through an empty pipe, a fluid type is assigned to it's whole system and no other fluid can get in until the system is drained first to restart it.

so once a fluid is in that system, no other fluid can enter it.

Not sure if I get it - so what will happen if there are two separate fluid-filled pipe systems and a new single pipe section is put between them, connecting them? Which fluid will it be assigned with?

And what if there is a water pipe system, we drain it fully and then want to use it again without rebuilding as an oil system? Do empty pipes get reassigned? Either way it still creates possible problems, hope this new model have test cases in place for them.

[FasterJump]

The only difference is the saturation of color. I just added some sugar to the Factorio look.

This looks great, why not make a console command that could tweak the saturation?

[Nightinggale]

One thing I find really interesting is that people in this thread focuses on the poster graphics while if you read the comments on Xterminator's commentary on youtube (he does that for all FFF), people talk about "reasonable i7-4790k" as well as comments in general on CPUs and Factorio.

I agree that the i-4790k is not your average CPU. I have one and it's an absolute beast. When Apple released their new macs in November, the fastest you could get became a core i7-8700B. I saw Geekbench scores for it and decided to test my own setup for comparison. It turns out my computer won. Granted not by much, but my 4 year old system with 4 cores beat a brand new 6 core flagship in multi core testing. There was no content for single core performance. It could be interesting to see scores with after market cooling through, but it doesn't change the fact that i-4790k rules in single core performance against the vast majority of CPUs.

So what does it mean that the benchmarks were done with a CPU, which is faster than the average player's hardware? In a sense it's actually a good thing. When you speed up the CPU, memory latency becomes more of an issue. If it was the other way around like a CPU with poor single core performance, but insanely fast memory, then the results would get a lot more noise from the CPU trying to keep up. This means even if it might not be representative for the average player, it seems like a very good choice for testing code where performance depends on memory latency.

One thing I haven't seen anybody commenting on (in writing) is the part about placing fluid boxes in sequential memory. I have a feeling people don't realize the impact from this. The time it takes the CPU to read a new fluidbox from memory can easily be reduced by something like 300 CPU cycles. Multiply this by the number of fluidboxes on the map and multiply this with ticks/second and you have how many CPU cycles the CPU saves every second from just accessing fluidboxes. This is actually rather impressive when you consider that this is before enabling more than one CPU core for the task and there is no chance in what is calculated. It's purely reducing the amount of time the CPU has to idle.

I have to say that I'm rather impressed with both the memory I/O improvement and multi core support. As somebody trying to improve those two in existing code, I know it's far from easy to do in already working code when you don't want to change functionality and most certainly don't want to introduce bugs because you overlooked that some other system is also affected by the code you are working on. While it might be overlooked by the majority, I still want you to know your hard work is appreciated as well as you taking your time to tell about it.

Now the only "problem" is that I haven't really run into UPS problems myself, making me wonder how much of a megabase I need to build before I can tell a difference.

[Cadde]

words...

Now the only "problem" is that I haven't really run into UPS problems myself, making me wonder how much of a megabase I need to build before I can tell a difference.

For me personally, my biggest wish is for the game to not run the graphics update (side scrolling, regardless of if the player is moving or not) in locked step with the simulation.
Because it really hurts my eyes when the game runs at 60 "ish" FPS and my screen is 144 hz.
That "ish" fps is even worse btw. Because you can really tell when the graphics update changes as either the internal clocks become out of sync with the intended 60 FPS update or the game bogs down. The latter is "my fault" but even so, if the game focused more beef on keeping the graphics running smooth GTA style then it wouldn't bother me that much.

I get it, the game is pushing a lot of stuff around in memory and loading/releasing resources all the time to be able to have such a rich environment. But no matter what i do, the game just seems to love getting the jaggies for no apparent reason.
I've never run into UPS issues either. But even from day one, the game has had the jaggies.

[Nightinggale]

I just realized something. The new pipe system will reward designs, which can segment pipe systems into multiple parts as it will utilize more CPU cores. Splitting a giant pipe system into multiple systems would take an entity, which inputs and outputs liquid. Wouldn't it be as simple as adding a pump? If you are one of those who connect a pipe to all oil fields to gather crude oil, wouldn't that pipe be calculated on multiple cores just by adding pumps? Considering that would be something the player would likely do for game mechanics, that sounds like a huge benefit.

Because it really hurts my eyes when the game runs at 60 "ish" FPS and my screen is 144 hz.
That "ish" fps is even worse btw. Because you can really tell when the graphics update changes as either the internal clocks become out of sync with the intended 60 FPS update or the game bogs down. The latter is "my fault" but even so, if the game focused more beef on keeping the graphics running smooth GTA style then it wouldn't bother me that much.

Uneven time between frames is viewed as unstable images. Fixed 60 FPS is better than 62-70 FPS when it comes to visual impression.

It sounds to me like what you need is either to switch the monitor to 60 Hz or make an official feature request to Factorio regarding a setting to optionally cap UPS to some custom value. The important part is that you get consistent timing between frames.

While I agree this could be a significant issue for some people, I'm not sure it relates directly to pipe updates. Getting a few more UPS more will likely not help you much as it won't fix the variation in time between frames. If anything it might actually make it worse.

[pleegwat]

Splitting networks only has effect if you aren't yet loading all cores. Most systems have at most 8 logical cores, and the example refinery setup already has 5 fluid systems. Add the lubricant and sulphuric acid systems and the non-fluid thread and you're already at 8.

[Drury]

Splitting networks only has effect if you aren't yet loading all cores. Most systems have at most 8 logical cores, and the example refinery setup already has 5 fluid systems. Add the lubricant and sulphuric acid systems and the non-fluid thread and you're already at 8.

Presumably with further optimization the systems won't update in every tick, freeing up the core in the meantime.

[Klonan]

Splitting networks only has effect if you aren't yet loading all cores. Most systems have at most 8 logical cores, and the example refinery setup already has 5 fluid systems. Add the lubricant and sulphuric acid systems and the non-fluid thread and you're already at 8.

It is still always faster, with 2 cores the fluid update will be 2x as fast, regardless of how many total fluid systems there are.

The only cases where it will match the old speed (Aside all the other optimizations), will be if you have only a single massive fluid system, which will be single-threaded.

However most factories have at least 20+ individual systems, so will always benefit from the parallisation

[Twisted_Code]

you made your official company logo, the main reason you stated for not having a Developer Page on Steam so we could nominate you for Best Developer, less than a week after nominations closed? :-/

[Cadde]

It sounds to me like what you need is either to switch the monitor to 60 Hz or make an official feature request to Factorio regarding a setting to optionally cap UPS to some custom value. The important part is that you get consistent timing between frames.

While I agree this could be a significant issue for some people, I'm not sure it relates directly to pipe updates. Getting a few more UPS more will likely not help you much as it won't fix the variation in time between frames. If anything it might actually make it worse.

I already tried 60 hz and the issue isn't with the monitor refresh rate but rather that the game waits for the next frame. It's really hard to match 60 frames/second in software consistently because of how we split seconds. 1000/60 = 16.66666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666666..........
You never really can get that kind of accuracy. Only on monitors that have a dedicated hardware clock exactly tuned to that frequency.

I don't know the resolution of their timers but if they are using float, the best they can do is "16.6666660308837890625"
Or down to the 6th decimal. Once you are out of sync, you are out of sync. Then you wait another 16.666... milliseconds to get in sync again.
The game just gets into this perpetual sync catch up mode sometimes it seems.

My wish therefore is that the games graphics were decoupled from the simulations fixed step. So the graphics would pump as fast as it could with the latest data it has and some predictions (camera is moving left, keep moving left until we have an update in data)

Of course pipe updates doesn't have anything to do with it. It happens on a newly started vanilla game without anything built yet. And i am overly sensitive to frame jitters. I can easily tell when a single frame drops.

[Nightinggale]

Splitting networks only has effect if you aren't yet loading all cores. Most systems have at most 8 logical cores, and the example refinery setup already has 5 fluid systems. Add the lubricant and sulphuric acid systems and the non-fluid thread and you're already at 8.

It's not that simple. Say we have 8 cores and 8 fluid systems. Sounds perfect right? No, one can be 2000 pipes while another is just 4 pipes. This means in no time you have an idle CPU core just waiting for the other cores to finish. In other to scale tasks well between multiple CPU cores, you want more tasks than cores. In fact you often want a lot more tasks than cores.

If you use a library like Threading Building Blocks, then you can just add a bunch of tasks and let the library handle load balancing between cores to get the maximum overall performance. It does however work best if the number of tasks is way higher than the number of cores.

Besides don't look at the fluid system as the only thing running. If I understand it correctly, the fluid system runs in parallel to all other tasks. This mean even if the fluid system leave a core idle, it might not actually be idle because some other game mechanic is using it.

[RaviorMetal]

Hi,

Got a question: Whats this bronce looking pipes? I'm not sure that I've seen that before. Mabye I missed something?

bronce_stuff.png (397.98 KiB) Viewed 7338 times

[Nova]

That are heat pipes, used in nuclear setups to transfer the heat of the reactor to generate steam for energy production.

[pleegwat]

Splitting networks only has effect if you aren't yet loading all cores. Most systems have at most 8 logical cores, and the example refinery setup already has 5 fluid systems. Add the lubricant and sulphuric acid systems and the non-fluid thread and you're already at 8.

It is still always faster, with 2 cores the fluid update will be 2x as fast, regardless of how many total fluid systems there are.

The only cases where it will match the old speed (Aside all the other optimizations), will be if you have only a single massive fluid system, which will be single-threaded.

However most factories have at least 20+ individual systems, so will always benefit from the parallisation

I mostly meant that at the poster above me who claimed splintering your fluid would get you a lot of performance. I think you'd run into diminishing returns quickly. Also it's not that slow, is it?

[fur_and_whiskers]

But the second advantage is even greater, as the fluid systems are now independent and fluid flow doesn't interfere with anything else in the map, their update can be completely parallelized without any worries. So we just did that.

From https://forums.kleientertainment.com/fo ... ng-295825/

Flow of gasses/liquids in pipes is now multithreaded for improved CPU usage

Interesting that both Factorio and Oxygen Not Included have announced they're doing this within a month of each other. Really hope its a result of colaboration, if so everyone wins.

[Lubricus]

words...

Now the only "problem" is that I haven't really run into UPS problems myself, making me wonder how much of a megabase I need to build before I can tell a difference.

For me personally, my biggest wish is for the game to not run the graphics update (side scrolling, regardless of if the player is moving or not) in locked step with the simulation.
Because it really hurts my eyes when the game runs at 60 "ish" FPS and my screen is 144 hz.
That "ish" fps is even worse btw. Because you can really tell when the graphics update changes as either the internal clocks become out of sync with the intended 60 FPS update or the game bogs down. The latter is "my fault" but even so, if the game focused more beef on keeping the graphics running smooth GTA style then it wouldn't bother me that much.

I get it, the game is pushing a lot of stuff around in memory and loading/releasing resources all the time to be able to have such a rich environment. But no matter what i do, the game just seems to love getting the jaggies for no apparent reason.
I've never run into UPS issues either. But even from day one, the game has had the jaggies.

Is there any changes in the image to render between game-ticks? I think your movement is part of the game logic and the ticks...
Could Menus and map-view be separate from the main game-loop?