Friday Facts #203 - Logistic buffer chest

[deef0000dragon1]

Decouple draw logic.

It already is.

Decouple routing decisions. Keep a separate path network (robots, TRAINS) and handle pathfinding on separate threads.

Not feasible. Routes become invalid during game updates which means you can't calculate a path while the game updates. Routes also effect the cost of the next route so they can't be done in parallel.

Decouple AI At least high level (i.e. groups of biters).

Exact same problem as trains.

I am seeing a problem with what is going on here. As you go further and further into development, you are locking more and more behind single threaded only processes. As the base grows larger, it (obviously) gets harder and harder for a single core to process everything that it needs. Considering that most people have more processing available to them than a single core, this problem has a rather simple (sounding) and obvious solution.

I dont know a single person who has a single core CPU, and most of the gamers that I know have 4 threads or more. You have said that you want to move towards multi-threading Factorio, but as time goes on, you keep making it harder. The optimizations that you are spending a large amount of time on today, while nice, are going to be a problem in the future when you rewrite the engine. Its already going to be hard to rewrite, but you keep making it harder. You need to either start writing with threading in mind, or you need to define a point that you are going to rewrite the engine. It needs to take priority.

If you want my opinion, the engine rewrite should be PART OF 0.16, or 0.17 at the latest, and probably THE ONLY thing in that particular update. At the worst, Factorio should not have a 1.0 version until the main engine is threaded.

I know easier said than done, but it needs to be done at some point anyway, so that doesn't matter.

Oh, and considering that AMD is now pushing Intel, I have the sneaking suspicion that we are going to see a lot more people with 4-8 cores in the future. Best get on that boat now.

[brunzenstein]

What kind of bugs me is high CPU usage when the game is paused in the background. Yeah, it's "useful" in winter but I can use other commands for that...

Are you on mac/linux? For me (on Windows) it uses almost no CPU when paused.

I'm using the fastet MacBook Pro on the market - in pause mode Factorio still uses above 20% CPU power - on a very small map
On a larger map 130% when running but also about 60% CPU power in sleep mode thought...

[Rhamphoryncus]

Any chance you'll reconsider the issue of construction bots ignoring/not prioritizing storage chest items? Sure, I could work around that now by having a set of buffer chests with 1 of every placeable item in the game, but that's an ugly workaround for what I'd expect to be basic behaviour.

Existing thread: viewtopic.php?f=6&t=50416

[Braxus]

Thx i need them so much.

[Rseding91]

If you want my opinion, the engine rewrite should be PART OF 0.16, or 0.17 at the latest, and probably THE ONLY thing in that particular update. At the worst, Factorio should not have a 1.0 version until the main engine is threaded.

The engine has nothing to do with threading. It's entity <> entity logic - if you want a game that isn't an Excel sheet simulator entities will effect other entities and if they do that then they don't work in threaded environments.

You speak as if you're reciting something you read online. I'm suspecting you aren't a programmer (are you?) It's not a matter of "it's hard", it's a matter of "it doesn't work".

[deef0000dragon1]

If you want my opinion, the engine rewrite should be PART OF 0.16, or 0.17 at the latest, and probably THE ONLY thing in that particular update. At the worst, Factorio should not have a 1.0 version until the main engine is threaded.

The engine has nothing to do with threading. It's entity <> entity logic - if you want a game that isn't an Excel sheet simulator entities will effect other entities and if they do that then they don't work in threaded environments.

You speak as if you're reciting something you read online. I'm suspecting you aren't a programmer (are you?) It's not a matter of "it's hard", it's a matter of "it doesn't work".

I am a programmer, but I (obviously) dont have enough understanding of your code to really appreciate what can and cant be threaded. I apologize for my original post. I made a number of assumptions that I should not have.

I should note that I have never worked on a game that got to this scale, and have only done thought experiments on how I would thread something like the factorio. I felt that if I could come up with (what I thought to be) an easily thread-able engine that I am not the first, and that it should be implementable. I was too big headed in assuming that what I came up with would work without touching any code or doing any testing. My apologies. (I still feel that you need to strive to thread what you can as soon as possible.)

[jo2k]

I'm using the fastet MacBook Pro on the market - in pause mode Factorio still uses above 20% CPU power - on a very small map
On a larger map 130% when running but also about 60% CPU power in sleep mode thought...

Ahh, you're doing it wrong. This is the proper way to play on a mac:

[Koub]

If you want my opinion, the engine rewrite should be PART OF 0.16, or 0.17 at the latest, and probably THE ONLY thing in that particular update. At the worst, Factorio should not have a 1.0 version until the main engine is threaded.

The engine has nothing to do with threading. It's entity <> entity logic - if you want a game that isn't an Excel sheet simulator entities will effect other entities and if they do that then they don't work in threaded environments.

You speak as if you're reciting something you read online. I'm suspecting you aren't a programmer (are you?) It's not a matter of "it's hard", it's a matter of "it doesn't work".

I am a programmer, but I (obviously) dont have enough understanding of your code to really appreciate what can and cant be threaded. I apologize for my original post. I made a number of assumptions that I should not have.

I should note that I have never worked on a game that got to this scale, and have only done thought experiments on how I would thread something like the factorio. I felt that if I could come up with (what I thought to be) an easily thread-able engine that I am not the first, and that it should be implementable. I was too big headed in assuming that what I came up with would work without touching any code or doing any testing. My apologies. (I still feel that you need to strive to thread what you can as soon as possible.)

In case you're interested in this subject, you should definitely read this topic. All of it. It's, I think, the most complete topic about multithreading and Factorio, with a lot of dev posting.

[vanatteveldt]

I'm using the fastet MacBook Pro on the market - in pause mode Factorio still uses above 20% CPU power - on a very small map
On a larger map 130% when running but also about 60% CPU power in sleep mode thought...

The spaghetti! The goggles do nothing!

[Keks]

Buffer chest yay,

always thought it's just me who would like something like that. Otherwise I figured there would have to be a very simple circuit solution and was to lazy to look into it it's just a minor issue anyway but I think that's what makes Factorio so much fun you never stop improving things even if it's something you could consider negligible.

I love you guys moreover, it has to be mutual since you do the things I want (most of the time) without me having to say (write) anything (most of the time) (and when you don't, you come around to my view sooner rather then later anyways (most of the time )).

You are amazing.

[nthexwn]

Buffer chests sound really useful, but I'm concerned about the added complexity for new players. People already get confused by having 5 different chest types. Adding another into the mix is definitely going to perplex people. Just look earlier in this thread for examples!

I've often thought that adding per-slot middle-mouse filters to chests (exactly the same as cargo wagons) would be a better solution to this problem. They'd allow players to explicitly configure what item types to place in specific chests and in what proportions. When bots/inserters placed other items in those chests they'd regard the filtered slots as occupied and wouldn't consider them when calculating available storage/capacity. The chest functionality (vanilla/storage/requester/passive_provider/active_provider) wouldn't have to change at all (from a player perspective) since it would only affect the insertion of items.

Two concerns I'd have with this approach versus the proposed buffer chest solution:

1.) Players might not know that it's there. As with cargo wagons there's no indication in the GUI that you can add filters to slots. Players would have to be explicitly told that they can do so by other players or by loading screen tips / tutorials / whatever. I feel that this is somewhat mitigated by players not needing such functionality until they're deep enough into the game that they're actively seeking out this sort of information on their own.

2.) I imagine that adding filters to chests would require a substantial amount of code refactoring. Rather than treating the logistics network as one continuous whole it might have to subdivided and maintained separately for each filtered item type. I can't really conjecture on the data structures used behind the scenes, but I do believe it would at least be possible to implement this without altering the big-O complexity.

I'd be very curious to hear the developers' thoughts on filtered chests since I'm sure they've already considered this approach and disregarded it for whatever reasons. From an outside perspective I still strongly feel that filters would be a better solution than these buffer chests.

EDIT: Now that I really think about it, I think I might see the problem with chest-filters: The filtered slots would have to be given a higher delivery priority to bots than the unfiltered slots, otherwise the bots would just continue to place items in the nearest available storage chests (filtered or not) instead of delivering them where you want them. I'm guessing that they currently deliver on a chest-by-chest basis and not a slot-by-slot basis. By making them consider individual slots you'd be expanding the problem space by 48x, and that probably would impact performance. With buffer chests they'd still just be considering entire chests, so it'd be on the same performance level that we have now. Is that maybe correct?

[Tinyboss]

Buffer chests sound really useful, but I'm concerned about the added complexity for new players. People already get confused by having 5 different chest types. Adding another into the mix is definitely going to perplex people. Just look earlier in this thread for examples!

By the time you've unlocked requester chests, I think you're at least an "intermediate" player. Sure, it's new, but is it any more perplexing than train signaling? And that's unlocked very early!

[QGamer]

I'm sorry, but I fail to see how the buffer chests would be better than adding filters to storage chests.

The buffer chest looks to me like a storage chest with filters, and I can't seem to tell the difference between the two. Could someone please enlighten me?

But I do agree the concept of a filtered storage/buffer chest is very useful. Thanks for adding it.

[Fridgeton]

Super excited to play around with this! But as someone else already mentioned:

We were also concerned of people segregating their logistic networks for more control, it seems to us it was a workaround to a problem we should fix.

Still need to do that when building perimeter wall. Since bots don't path too well around large areas of no coverage. They go straight for the destination and when they run out of power they go to closest roboport and that can be starting roboport. But buffer chests are nice addition that will help to keep things organized.

Are there any plans of adding something to restrict bots from leaving logistics network coverage? Perhaps a simple toggle on roboports?
Or a type of "robot stations" that bots will prefer to travel between, to form some kind of logistics travel-network (which could bring new nice bottlenecks for us to deal with ) for mid-distances (trains being better for long distances). Alternatively, such stations could be a way of building paths between separate logistics networks, without the stations providing logistics coverage around them.

The latter suggestion is obviously a bit of a bigger task, but if nothing else it could be a suggestion for a feature in an upcoming expansion.

The robot pathing is a bit of an annoyance when building non-convex bases, with lots of biters outside of the walls.

[jormaig]

I've been waiting many time for something like the buffer chest

[impetus maximus]

fixing the supply demand loops sounds awesome for repair packs etc.

taken from Friday Facts...
"Typically when you set up all the provider chests, they are spread out across your whole factory. When you return to base for a resupply, you end up waiting for a long time while the bots travel from all over the factory with items. By using a buffer chest, you can setup a dedicated 'supply area', where the buffer chest will already contain all the typical items, and the bots can quickly top-up your inventory."

i assume the buffer chests will unlock before active provider chests then?

[torne]

I'm sorry, but I fail to see how the buffer chests would be better than adding filters to storage chests.

The buffer chest looks to me like a storage chest with filters, and I can't seem to tell the difference between the two. Could someone please enlighten me?

But I do agree the concept of a filtered storage/buffer chest is very useful. Thanks for adding it.

Adding filters to storage chests wouldn't compel robots to actually fill up the storage chests with the desired item - it would just prevent them from putting the *wrong* items there. The buffer chests will cause the robots to actually bring the requested number to that place, from providers or storage, which is useful to replenish supplies in a particular area.

[Selvek]

Yay! Much appreciated

Will bots deliver extra parts from deconstruction orders into buffers above the number of items requested? Or will those always go straight to storage?

I'm wondering if these can be used to solve the old problem of when you want to recycle many of your yellow (or red) belts but still have some available for construction orders.

[exi2163]

I am a programmer, but I (obviously) dont have enough understanding of your code to really appreciate what can and cant be threaded. I apologize for my original post. I made a number of assumptions that I should not have.

I should note that I have never worked on a game that got to this scale, and have only done thought experiments on how I would thread something like the factorio. I felt that if I could come up with (what I thought to be) an easily thread-able engine that I am not the first, and that it should be implementable. I was too big headed in assuming that what I came up with would work without touching any code or doing any testing. My apologies. (I still feel that you need to strive to thread what you can as soon as possible.)

One of my biggest learning projects was a Java application i used which took a lot of time to run. It drew a sovereignity map for EVE Online which took about 18 minutes on my dual core pc back then.
It seems to have been written by some math-heavy guy who knows nothing about programming... i knew nothing about java but i gave it a try. Managed to cut the runtime to about 15 minutes with some array optimizations and caching windows.
I then went on and thought "oh i have 2 cores in my machine why cant it just use both" and i went down the rabbit hole. The programm used a ray-tracing approach to calculate the color of each pixel of the map from top to bottom. Because a pixel in the pre-calculation only needed the values of the row above it i could separate each line into multiple sectors and let that run on each core in its own thread.
So i wrote a thread handler and a main thread which handed out work units to the workers. Nobody could explain why but Javas synchronize did not do what it was expected to do and i had to add some custom logic to make sure each work unit is only executed once. Multithreading is no fun
After a few days i got the program to run correctly in about 9 minutes (Quad core was around 6mins).

Because my Java knowledge is limited i rewrote the algorithm in PHP (my "native" language). Runtime: 30 mins . I used the PHP program to profile the workload and identified a heavy used code section which could not be sped up within PHP (heavy array handling which results in many C-checks in the underlying intepreter code). So i wrote an interface which dumps the current array states in a file and feeds that into a small C programm (my very first and stil only C program ever). So i replaced 3 lines of PHP whith 300 lines of C and the runtime was down to 2-3 minutes (The intermediate memory state was about 50megs of ascii file).
After i spent all that work on it some guy came by and said "i got an idea". After a few hours he presented me his solution: using a python program it cut the runtime down to 8s. How? He created an alternative algorithm which resulted in the same output but used far less processing power as it iterated around the power blobs instead of raytracing every pixel.

If you want to get experience with algorithms and multithreading don't just think about it... go for it. The fastest way to learn something is by doing it and by making some mistakes on the way. There are many open source projects out there which are looking for help or just start your own. Learning that your way may not be fast or even feasible is part of the journey. Just dont try to solve problems 100% by thought, it won't work

[golfmiketango]

This logistic buffer idea is pretty great! I'm thrilled to hear that this is being looked at.

As described in the FFF, I've lately taken to using multiple networks and this fancy "managed logistic internet switch" gizmo to make inter-logistic-network item-flows more controllable. It allows me to micro-manage bot delivery and buffering much more precisely. But it only really works so long as major production for factor products occurs centrally and distribution follows a "waterfall" pattern.

Eventually, I believe, the thing I really want is to distribute production where I think products are likely to be needed, but then drop-ship items to where they're needed if I happen to have a glut in one place and a shortage in another. To make that work is.. well, hard.

As the game engine continues to improve, I envision an "even-later game" frontier of factorio: a "big data" era of gameplay where distribution (rather than scarcity) emerges as the biggest challenge of the game. If I let my imagination run wild, I see myself using circuit-controlled exponential-decay multivariate rolling-average time-series regression analysis to predict future demand and preemptively route items to my various production areas. In this fantasy, I spend much of my time seeking nice clean regression variables with high explanatory power and low autocorrelation, and am rewarded with automagically adaptive just-in-time logistic fulfillment, without wasteful buffering.

For this sort of thing, a circuit-controllable logistic buffer chest would presumably be a powerful tool.... Yet, I wonder: would a logistic buffer chest screw up the logistics stats? Cause unforeseen confusion? Incentivise logistic-network sprawl? Actually, I wonder if micro-logistic-area roboports might ultimately be an equally or even more powerful tool, to realize my "distribution tycoon" fantasy?

This is the proper way to play on a mac:

This is pretty funny until you notice that the guy appears to be keeping the corpse of a half-naked child in a potato sack under his sofa...