[0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Bugs that are actually features.
apriori
Filter Inserter
Filter Inserter
Posts: 274
Joined: Thu Feb 18, 2016 8:13 pm
Contact:

[0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by apriori »

Save: https://yadi.sk/d/7ZdPkRtHpJk6D
Mods: https://yadi.sk/d/Lr9IkrAqpJk9r
Logs: https://yadi.sk/d/p7VD8_0QpPdFc
Debug mode screenshot
PC: i5-3570, 16GB RAM, Radeon R9-380 2GB. 64-bit Factorio v12.21. OS win 8.1, res FHD windowed

There is an assumption that the TPS/UPS decrease (60 -> 20) is associated with inserters taking a lot of time when adding items to big (1800 slots) containers.
Any code or mods posted by me are WTFPL, unless otherwise copyrights are specified.

Rseding91
Factorio Staff
Factorio Staff
Posts: 14150
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.12.21] UPS decrease when inserters put into large inv

Post by Rseding91 »

I don't consider this a bug as the mechanics are all working exactly as expected. You've simply cranked the numbers far beyond what they where intended to be which is (as you're seeing) hurting performance.

A more technical description of what's causing it to run slow: Every time an inserter looks to insert something into the large inventory it gets the item and it asks the inventory if it can insert any of it. The inventory logic iterates over each available inventory slot it has and checks if it's blank or the count is < the stack size of the item in the slot.

For an inventory with 1800 slots that means it has to do that check somewhere between 1 and 1800 times for each "can insert" check run against it. As the inventory gets fuller the stacks at the front start to return false for that check and so the number of checks per item inserted goes up.

There might be some potential for optimization here but the use case is so small that it's probably not worth the time to look into this. What you could do instead is have the assembling machines export into active provider chests which the robots will then move off into the next available logistic storage chest and then request those items in the final destination. The robots will auto-find a valid chest for items and auto-find valid chests with items for picking up. You can expand that far more without hurting performance since each inventory is smaller and doesn't have a constant stream of items being put into and out of it.
If you want to get ahold of me I'm almost always on Discord.

apriori
Filter Inserter
Filter Inserter
Posts: 274
Joined: Thu Feb 18, 2016 8:13 pm
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by apriori »

Ok, now I see... I'm not enough experienced factorio modder yet, but have a couple ideas about how to solve this issue - I should check it out. Is there a way to override standard inventory check method?

Look, if I have 1.5M plates in 10 warehouses of 1.5k slots each OR in 300 shests of 50 slots each - isn't it the same for game performance? In both cases 15k slots will be checked by inventory logic. Or I shouldn't have such stocks?
Any code or mods posted by me are WTFPL, unless otherwise copyrights are specified.

Koub
Global Moderator
Global Moderator
Posts: 7744
Joined: Fri May 30, 2014 8:54 am
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by Koub »

Imagine you have a continuous stream of ... iron plates (could be anything).

Case 1 : 10 warehouses, each has 1500 slots full.
An iron plate arrives in front of the first warehouse. The inserter checks ... 1501 times, and finds that 1501th slot is empty. Grab, insert.
Second iron plate, check 1501 times, insert at 1501 position
...
100th iron plate, check 1501 times, insert at 1501 position
You have computed 100x1501=150 100 checks for 100 plates

Case 2 : 300 chests, each with 50 slots full
An iron plate arrives in front of the first chest. The inserter checks ... 51 times, and finds that 51th slot is empty. Grab, insert.
Second iron plate, check 51 times, insert at 51 position
...
100th iron plate, check 51 times, insert at 51 position
You have computed 100x51=5 100 checks for 100 plates

Well it's just an exercice of thought, but it's the idea.
Koub - Please consider English is not my native language.

apriori
Filter Inserter
Filter Inserter
Posts: 274
Joined: Thu Feb 18, 2016 8:13 pm
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by apriori »

Yeah, thanks, I've understood the same right after posting my last message. That's why I'd like to test some cases of inventory remembering last empty slot or last used slot and item inserted into - most of all (my) chests are being used for the same items (this is for plates, that is for batteries, those are for coal and so on). IMHO it's not the best situation, when we can build monstrous factory but are not available to store its resources and products.
Anyway, I love this game! :)
Any code or mods posted by me are WTFPL, unless otherwise copyrights are specified.

Rseding91
Factorio Staff
Factorio Staff
Posts: 14150
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by Rseding91 »

apriori wrote:Ok, now I see... I'm not enough experienced factorio modder yet, but have a couple ideas about how to solve this issue - I should check it out. Is there a way to override standard inventory check method?

Look, if I have 1.5M plates in 10 warehouses of 1.5k slots each OR in 300 shests of 50 slots each - isn't it the same for game performance? In both cases 15k slots will be checked by inventory logic. Or I shouldn't have such stocks?
The logistic network knows which chests have space for a given item type VS. have empty slots so it can skip virtually all of those checks and go right to a chest that it knows has room. So, at most it does: final chest slot count number of checks when it goes and inserts the item(s).

There is no way to override the standard inventory check method. It's already incredibly fast - there's no system that could be setup in Lua that would execute faster because the Lua code would still have to call back into the C++ and execute the same checks that the C++ is already doing.
If you want to get ahold of me I'm almost always on Discord.

apriori
Filter Inserter
Filter Inserter
Posts: 274
Joined: Thu Feb 18, 2016 8:13 pm
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by apriori »

Rseding91 wrote: The logistic network knows which chests have space for a given item type VS. have empty slots so it can skip virtually all of those checks and go right to a chest that it knows has room. So, at most it does: final chest slot count number of checks when it goes and inserts the item(s).
It's enough to optimize my storage facility, thank you, Rseding91. As I undersrtood, it means, that I shouldn't put items into warehouses (any of them) using inserters - only with robots and only into yellow (storage) warehouses. Of course if I want to place a lot of them without causing UPS decrease. Correct?
Any code or mods posted by me are WTFPL, unless otherwise copyrights are specified.

clusterfack
Inserter
Inserter
Posts: 23
Joined: Wed Oct 12, 2016 1:44 am
Contact:

Re: [0.12.21] UPS decrease when inserters put into large inv

Post by clusterfack »

Rseding91 wrote:I don't consider this a bug as the mechanics are all working exactly as expected. You've simply cranked the numbers far beyond what they where intended to be which is (as you're seeing) hurting performance.

A more technical description of what's causing it to run slow: Every time an inserter looks to insert something into the large inventory it gets the item and it asks the inventory if it can insert any of it. The inventory logic iterates over each available inventory slot it has and checks if it's blank or the count is < the stack size of the item in the slot.

For an inventory with 1800 slots that means it has to do that check somewhere between 1 and 1800 times for each "can insert" check run against it. As the inventory gets fuller the stacks at the front start to return false for that check and so the number of checks per item inserted goes up.

There might be some potential for optimization here but the use case is so small that it's probably not worth the time to look into this. What you could do instead is have the assembling machines export into active provider chests which the robots will then move off into the next available logistic storage chest and then request those items in the final destination. The robots will auto-find a valid chest for items and auto-find valid chests with items for picking up. You can expand that far more without hurting performance since each inventory is smaller and doesn't have a constant stream of items being put into and out of it.
Sorry for the necro bump though I don't think that matters in this forum. However, from your description, the way inserters work seems rather naiive and not a very efficient/performant way for inserters to work, and particularly causes problems with mods like this. If you've already change this since then, that's fine, ignore this.

If not, allow me to propose to you how I would optimize them were I given your code, and a bit of free memory.

First, track the first empty slot of the chest on the chest entity.
Second, track the less than max size stacks within the chest using a dictionary that holds the types of all entities within the chest that have at least one entry that is less than max stack size. The dictionary points to a list containing all the slots that these <max stacks are holding.

If you want it to always insert in 'the first slot that can accept an entry', you make sure that when you add a value to the <max size stack list, it gets sorted into the correct position. Then you can compare entries of the list versus the tracking value of the first empty slot of the chest.

Or to be slightly faster, and use less memory you could have it insert using 'fill <max stacks first, then insert into first empty slot', which would require no sorting of the list whatsoever, and help keep the memory cost of the dictionary/lists much lower (on an aggregate scale) as well.

Rseding91
Factorio Staff
Factorio Staff
Posts: 14150
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.12.21] UPS decrease when inserters put into large inv

Post by Rseding91 »

clusterfack wrote:Sorry for the necro bump though I don't think that matters in this forum. However, from your description, the way inserters work seems rather naiive and not a very efficient/performant way for inserters to work, and particularly causes problems with mods like this. If you've already change this since then, that's fine, ignore this.

If not, allow me to propose to you how I would optimize them were I given your code, and a bit of free memory.

First, track the first empty slot of the chest on the chest entity.
Second, track the less than max size stacks within the chest using a dictionary that holds the types of all entities within the chest that have at least one entry that is less than max stack size. The dictionary points to a list containing all the slots that these <max stacks are holding.

If you want it to always insert in 'the first slot that can accept an entry', you make sure that when you add a value to the <max size stack list, it gets sorted into the correct position. Then you can compare entries of the list versus the tracking value of the first empty slot of the chest.

Or to be slightly faster, and use less memory you could have it insert using 'fill <max stacks first, then insert into first empty slot', which would require no sorting of the list whatsoever, and help keep the memory cost of the dictionary/lists much lower (on an aggregate scale) as well.
The cost of maintaining such a dictionary and indexing into it to find information is almost the same as simply iterating the inventory contents. In fact probably more because you have to maintain essentially the same data structure that the inventory has but the possible index size is *much* larger - all item types in the game (hundreds) for an inventory that's of size 50~ would be an incredible waste of memory.

Also, the inventory has nothing to do with the chest - the chest simply saves/loads/provides the inventory. All of the logic is done inside the inventory class.
If you want to get ahold of me I'm almost always on Discord.

clusterfack
Inserter
Inserter
Posts: 23
Joined: Wed Oct 12, 2016 1:44 am
Contact:

Re: [0.12.21] [Rseding91] UPS decrease when inserters put into large inv

Post by clusterfack »

Well naturally the dictionary would only hold keys for objects currently within the chest, so the maximum amount of entries in the dictionary would be the maximum amount of slots in the inventory, and that would only be assuming you filled every slot with a unique, less than max size stack. You don't need to initialize every possible entry to the dictionary, only maintain the keys of types which are holding stacks of less than max size.

Now whether maintaining it would be more costly, I don't initially believe that would be the case, as it would just hook into when an object of less than max size gets inserted/fully removed from a slot, when an object gets filled, when an object gets partially removed from max size to less than max size. And the logic done to maintain it with a single one of these operations is rather short. But it's not as though I have the capability to run a thorough profiling test of this myself with your source.

Post Reply

Return to “Not a bug”