[minor] Production stats are weird - consumption

[FrozenOne]

When i check production statistics, the 5s are belivable, but higher up, the numbers on consumption cant be correct, for example 1h shows consumption of 600 iron ore per minute, but only 29 iron plates per minute, while both should be almost the same. I think that higher times are divided by wrong numbers.

[SilverWarior]

You do realize that your furnaces have their internal storage?
That means that even ywhen your iron plate consumtion has stopped and all belts are full the furnaces might still be consuming irone ore for iron plate production till the point when their internal storage gets filled.

[FrozenOne]

Dude i'm not stupid.

(At least not in this one )

[SilverWarior]

Dude i'm not stupid.

It wasn't my intention to call you stupid or anything like that.
I must say that when I was observing production stats for the first time I also thought that they are wrong. But afte giving it quite some thoughts and bserving them some more I concluded that they are probably right.
But I still may be wrong.

Anywhay I suggest you make a test scenario using factorio editor to test this. It would be best if in this scenario you have ability to mass produce iron plates in a way that their consumtption doesn't affect the production speed. I recomend just simply storing all those iron plates in bunch of chests.

[FrozenOne]

In my base everything is fluent, no large storage, all made goes to next production step. Cant be 600 consumed ore and 29 consumed plates per minute. There is no way where 500 iron could disappear every minute.

[Nova]

Could you upload the save file?

[FrozenOne]

Pic 1: 5sec
Notice how on consumption, the graph below doesn't match the data and colors above. Is it due to poor sampling and badly weighted averaging? Consumption here seems ok, 1 copper ore -> 1 bar -> 2 wires.


Pic 2: 10min
Sustaining relatively same production as in pic 1, numbers on production are way different. Consumption is totally off between ore and plates. Also the graphs could use some smoothing, because they can be hardly useful with only uncountable spikes.


The save:
(dont mind the electric failures, those are due to 20 radars directly to east that fail to scan the hidden enemy base that is east of them - related to my other radar thread)
http://sc2.ic.cz/asdf.zip

[ficolas]

1 hour is not 1min*60, the factory needs to be running for 1 hour w/o modifications for this to be 100% correct, and if you take iron plates from belts/chests by hand, that is production but not comsumion.

[Hazard]

That still doesn't mean that you need to consume 15k iron ore to produce 1.3k iron plates though, that should be a 1:1 conversion.

[slpwnd]

AFAIK this is solved now (as of 0.9.6). If you still have some issues with production statistics please speak up.

[SilverWarior]

@developers

What approach are you using when calculating average production/consumption values?
I seriously hope you are not calculating average of averages of averages (average 10 hour production = average of last 10 hourly averages, hourly average = average of last 60 minute averages, etc.). if you do use such approach there is no wonder there are results discrepancies. Every such level could result in up to half percent of result discrepancy and raises eponentionaly with the number of levels used.

Anywhay I have spent several day figuring out a nice system which alows calculating average production/consumption for multiple intervals with no calculation errors which is verry quick and demands verry litle processing (the amount of processing does not depend on the number of imput values).
I'm also writing a tutorial about this for Pascal Game Development site I'm a member of. Unfortunately my laptop broke a few days ago so I don't have source code nor examples ready to show you. I explect my laptop be fixed before the weekend.
But if you are realy interested in this I can write a quick explanation of how this system works.

[kovarex]

Hello, the statistics are counted separately for all interval sizes, so this kind of error shouldn't appear.

[SilverWarior]

Hello, the statistics are counted separately for all interval sizes, so this kind of error shouldn't appear.

Can you explain that in more detail please?

[kovarex]

Ok, so we have 6 open "shelves" at each time.
One for 5s (1 tick interval) etc, there are 300 values for each graph.

When something is produced/consumed, I add to the number to all of the shelves.
When the interval of some shelf ends (every tick for 5s interval, once per 12 ticks for 10m interval), it is added to the circular buffer of the statistics of the interval and the shelf is cleaned.

This means I have 300 values for each item used and interval.
So if 40 item types were used in the production process, the amount of data would be 300*40*6 (interval types) * 2 (input/output) * 4 (float bytes) = 576Kb

[SilverWarior]

Ok, so we have 6 open "shelves" at each time.
One for 5s (1 tick interval) etc, there are 300 values for each graph.

When something is produced/consumed, I add to the number to all of the shelves.
When the interval of some shelf ends (every tick for 5s interval, once per 12 ticks for 10m interval), it is added to the circular buffer of the statistics of the interval and the shelf is cleaned.

This means I have 300 values for each item used and interval.
So if 40 item types were used in the production process, the amount of data would be 300*40*6 (interval types) * 2 (input/output) * 4 (float bytes) = 576Kb

What?
I'm sorry to say this but I have to. Your system is AWFULL!

My system looks like this:
I have general class for defining and storing data for time interval. This class contains next:
- array of items (this is what you call "shelf" I belive): This array contains the needed items to alow me to render the graph.
- starting position: Starting position defines which item in the array will be considered as first item. I use this in order to avoid actually moving item data becouse I'm always removing/replacing the oldest item with the new one (fixed array size).
- sum value of all the array items: Whenever I add a new item to the array I recalculate the sum of all items (I'll explain how and why later)
- average value: I also recalculate average value every time an item is added and then store this value so I can reuse it later (you refresh screen more often than you update your graph data)
- ItemSize: This value determines how many input values will be added together before creating one array item (larger the interval larger the value - I'll explain this better below).

So for 5s interval I would create a class with next properties:
Array of 12 Floating point numbers (32 bit of 4 bytes). This is enough data to render the 5s graph for duration of 1m.
Stating position is a 8 bit (1 byte) integer value.
Sum value is a 64 bit integer value
Average value is a 32 bit Floating point value.
Item size is 8 bit integer constant vith value of 1.

For 1m interval the class would have properties like so:
Array of 10 Floating point numbers. Enough data to render 1m graph for duration of 10m
Starting position, Sum value adn Average value are same as above
Item group size has now value of 12 (12 5s interval are needded for 1m interval).

etc.

So now how do I handle data storage (array handling)
In order for you to render a graph from stored items you need to make sure that those items are organized in your arrays (olders item at the start and newest item and the end).
When adding new items you would go and remove oldest item from the array first in order to maintain constant array size. But that means that every time you remove the first item you need to reposition all the oter items (lower their position by one). This results in quite some data movment necessary.
So in order for me to avoid this I added the Starting position variable which simply tels me for how much must I shift when I'm readin or writing items from the array.
Starting position of 0 means that first array items has ItemIndex = 0.
Starting position of 5 means that first array items has ItemIndex of 0+5=5. Ofcourse special checking is needed so that when desired item position and Start position exceds the array size the array size is substracted othevrice we would try to read or write out of array bounds.
So when adding a new value to the array I just overwrite the oldest item (last item based on using of above mentioned shift) and simply increase the Starting position by one.
When adding items to classes which are used for longer intervals each item value is actually sum of the lower interval items needed to form desired interval. Example:
First 1m item is a sum of first 12 5s items, second 1m item is sum of 13th to 24th 5s items, etc.

What is the Sum value used for and how do I manage it.
Sum value is a sum of all the array items. Normaly you would calculate it by reading and adding every array item value. But becouse that would be to slow I use a shortcut.
Instead of calculating sum of all items every time I update an array I simply reuse already calculated sum of all array items before update, substract the value of the array items I will be replacing and then add the new value. This alows me to update the sum value for any sized array with just two simple matemathical functions (one substraction and one addition) which makes this much fater than doing Number of items times addition.

How do I calculate average value
For the smallest 5s interval you probably already know: SumValue / NumberOfitems (Array size)
But how do I do it for larger intervals? SumValue / (NumberOfItems * ItemSize)
Now you are probably thinking to yourself: "Wait that can't be right!"
But it is. Why? Each item value in larger interval class array is the sum of several items from lower interval class.

As you see I'm storing much lesser amount of data. Actually I should say that I'm storing concentrated data instead.

So in the end my system would require something like this this:
5s interval: 12*4B= 48B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 62B
1m interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
10m interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
1h interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
10h interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
50h interval: 50*4B=200B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 214B
All together would be 492B for each item.
And that is much much lower memory requirement that your system needs

DISCLAIMER: I apologize for any posible typos but I was quite tired when writing this

[ssilk]

This is not a programming challenge, this is just make the things working in the shortest possible time.
Or in other words: do not optimize if not needed. Make it run in the easiest possible way, then optimize. This is much easier to find the bugs.

Kovarex did exactly that and with my own experience I would say, I would do it exactly the same.

[SilverWarior]

Kovarex did exactly that and with my own experience I would say, I would do it exactly the same.

Belive it or not but I understand why Kovarex did it the way he did even thou I probably woudn't have done it that way myself. But that is becouse I have a different coding style and coding preference. And result of my different coding style is that he has already made game while my game is still far from being finished as I spend much more time planning on writing a bit more optimized code.

Or in other words: do not optimize if not needed. Make it run in the easiest possible way, then optimize. This is much easier to find the bugs.

To tell you the truth I'm begining to dislike pepole who are saying this. Why? Becouse of such pepole we have tons of applications/programs/games which require verry powerfull computers just so you can run them. And many of these applications have not much more functionality than some of the similar applications which were made years back.
Wanna example? Winamp. Belive it or not but I'm still using Winamp 2.62. Why? Becouse version 2.62 is capable of playing mp3 music with barely 2 to 4 percent of my laptops CPU usage. Winamp 5 which I tried once on the other hand used around 30 to 40 percent of my laptops CPU to play same mp3 songs. And when you have laptop running on bater that could mean barely 45 minutes of listening to music in comparison to 2 hour when listening with Winamp 2.62.
So why is there such a big difference? Becouse pepole who made Winamp 5 didn't care about optimization since it wasn't necessary but pepole who made Winamp 2.62 had taken much care for optimization as it was compleetly necesary at that time.
And I'll tel you something else. I must admit that I kinda enjoj showin such pepole what can be done with just a litle optimization.

Anywhay the main reason why I even bothered with this is that recently there were a few bugs with ingame statistics so I wanted to try and understand why they are ocuring and how they could be solved.
But I have not explected that this would lead me to the pint where I would go and try to make the whole system from scratch on my own. But I did it and I'm proud of it.
And since it seems verry good to me and I like developers of Factorio a lot I decided that I'll share my design here even before I manage to share it on Pascal Game Development comunity (which I'm a proud member of being a Pascal programmer myself) for which I'm writing a full blown tutorial about how to make this system.

[kovarex]

Hello,

• About the array handling, I wrote, that I use CircularBuffer right? (http://www.boost.org/doc/libs/1_55_0/do ... uffer.html), it is basically the same as your custom structure, but using structures like this, that are already debugged and have all the standard STL interface is much better than making custom structures that would just mess the code, I guess you would write your own vector.
• In your calculation of memory requirements, you use just 12 values (instead of 300) for the graph, 12 values would just make the graph much more ugly, it is natural that you can save space by making the number of values lower.
• In your calculation you requirements is just for 1 item (out of the 40 that I guessed) and for only input (you need input/output).
• So your 492B needs to be multiplied by 40, then by 25 to get make it precise enough and then by 2 to have input output 492*40*25*2 = 984Kb
• Why would I keep the average in the data? I could do it, but there is no need for it, as I need to average only when I'm showing the graph and I need to go through the values to show the graph anyway. So when I go through the data, and I have it in cache, another += while doing so for the elements we have really no effect on the speed at all. If I spent the time optimizing this on optimizing for example, making inserters go to sleep when they are grabbing from belts I the ratio of spent time vs CPU time saved would be at least 100 times bigger. And that is the point of optimisations, you always need to optimize the things that are the most important in the start.

[ssilk]

you always need to optimize the things that are the most important in the start.

Exactly.

@SilverWarior: Ok, there are some examples, which needs to be optimized in the first implementation, just because of such calculations. But the general case and much more useful approach is not to optimize and look, what is the most slowest for this case.

And to your example with winamp: They changed too much programmers, added too much fancy, unneeded stuff, and did no optimization.

[SilverWarior]

I must humby admit that both of you guys are correct.
I have havily overestimated my design. That happens when you are doing math by hand without actual working example (my laptop is still being repaired).

The reason why I would also store calculated average is to avoid the need of recalculating it evry tie the graph needs to be rendered. This could save some processing power when rendering graphs with larger time intervals. But due the fact that my average calculation method is quite fast (I belive) not storing it and recalculating it everytime and not only on graph data uptate probably wouldn't cause to much performance loss.
Anywhay this was one of the first optimization decisions I came up while I was still using rather slow average calculation method (walking through array and adding up all the items every time) which brought quite a bit of performance increase at the time.