kovarex wrote:Ok, so we have 6 open "shelves" at each time.
One for 5s (1 tick interval) etc, there are 300 values for each graph.
When something is produced/consumed, I add to the number to all of the shelves.
When the interval of some shelf ends (every tick for 5s interval, once per 12 ticks for 10m interval), it is added to the circular buffer of the statistics of the interval and the shelf is cleaned.
This means I have 300 values for each item used and interval.
So if 40 item types were used in the production process, the amount of data would be 300*40*6 (interval types) * 2 (input/output) * 4 (float bytes) = 576Kb
What?
I'm sorry to say this but I have to. Your system is AWFULL!
My system looks like this:
I have general class for defining and storing data for time interval. This class contains next:
- array of items (this is what you call "shelf" I belive): This array contains the needed items to alow me to render the graph.
- starting position: Starting position defines which item in the array will be considered as first item. I use this in order to avoid actually moving item data becouse I'm always removing/replacing the oldest item with the new one (fixed array size).
- sum value of all the array items: Whenever I add a new item to the array I recalculate the sum of all items (I'll explain how and why later)
- average value: I also recalculate average value every time an item is added and then store this value so I can reuse it later (you refresh screen more often than you update your graph data)
- ItemSize: This value determines how many input values will be added together before creating one array item (larger the interval larger the value - I'll explain this better below).
So for 5s interval I would create a class with next properties:
Array of 12 Floating point numbers (32 bit of 4 bytes). This is enough data to render the 5s graph for duration of 1m.
Stating position is a 8 bit (1 byte) integer value.
Sum value is a 64 bit integer value
Average value is a 32 bit Floating point value.
Item size is 8 bit integer constant vith value of 1.
For 1m interval the class would have properties like so:
Array of 10 Floating point numbers. Enough data to render 1m graph for duration of 10m
Starting position, Sum value adn Average value are same as above
Item group size has now value of 12 (12 5s interval are needded for 1m interval).
etc.
So now how do I handle data storage (array handling)
In order for you to render a graph from stored items you need to make sure that those items are organized in your arrays (olders item at the start and newest item and the end).
When adding new items you would go and remove oldest item from the array first in order to maintain constant array size. But that means that every time you remove the first item you need to reposition all the oter items (lower their position by one). This results in quite some data movment necessary.
So in order for me to avoid this I added the Starting position variable which simply tels me for how much must I shift when I'm readin or writing items from the array.
Starting position of 0 means that first array items has ItemIndex = 0.
Starting position of 5 means that first array items has ItemIndex of 0+5=5. Ofcourse special checking is needed so that when desired item position and Start position exceds the array size the array size is substracted othevrice we would try to read or write out of array bounds.
So when adding a new value to the array I just overwrite the oldest item (last item based on using of above mentioned shift) and simply increase the Starting position by one.
When adding items to classes which are used for longer intervals each item value is actually sum of the lower interval items needed to form desired interval. Example:
First 1m item is a sum of first 12 5s items, second 1m item is sum of 13th to 24th 5s items, etc.
What is the Sum value used for and how do I manage it.
Sum value is a sum of all the array items. Normaly you would calculate it by reading and adding every array item value. But becouse that would be to slow I use a shortcut.
Instead of calculating sum of all items every time I update an array I simply reuse already calculated sum of all array items before update, substract the value of the array items I will be replacing and then add the new value. This alows me to update the sum value for any sized array with just two simple matemathical functions (one substraction and one addition) which makes this much fater than doing Number of items times addition.
How do I calculate average value
For the smallest 5s interval you probably already know: SumValue / NumberOfitems (Array size)
But how do I do it for larger intervals? SumValue / (NumberOfItems * ItemSize)
Now you are probably thinking to yourself: "Wait that can't be right!"
But it is. Why? Each item value in larger interval class array is the sum of several items from lower interval class.
As you see I'm storing much lesser amount of data. Actually I should say that I'm storing concentrated data instead.
So in the end my system would require something like this this:
5s interval: 12*4B= 48B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 62B
1m interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
10m interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
1h interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
10h interval: 10*4B=40B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 54B
50h interval: 50*4B=200B (Array) + 1B (starting position value) + 8B (Sum value) + 4B (average value) + 1B (ItemSize) = 214B
All together would be 492B for each item.
And that is much much lower memory requirement that your system needs
DISCLAIMER: I apologize for any posible typos but I was quite tired when writing this
)
