Page 1 of 1

[0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Sun Mar 22, 2015 5:12 am
by ZorbaTHut
First, some math:

It takes 0.5s to produce Copper Cable, and you get 2 of them in output. The speed of an Assembling Machine 2 is 0.75. Therefore, the production of three Assembling Machine 2's should be 3*2/0.5*0.75*60=540 items per minute.

It takes 0.5s to produce Electronic Circuit. Two Assembling Machine 2's should produce them at a rate of 2/0.5*0.75*60=180 items per minute.

Finally, smelting an Iron Plate takes 3.5 seconds. An Electric Furnace runs at 2x speed, so a single Electric Furnace should be able to produce Iron Plates at a rate of 2/3.5*60=34.29 (approx) items per minute.

Now let's look at the actual game. Here is a game in which I've been doing that (click the "download" icon at the top) - you'll need Y.Petremann's Creative Mode mod and rk84's Test Mode mod in order to run. As you can see:

Image

this is a very simple "factory" containing all three of those production chains.

So now let's look at the P window. 5s is too inconsistent to get useful numbers out; let's skip ahead to 1m.

Image

As you can see, something has already gone wrong. Each production is 11% too fast. But it gets weirder! Check out the 10m stats next:

Image

Now we're about 1.5% too fast. But we're not done yet . . .

Image

Now we're about 1.1% too *slow*. 10h makes things worse:

Image

2.5% too slow. I haven't sat down to try 50h yet, but I'm expecting 3-4% too slow. Oh, and let's take a look at 5s again!

Image

Yeah, that's right - it's wrong by about 45%. Holy moly. Look at that - it says it's completed 66 copper wire, when 1m says 600 copper wire! That means there are less than 50 seconds in a minute! Something is royally broken here.

Note that most of these numbers were gathered with "/c game.speed = 32". However, I assume this uses the same game speed code that the replay system does, and that system would break catastrophically if there were flaws in it. Changing the game speed to normal doesn't cause any visual discontinuities in any graph, nor does it change the numbers on the low-duration graphs, so I'm guessing the game speed isn't a factor here.

So, aside from the obvious bug report, I've got two questions:

* Are any of the production charts accurate?

* Are the producers constructing at the right speed, or is the entire game slightly mistimed?

Re: [0.11.19] Production window stats highly inaccurate

Posted: Mon Mar 23, 2015 4:19 pm
by SHiRKiT
This is really interesting, I never thought that the Production Window could be mistaken. This along with the bug in the Inserter's rotation speed are really worrying to me.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Thu Mar 26, 2015 5:49 pm
by slpwnd
Thanks for the nice analysis. The problem was in our smoothing window for the statistics. This is necessary otherwise the graphs would just look rubbish. So the smoothing window actually changes the size of the interval. This is fine for the graphs as they are basically an average from the past smoothing window. However there was no logic to compensate for this in calculation of average output in the overview. I have added that for the 0.11.20. Now the output shows consistent output across different time spans. It is still not equal to theoretical calculations (it is about 2.5% less than that) but imho it is good enough.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 9:12 am
by SilverWarior
How are these averages production values calculated?
I hope you are not calculating average for longer period from multiple average values for shorter periods?
If you are then this is the cause for wrong results.
You should always operate on base data and not intermediate calculations when dealing with averages.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 9:26 am
by slpwnd
SilverWarior wrote:How are these averages production values calculated?
I hope you are not calculating average for longer period from multiple average values for shorter periods?
If you are then this is the cause for wrong results.
You should always operate on base data and not intermediate calculations when dealing with averages.
The game is collecting the samples per tick. Then all the overviews for different periods are calculated from these data. As mentioned before the problem with accuracy was related to using a sliding window. Sliding window is necessary for visual graphs, otherwise the graphs output would show very aggressive values and would not really be usable.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 9:32 am
by SilverWarior
Wait are you saying that you store 36000 values (5s production productions) for each product so that you can calculate 50h average and draw that specific graph?

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 9:36 am
by quinor
that should be less than ~2mb of data :) storing numbers is quite easy and efficient.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 10:34 am
by SilverWarior
psorek wrote:that should be less than ~2mb of data :) storing numbers is quite easy and efficient.
About 140KB for each product. Multiply this with the number of vanilla products and you are already on 20MB.
Then add all those new items from other mods and the amount isn't so small as it seemed in the beginning.

Anyway the main reason why I'm asking this is because I know a way to implement this with much lower memory footprint which is faster to process and doesn't suffer same precision problems as calculating average of averages would.

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 11:45 am
by slpwnd
Each interval actually uses the same amount of samples at different precision levels. If we forget the smoothing window then there is 300 samples for each interval.

So:
5s ~ 1 sample per tick
1min ~1 sample per 12 ticks
10min ~ 1 sample per 120 ticks
...

Re: [0.11.19][slpwnd] Production window stats highly inaccurate

Posted: Fri Apr 03, 2015 2:22 pm
by Holy-Fire
Happy to see this finally resolved. Ironically, it has been exactly 1 year since my last report on the issue:
https://forums.factorio.com/forum/vie ... =10#p22419

I didn't feel like dedicating the time to fight for this to be fixed at the time, and hoped it would be resolved eventually. And it was :)

By the way, I think the largest errors are for 5s, so looking at larger intervals underestimates the problem.
slpwnd wrote:It is still not equal to theoretical calculations (it is about 2.5% less than that) but imho it is good enough.
I guess we can live with that, but I'll be happy to see it made even more accurate.