CPU performance benchmarks

[jape3]

Just tried on the ryzen 3700x / 32GB 3600MHz ram, factorio versio 18.34. Got an average of 86.7UPS on 5 runs

Code: Select all

./factorio --benchmark "stevetrovs_10K_mega_belt_base.zip" --benchmark-ticks 1000 --disable-audio --mod-directory /dev/null --benchmark-runs 5

52.028 Loading map /home/janne/factorio_Krastor2_SP/saves/stevetrovs_10K_mega_belt_base.zip: 40021782 bytes.
  52.036 Loading level.dat: 145798854 bytes.
  52.044 Info Scenario.cpp:187: Map version 0.17.31-0
  52.046 Info PrototypeMigrationList.cpp:61: Activating migration base/2019-04-27_Factorio_0.17.35.json
  52.046 Info PrototypeMigrationList.cpp:61: Activating migration base/2019-04-30_Factorio_0.17.35.json
  52.046 Info PrototypeMigrationList.cpp:61: Activating migration base/2019-05-01_Factorio_0.17.35.json
  64.270 Loading script.dat: 339 bytes.
  64.272 Checksum for script /home/janne/factorio_Krastor2_SP/temp/currently-playing/control.lua: 957497751
  64.272 Applying migration: Base Mod: 2020-01-27_Factorio_0.18.02.lua
  64.273 Applying migration: Base Mod: 2020-03-18_Factorio_0.18.14.lua
  64.373 Applying migration: Base Mod: 2020-05-27_Factorio_0.18.28.lua
  64.378 Checksum for script /home/janne/factorio_Krastor2_SP/temp/currently-playing/control.lua: 4108740459

[azesmbog]

This benchmark shows how bad Ryzen is for Factorio:
https://factoriobox.1au.us/result/f4fe9 ... 941ac1ea8f
Thats 75 UPS only compared to more then 100 for intel CPU's.

The developers have already quite plausibly explained why the parallelization of tasks is done and as much as possible.
And I believe them.
But I would also like to hear from the developers about such a big difference in the operation of INTEL vs AMD processors.
What is the reason for this and how do they understand it and let them explain on the fingers, as for the smallest)
Why are the best processors from AMD 8 cores 16 threads lose to a processor 10 years ago three generations ago costing less than $ 80 with 4 cores )

[Rseding91]

This benchmark shows how bad Ryzen is for Factorio:
https://factoriobox.1au.us/result/f4fe9 ... 941ac1ea8f
Thats 75 UPS only compared to more then 100 for intel CPU's.

The developers have already quite plausibly explained why the parallelization of tasks is done and as much as possible.
And I believe them.
But I would also like to hear from the developers about such a big difference in the operation of INTEL vs AMD processors.
What is the reason for this and how do they understand it and let them explain on the fingers, as for the smallest)
Why are the best processors from AMD 8 cores 16 threads lose to a processor 10 years ago three generations ago costing less than $ 80 with 4 cores )

We don't write code for any specific processor: just pure C++. So, any differences in performance being seen is due to the hardware itself. AMD CPUs are slower than Intels when it comes to memory access and games do a ton of that. Factorio optimizes actual calculations done by the CPU a lot and so the limiting factor ends up being how fast memory can be pulled/pushed to system RAM.

[Jap2.0]

This benchmark shows how bad Ryzen is for Factorio:
https://factoriobox.1au.us/result/f4fe9 ... 941ac1ea8f
Thats 75 UPS only compared to more then 100 for intel CPU's.

The developers have already quite plausibly explained why the parallelization of tasks is done and as much as possible.
And I believe them.
But I would also like to hear from the developers about such a big difference in the operation of INTEL vs AMD processors.
What is the reason for this and how do they understand it and let them explain on the fingers, as for the smallest)
Why are the best processors from AMD 8 cores 16 threads lose to a processor 10 years ago three generations ago costing less than $ 80 with 4 cores )

Ryzen has on the order of double the memory latency of Intel chips (due to architectural differences, there's plenty of material online if you look), which is a significant bottleneck in Factorio.

[azesmbog]

Ryzen has on the order of double the memory latency of Intel chips (due to architectural differences, there's plenty of material online if you look)

Factorio optimizes actual calculations done by the CPU a lot and so the limiting factor ends up being how fast memory can be pulled/pushed to system RAM.

Probably this is what I wanted to hear and understand.
What is the main problem with the speed of access of the processor to the system memory.
Thanks for the clarification.

[azesmbog]

Hey, sorry about that. I don't log into the forums often.
I've gone ahead and fix it and all of them show up now.

Hey.
I hope you come here again.
thanks for correcting the script, but my results for 6400 could have been deleted, I would have made new ones. Anyway.
Now the question is different.
Here is a screenshot of two results from the same map.

stevetrovs.jpg (71.28 KiB) Viewed 9175 times

As you can see, the difference in 7 UPS is quite significant, someone will have more.
I certainly understand that everything is in the same conditions, and the results are therefore valid,
but maybe you need to change the amount of update time in the script,
(instead of 1000, make 5000/10000?) to neutralize the difference in the screenshot.
I hope it's clear where it came from ??

[Syhn]

As you can see, the difference in 7 UPS is quite significant, someone will have more.
I certainly understand that everything is in the same conditions, and the results are therefore valid,
but maybe you need to change the amount of update time in the script,
(instead of 1000, make 5000/10000?) to neutralize the difference in the screenshot.
I hope it's clear where it came from ??

What versions are those benchmarks from? Base on the first one being 10 runs, it looks like that one is a bit older as I changed that a while ago. A lot can change between Factorio versions so performance shouldn't be expected to be the same. Also the map is fairly outdated at this point and I should update it, though that would make it impossible to compare against other benchmarks until there are a lot of benchmarks with whatever the new map is. I'm soon going to let the use choose which map to benchmark, so maybe the user can choose to benchmark both maps...

[azesmbog]

What versions are those benchmarks from? Base on the first one being 10 runs, it looks like that one is a bit older as I changed that a while ago. A lot can change between Factorio versions so performance shouldn't be expected to be the same. Also the map is fairly outdated at this point and I should update it, though that would make it impossible to compare against other benchmarks until there are a lot of benchmarks with whatever the new map is. I'm soon going to let the use choose which map to benchmark, so maybe the user can choose to benchmark both maps...

This is it, that this is the same card, it is tested on the same version, but the first version is the offline version.
Now why is there such a big difference.
You have a map on the website version 0.17.31
Therefore, on any other version other than 0.17.31, the save file is first converted to the current version, and then testing. This is where 7 UPS is lost.
I just took it uploaded first in the game for version 0.18.45 this save, waited for it to convert it, and saved it as 0.18.45, without even leaving the place.
Then I just tested this map offline. Hence the difference in the UPS, you see, a rather significant difference.
And this is only with this map, with others, the difference can be even more significant, up to 20 UPS between the same versions of the game-save, and different game-save.
I expressed myself too incomprehensibly ??

[Syhn]

Now why is there such a big difference.
You have a map on the website version 0.17.31
Therefore, on any other version other than 0.17.31, the save file is first converted to the current version, and then testing. This is where 7 UPS is lost.

This should not be the case. Benchmarks do not include the conversion time, only the time running the ticks. I just tested it myself with the original map and pre-converted map:

Code: Select all

factorio --benchmark 0.17.31.zip --benchmark-runs 3
Performed 1000 updates in 13639.490 ms
Performed 1000 updates in 13274.844 ms
Performed 1000 updates in 13551.489 ms

factorio --benchmark 0.18.45.zip --benchmark-runs 3
Performed 1000 updates in 13345.249 ms
Performed 1000 updates in 13328.553 ms
Performed 1000 updates in 13630.641 ms

As you can see they are nearly identical. Here is the map if you want to test yourself: https://drive.google.com/file/d/1NWSnQx ... sp=sharing

I do not know where you are getting the 7 UPS difference that you are.

[azesmbog]

The link is not public.

[Syhn]

Whoops... Fixed.

[thetalon]

My first thought upon seeing ryzen was that as I currently have an 8350. My mega factory is running at 20 25 ups. The one core is locked at 4.2ghz the rest are chillin. We'll probably see the same in this, when they say how much "better" the cpu is they're on about multi core tasks typically rendering or runnin multiple cpu intensive tasks. My question is how much better are the individual cores? Look at what they compare it to, I'm still thinking to lug it out with this old cpu, it's never done 8 cores work since I found facto rio. a decent motherboard and faster ram is required it seems

I had an FX8350, and I upgraded to a Ryzen 5 2600. My advice to you is to get rid of that thing ASAP. It's a decent CPU for its age, but it doesn't compare to anything that has hyperthreading or multithreading. Your CPU performance will at minimum double, in almost anything that supports multithreading. I say at minimum because the Ryzen CPU has better IPC than the FX series, as well. It's a huge upgrade as far as gaming is concerned

[xxdolcexx]

https://factoriobox.1au.us/result/d0b02 ... c01985c182

why do i have so bad results 4.5 ghz 3600ddr4 i need new cpu wich>?

[azesmbog]

why do i have so bad results

I would also say that the results are low. I have a much higher result on Skylake 6400, and the processor is much worse

[ske]

https://factoriobox.1au.us/result/d0b02 ... c01985c182

why do i have so bad results 4.5 ghz 3600ddr4 i need new cpu wich>?

Do you have something else running in the background? (Could be the antivirus doing a full scan or something else.)

Does the benchmark get slower and slower if you run it multiple times? (Check temperature and frequencies. It could be that your cooling is not good enough.)

i need new cpu wich>?

If you're on a tight budget Ryzen 3950X. If you can afford something decent try Threadripper. If you need more power, go with EPYC.

[azesmbog]

As you can see they are nearly identical. Here is the map if you want to test yourself: https://drive.google.com/file/d/1NWSnQx ... sp=sharing

I do not know where you are getting the 7 UPS difference that you are.

With this saving, the results are identical, but I did two resave myself - with them I got the result 6 UPS higher. It does not matter.
I very often add results to the site from everything I can reach. Recently I have added results from several laptops and from one computer with an AMD processor
But yesterday I tested my old I5-6400-non-K 2.7 GHz processor. Some of the results are on the site, but I also did a lot of offline tests.
I will share and try to explain the results.
The first stage of tests - with a constant processor frequency of approximately @ 4.7 GHz and fixed primary timings - the memory frequency changed from the minimum to the maximum. This is not entirely correct for real memory. at a lower frequency, the timings should be less, but this is exactly a test.
So the results are:

I5-6400-non-K 2,7@4,7 GHz
18-19-19-38 1390 MHz - 60 UPS
18-19-19-38 1622 MHz - 68 UPS
18-19-19-38 1854 MHz - 76 UPS
18-19-19-38 2086 MHz - 82 UPS
18-19-19-38 2316 MHz - 88 UPS
18-19-19-38 2548 MHz - 93 UPS
18-19-19-38 2780 MHz - 97 UPS
18-19-19-38 3010 MHz - 102 UPS
18-19-19-38 3474 MHz - 110 UPS
18-19-19-38 3704 MHz - 113 UPS
18-19-19-36 3712 MHz - 113 UPS

Note that two memory sticks with a nominal operating frequency of 3000 MHz (16-18-18-38 timing, 1,35 V) worked at the maximum safe voltage of 1.45 V.
Therefore, the second stage of tests was designed for the nominal frequency, but with different timings at the same processor frequency of -4.7 GHz
So the results are:

I5-6400-non-K 2,7@4,7 GHz
18-19-19-38 3010 MHz - 102 UPS
16-18-18-38 3010 MHz - 105 UPS (2T, nominal, default memory)
16-18-18-38 3010 MHz - 106 UPS (1T, nominal, default memory)
15-16-16-34 3010 MHz - 106 UPS
15-16-16-32 3012 MHz - 108 UPS
14-16-16-32 3012 MHz - 108 UPS
14-16-16-32 3016 MHz - 108 UPS

Draw your own conclusions from the tests
I can only say that at maximum acceleration
an old five-year-old processor released and several generations ago performs on equal terms in this game with some top-end processors I7-I9 and surpasses any top-end processors from AMD.
I emphasize - with a good memory

[Scifimyth]

So I just found out about this test and decided to run it to compare to some of the others in this thread. This is on an i7 8700k @ 5ghz with 32gb (4x8gb) of 3200mhz Ram. No single core ever went above 32% under either Task Manager or Resource Monitor.

powershell_2020-10-02_03-33-07.png (21.02 KiB) Viewed 8179 times

[sushi_eater]

When using huge pages on Linux, Ryzen 3900x is as fast as a i9-9900K. I'm getting 114UPS:

https://factoriobox.1au.us/result/880e5 ... 4e666983c9

Transparent huge page support needs to be enabled in the kernel:
/sys/kernel/mm/transparent_hugepage/enabled:always [madvise] never
/sys/kernel/mm/transparent_hugepage/defrag:always defer defer+madvise [madvise] never

The following environment variables need to be set when running Factorio (libhugetlbfs needs to be installed):
MALLOC_ARENA_MAX=1
LD_PRELOAD=/usr/lib64/libhugetlbfs.so
HUGETLB_MORECORE=thp
HUGETLB_RESTRICT_EXE=factorio

The MALLOC_ARENA_MAX is critical. If it's not set, only a part of the memory allocations use huge pages.

The CPU is not overclocked and PBO is disabled. The RAM is overclocked and runs at 3600MHz CL16.

[azesmbog]

An interesting result.
Is it possible another test, at the same frequency and with the same memory, but without these kernel tweaks ?? To see the difference, before and after
there is a very big difference between 114 and 85 (the usual result is also with good memory

[sushi_eater]

An interesting result.
Is it possible another test, at the same frequency and with the same memory, but without these kernel tweaks ?? To see the difference, before and after
there is a very big difference between 114 and 85 (the usual result is also with good memory

The result is 94 without huge pages. I'm using Factorio 1.0, Kernel 5.8.13, PDS scheduler, performance governor. There is a HUGE amount of TLB misses with standard pages (around 12'000'000/s with standard pages vs. 1'000'000/s with huge pages).

The CPU boosts to 4.625GHz.