OS X Ventura massive performance improvements (M1)

Post all other topics which do not belong to any other category.
eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

OS X Ventura massive performance improvements (M1)

Post by eangineer »

For those playing on an M1 Mac, just wanted to share some testing data. TLDR: Ventura more than doubled my UPS on a Mac Studio (Base Model). More testing with a 14" M1 MBP and base model M1 Mac Mini to come.

Mac Studio:

Original: 169.05 UPS, fastest run of 5: 5710
Ventura:
Run 1: 419.28 UPS, fastest run of 5: 2340
Run 2: 418.44 UPS, fastest run of 5: 2367

For comparison: my 5800X running Fedora 37:
226,01 UPS, fastest run of 5: 4416

Console output( flame.zip is the Flame-10K map from factoriobox):

Eans-Mac-Studio MacOS % ./factorio --benchmark ~/Downloads/flame.zip --benchmark-runs 5 --benchmark-sanitize
Performed 1000 updates in 2340.689 ms
avg: 2.341 ms, min: 1.376 ms, max: 115.378 ms
checksum: 4176999040
Performed 1000 updates in 2403.707 ms
avg: 2.404 ms, min: 1.318 ms, max: 119.477 ms
checksum: 4176999040
Performed 1000 updates in 2405.351 ms
avg: 2.405 ms, min: 1.392 ms, max: 125.857 ms
checksum: 4176999040
Performed 1000 updates in 2389.285 ms
avg: 2.389 ms, min: 1.374 ms, max: 127.667 ms
checksum: 4176999040
Performed 1000 updates in 2388.715 ms
avg: 2.389 ms, min: 1.379 ms, max: 128.288 ms
checksum: 4176999040
Eans-Mac-Studio MacOS % ./factorio --benchmark ~/Downloads/flame.zip --benchmark-runs 5 --benchmark-sanitize
Performed 1000 updates in 2367.806 ms
avg: 2.368 ms, min: 1.376 ms, max: 114.690 ms
checksum: 4176999040
Performed 1000 updates in 2408.593 ms
avg: 2.409 ms, min: 1.385 ms, max: 119.557 ms
checksum: 4176999040
Performed 1000 updates in 2393.801 ms
avg: 2.394 ms, min: 1.409 ms, max: 118.345 ms
checksum: 4176999040
Performed 1000 updates in 2402.823 ms
avg: 2.403 ms, min: 1.319 ms, max: 127.977 ms
checksum: 4176999040
Performed 1000 updates in 2379.317 ms
avg: 2.379 ms, min: 1.421 ms, max: 119.104 ms
checksum: 4176999040

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

14" M1 MBP: 400 (lowest run of 2) UPS! This just does not seem right. Someone please check my math?

Eans-MacBook-Pro MacOS % ./factorio --benchmark ~/Downloads/flame_10k.zip --benchmark-runs 5 --benchmark-sanitize
Performed 1000 updates in 2446.714 ms
avg: 2.447 ms, min: 1.386 ms, max: 119.440 ms
checksum: 4176999040
Performed 1000 updates in 2484.253 ms
avg: 2.484 ms, min: 1.350 ms, max: 122.133 ms
checksum: 4176999040
Performed 1000 updates in 2469.349 ms
avg: 2.469 ms, min: 1.335 ms, max: 130.710 ms
checksum: 4176999040
Performed 1000 updates in 2471.417 ms
avg: 2.471 ms, min: 1.351 ms, max: 127.201 ms
checksum: 4176999040
Performed 1000 updates in 2474.948 ms
avg: 2.475 ms, min: 1.420 ms, max: 122.398 ms
checksum: 4176999040
Eans-MacBook-Pro MacOS % ./factorio --benchmark ~/Downloads/flame_10k.zip --benchmark-runs 5 --benchmark-sanitize
Performed 1000 updates in 2488.730 ms
avg: 2.489 ms, min: 1.305 ms, max: 118.320 ms
checksum: 4176999040
Performed 1000 updates in 2511.409 ms
avg: 2.511 ms, min: 1.383 ms, max: 127.855 ms
checksum: 4176999040
Performed 1000 updates in 2500.475 ms
avg: 2.500 ms, min: 1.358 ms, max: 121.824 ms
checksum: 4176999040
Performed 1000 updates in 2494.743 ms
avg: 2.495 ms, min: 1.354 ms, max: 130.050 ms
checksum: 4176999040
Performed 1000 updates in 2483.915 ms
avg: 2.484 ms, min: 1.358 ms, max: 123.762 ms
checksum: 4176999040

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

M1 Mac Mini (base Model): 133.82 UPS (lowest of 2 runs) which is still higher than the 14" MBP with Monterey.

I do not have a previous benchmark for this system as, I had not even considered using it for Factorio.

Additional info:

Mac Studio: M1 Max CPU/32GB RAM
14" MBP: M1 Pro/16GB RAM
Mac Mini: M1/8GB RAM

Mac Studio w/ Monterey: 169 UPS
Mac Studio w/ Ventura: 418 UPS
14" MBP w/ Monterey: 131 UPS
14" MBP w/ Ventura: 400 UPS
AMD 5800X (Windows 11): 176 UPS
AMD 5800X (Fedora 37): 226 UPS

Highest Entry I could find on factoriobox with Flame SLA 10K map: 457 UPS: AMD 5800X3D (this result is an extreme outlier), most results for this CPU are mid-high 300's.

This is crazy!

StrangePan
Factorio Staff
Factorio Staff
Posts: 33
Joined: Fri Dec 07, 2018 3:41 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by StrangePan »

Thanks for posting this data! These are very interesting results. I'm going to see if I can reproduce these results. Is this the map you used for testing? https://factoriobox.1au.us/map/info/4c5 ... e161705504

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Yes, that was the map I used, just renamed to make the cli easier.

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

On a related note, I do not think there is any real FPS performance improvements.

Also, testing was done with stand-alone game, NOT Steam.

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

In a local game, the results are more accurate than in Steam.
If you have enough RAM :), then I can test my map, which in Monterey never shows more than 37-38 UPS).
And if you test maps with factoriobox, then I recommend checking the results for a heavy map
https://factoriobox.1au.us/results/cpus ... =1.0.0&vh=

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Will test with the 30K map in just a moment!
Last edited by eangineer on Thu Oct 27, 2022 12:58 pm, edited 1 time in total.

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

I'll wait for your result.
My computer on map Flame_Sla's 30k SPM belt shows 70 UPS, under Windows.
Under MacOS, I probably will never wait for the script :))

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Benchmarks ran against Flame_SLA_30K map.

SYSTEM RUN # 1 2 3 4 5 UPS
M1 Mac Mini (Ventura) 1 23924.167 23824.025 24246.159 24343.166 24380.74 41.42
M1 Mac Mini (Ventura) 2 24173.904 23841.259 24194.828 24160.246 24244.875 41.45
M1 Mac Mini (Ventura) 3 24283.684 24292.668 24341.239 24245.545 24300.415 41.16
14" MBP (Ventura) 1 8236.026 8226.669 8288.966 8225.369 8148.243 121.58
14" MBP (Ventura) 2 8371.04 8344.543 8330.605 8185.315 8205.804 120.66
14" MBP (Ventura) 3 8240.292 8399.127 8387.707 8405.06 8358.377 119.64
Mac Studio (Ventura) 1 7371.477 7364.876 7313.661 7283.633 7355.431 136.28
Mac Studio (Ventura) 2 7376.076 7447.627 7387.211 7408.66 7366.36 135.19
Mac Studio (Ventura) 3 7308.998 7240.606 7374.351 7354.566 7295.173 136.71

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Sorry formatting did not display correctly.

M1 Mac Mini: 41, 41, 41 UPS
14" MBP: 119,120,121 UPS
Mac Studio: 135,136,136 UPS

Highest score for that map I see is 86 on a 5800X3D, 14" M1 (Pro) MBP and M1 (Max) Studio destroying that score. The issue would be FPS.

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

1000 updates is too little
I always checked in such a script for 10000 updates.
And what a fantastic fabulous results.
So how much RAM do you have? can still check on a really heavy map, which needs more than 64GB of RAM :))

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

The Mac Studio has 32GB, 14" MBP has 16GB, Mac mini has 8GB.

I will run with 10000 during lunch and post the results.

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

I'm quite satisfied with one result on the fastest model :)
You can also set the number of passes to 10. To be even more precise. The result is of course fantastic, much faster even 5800X3D with a huge cache size

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Here is 10K ticks, 5 runs On Mac Studio (10K/10 runs in process right now):

Mac Studio (Ventura) 1
59999.492
59473.464
59294.221
59239.444
59743.608

167.93 UPS
Seems odd to me that the UPS went up from the 1K ticks result.
Please verify my math?

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

Mac Studio 10000 Ticks for 10 runs: 166.25 UPS.

Raw output:

Eans-Mac-Studio MacOS % ./factorio --benchmark ~/Downloads/flame_30k.zip --benchmark-runs 10 --benchmark-sanitize --benchmark-ticks 10000
Performed 10000 updates in 59452.624 ms
avg: 5.945 ms, min: 3.329 ms, max: 11211.274 ms
checksum: 2394868592
Performed 10000 updates in 61045.594 ms
avg: 6.105 ms, min: 3.482 ms, max: 11042.718 ms
checksum: 2394868592
Performed 10000 updates in 60077.796 ms
avg: 6.008 ms, min: 3.347 ms, max: 10734.180 ms
checksum: 2394868592
Performed 10000 updates in 60431.417 ms
avg: 6.043 ms, min: 3.485 ms, max: 10871.155 ms
checksum: 2394868592
Performed 10000 updates in 60123.177 ms
avg: 6.012 ms, min: 3.378 ms, max: 10801.910 ms
checksum: 2394868592
Performed 10000 updates in 60353.198 ms
avg: 6.035 ms, min: 3.376 ms, max: 10769.330 ms
checksum: 2394868592
Performed 10000 updates in 59912.976 ms
avg: 5.991 ms, min: 3.362 ms, max: 10817.412 ms
checksum: 2394868592
Performed 10000 updates in 60126.008 ms
avg: 6.013 ms, min: 3.447 ms, max: 10729.650 ms
checksum: 2394868592
Performed 10000 updates in 60097.606 ms
avg: 6.010 ms, min: 3.430 ms, max: 10824.767 ms
checksum: 2394868592
Performed 10000 updates in 59872.779 ms
avg: 5.987 ms, min: 3.339 ms, max: 10788.373 ms
checksum: 2394868592

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

maybe the script registers the maximum values, not the averages??
Can you figure out how to run this script??

https://gist.github.com/Weldawadyathink ... 56aea2bd70

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

The factoriobox script does not work on Mac, as Mac does not have lshw. I can try to remove the applicable portions. Won't be able to make an attempt until this evening.

All kinds of issues with that script:

Please sudo to run lshw and get hardware information
Found 1.0.0 at /Volumes/EAN-FAST/Factorios/1 - Vanilla/factorio.app/Contents/MacOS/factorio
./benchmark.sh: line 44: gmktemp: command not found
Downloading map...
Warning: output file name has no length
curl: option -o: is badly used here
curl: try 'curl --help' or 'curl --manual' for more information
./benchmark.sh: line 48: sha256sum: command not found
Running benchmark...
./benchmark.sh: line 52: stdbuf: command not found
./benchmark.sh: command substitution: line 53: syntax error near unexpected token `('
./benchmark.sh: command substitution: line 53: `--benchmark-verbose all --benchmark-sanitize | tee >(grep Performed >&5)'
Benchmark failed

azesmbog
Filter Inserter
Filter Inserter
Posts: 252
Joined: Mon Jan 28, 2019 12:05 pm
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by azesmbog »

However, it is written that:
"modified for macOS"
Maybe even you can run the script.
And about the results. Very strange maximum results. Huge spread.
I've run the tests many hundreds of times, usually the difference is much smaller. Here is an example:
avg min max
Test 1 5.545 4.971 18.497
Test 2 5.550 4.970 18.510
Test 3 5.553 4.946 18.435
Test 4 5.550 4.964 19.027
Test 5 5.555 4.982 18.636

Average (ms) 5.551 4.967 18.621

avg max min
UPS 180.16 201.34 53.70

eangineer
Burner Inserter
Burner Inserter
Posts: 16
Joined: Thu Oct 27, 2022 12:06 am
Contact:

Re: OS X Ventura massive performance improvements (M1)

Post by eangineer »

After some testing and fiddling with various permutations of the factoriobox script and simple CLI execution, I have discovered the following.

When using the Factoriobox full command string, results are what I would have expected prior to Ventura update. Difference in execution and results comes down to Factoriobox specifies "--mod-directory /dev/null". I have never used that in any of my test runs pre or post Ventura, nor have I had any mods installed. Executing the Factoriobox test command on Linux without the "--mod-directory /dev/null", results in scores in expected normal range for the system tested (5800X), although slight uplift (2-3 UPS from 6.06 kernel).

I do not have a decent explanation for the wide variation of min and max values for runs. Testing of M1 processors seems consistently high when not using the "--mod-directory /dev/null". I will probably run a test with "--mod-directory" set to a normal directory later today.

Post Reply

Return to “General discussion”