Performance on AMD Bulldozer

[wanne]

Hi,

I didn't play Factorio for a while. But in ~0.15 I was able to join smaller multiplayer sessions with my laptop with an i5-2520M/8G-DDR3.
Now I can't join most of the online games with my PC (FX-8370/GTX-960/16G-DDR4). On bigger maps the "Catching up"-Bar just goes backwards and after a few Seconds I get an DC. (Like it is when you don't have a fast enough PC.)
I know Bulldozers are not known for their FP/SC-performance. But this computer is able to run GTA V on all graphic-settings on "high" with 1080p@60Hz.
Do I nowadays need to have an high end gaming PC to run Factorio or is there something else wrong.
I saw that Wayland support was added. – Which I can't use because of my NVIDIA-GPU. Is this now needed?

Did someone made similar experiences?

[posila]

There are no particular system requirements. Your computer just needs to be able to run the simulation faster then the server. If a map is so large it starts to hit HW limits, people with slower CPUs than the server will start to get dropped, unless server host doesn't lower game speed. There are no graphics calculations happening while catching up.

[wanne]

Your computer just needs to be able to run the simulation faster then the server.

So if I run a Server with an Xeon Platinum 8256 and none would be able to join?! I know this and played on LANs with low performance-laptops for that reason with throttled Servers and with Set Game Speed/Global Tick Time Scale Mod. But not on "normal" games with PCs. That can't be intended.

There are no particular system requirements. […] If a map is so large it starts to hit HW limits,

The Question was if this is normal for more than average PCs when running a typical 48h+-Game. Or in other words: What Hardware do I need to be able to join 90% of the public games. Do I really need to outrun 90% of the servers since almost all maps are too big for any existing hardware?

[Loewchen]

So if I run a Server with an Xeon Platinum 8256 and none would be able to join?

Since single core performance and memory speed are the most important aspects for factorio, this would be not an issue for most systems.

The Question was if this is normal for more than average PCs when running a typical 48h+-Game.

The answer is no. But your cpu is way below average and what you experience is expected on bigger maps with modern server hardware.

[wanne]

Since single core performance and memory speed are the most important aspects for factorio, this would be not an issue for most systems.

I think there is no x86_64-CPU that can beat the Platinum 8256 on SC-FP-performance. Problem might be, that factorio dosn't use AVX2 but this is the case for all newer CPUs. And in things of memory bandwidth it will beat at least every desktop CPU.

But your cpu is way below average and what you experience is expected on bigger maps with modern server hardware.

Recommended (not minimal) requirements is a "Quad core 3GHz+-CPU". This is a Eight core CPU with 4.4GHz. So this is wrong now and the recommended hardware is by far not able to run the late game?
And like I said. It can run all mayor games including GTA V, Anno 1800, Whicher III and RDR II. So factorio is now the single most demanding game now? (For the CPU) And sorry the CPU might not be high-end but it is not below average. – At least not for causal gamers.

[CrushedIce]

What kind of servers are you trying to join? On Megabase servers with full game speed you will get problems but "smaller" servers should work fine. Maybe try a few maps in Singleplayer to see how many ups you get with your cpu.

Compared to 0.15, 0.17 is WAY more performant but of course at some point in the megabase or gigabase stage every cpu will struggle to keep up.

You can also try 0.18 which has again a few more performance optimizations.
Overall Factorio is by far the most performance optimized game I know. Just think about the insane amount of things that need to be simulated 60 times per second

[wanne]

What kind of servers are you trying to join?

Tried yesterday night a lot of servers with players (~20). There where only three I was able to connect.
There where only three I was able to connect all of them vanilla with less den 5h playtime. All the servers with bobs or angles where too fast for me.

Compared to 0.15, 0.17 is WAY more performant but of course at some point in the megabase or gigabase stage every cpu will struggle to keep up.

I think this is maybe my problem. If it is for real that every hardware struggles on bigger maps and you have to be faster than the server performance optimizations that are more useful for the server than me will be a problem for me.

So again the Question would be: What hardware is usually enough to join a typical 48+-game without bothering which serve is used and how big the map is.

Overall Factorio is by far the most performance optimized game I know. Just think about the insane amount of things that need to be simulated 60 times per second

Sorry, no by far not. Look at modern Games where millions of particles ore hairs simulated. A few k Objects shouldn't be that demanding.
The problem are a few initial design
I didn't went deeper into newer Versions so I don't know if they are still true. But I assume they are:
1. Making the game state deterministic avoids bugs. But it makes performance optimizing a lot more complicated. An serve which would be able to override the gamestate of the clients like it is common in every other game would have made thing much easier. But would ultimately lead to glitches. So I support this decision. But it is not the most performant.
2. Solving conflicts by giving actors a strict total order makes the game defacto singlethreaded wasting all the cache of the other cores. Instead of using a queue of actors they could have used an array of movable objects/stacks would have made things parallelizable. (With exception of the user interaction. But having this sequential afterwards wouldn't take much time.) – Of course this would kill all optimizations they made until now. Since all look from the actor viewpoint instead from an place-viewpoint. So I don't think anybody is willing to do this change. But with this design you could even run the simulation on a GPU with much faster memory and hundreds of shaders. I made a simulation like this with CUDA once. It works.
3. There is a lot of DP-FP-Calculation. There is almost no place where this needed and don't understand why this is happening for most uint16_t should be enough instead of using 64bits for a double.
4. Memory accesses looks awefull. I think there are a lot of lists where parts got deleted used. While this makes sense from a programmers perspective it hurts caches an making you bound on memory I/O-ops/latency instead of bandwidth. Using array (eventually with holes) and run from left to right eventually reordering from time to time is a lot faster. – But would make the code ugly as hell.
5. Do they still use the lua-interpreter instead of something with JIT?
So I understand the decisions they made. But factorio is by far not performance optimized.

[posila]

But with this design you could even run the simulation on a GPU with much faster memory and hundreds of shaders. I made a simulation like this with CUDA once. It works.

I am looking forward to joining multiplayer in your CUDA simulated builder game. Seriously, look how many copies Automation Empire sold. Factorio community is hungry for another title in this genre and if you can make a builder game that allows people to build much larger bases than Factorio, I promise you, it'll be very successful. (I am serious, not trying to mock you)

Btw. Regardless of how well or poorly the game is optimized, there will be a point (size of a factory) at which CPU will spend all its time doing simulation, so then we are back to the original problem of PCs with slower CPUs can't play on servers with faster CPUs. Players who want to build big will reach that point fast regardless of how far it is. Especially if they can carry over their builds from previous games. So you either have to limit how big players can build, or make it tedious to build on large scale (or don't make it a simulation). We won't do either.

It's neat games can do fancy hair simulations and simulate millions of particles nowadays, but those are just an eye candy and don't actually have any effect on what is happening in the game world. What does mater in GTA V is that only about 50 cars and 50 pedestrians that are around you actually exist in the game world. The fact your PC can run a game, that came out 7-5 years ago and was originally developed for 15 year old HW (Xbox360 and PS3) and is not all that relevant.

[CrushedIce]

So again the Question would be: What hardware is usually enough to join a typical 48+-game without bothering which serve is used and how big the map is.

I'm not playing on public servers, only coop with a friend so I dont know how big the bases on those servers are.
In my current 200h game with mods (~20k iron, copper, stone / min) I'm still at 60 ups with 10.6 ms update time, so I can still expand quite significantly before the game will start to slow down (at 16.6 ms update time).
I'm playing on a gaming laptop with an i7-8750H.
By the way 0.18 improved the update time in my case by ~ 2 ms.

Sorry, no by far not. Look at modern Games where millions of particles ore hairs simulated. A few k Objects shouldn't be that demanding.
The problem are a few initial design

It's not only a few k objects, in large bases there are millions of items and objects which can influence each other.
A big difference to other games is that Factorio simulates always the whole world (or even multiple worlds with mods). Most games simulate the world only in a certain area around the player. For example in Minecraft only 10 chunks around the player are loaded and machines will only work while the player is nearby.

1. Making the game state deterministic avoids bugs.

Determinism is mainly needed for multiplayer in Factorio because otherwise the amount of data the server would need to send would be huge.

[wanne]

I am looking forward to joining multiplayer in your CUDA simulated builder game. Seriously, look how many copies Automation Empire sold. Factorio community is hungry for another title in this genre and if you can make a builder game that allows people to build much larger bases than factorio, I promise you, it'll be very successful. (I am serious, not trying to mock you)

There is a big difference between being able to make fast simulation-lib and being able to design a playable game . Factorio has much more to offer than the simulation of a few inserters.

Btw. Regardless of how well or poorly the game is optimized, there will be a point (size of a factory) at which CPU will spend all its time doing simulation, so then we are back to the original problem of PCs with slower CPUs can't play on servers with faster CPUs. Players who want to build big will reach that point fast regardless of how far it is. Especially if they can carry over their builds from previous games. So you either have to limit how big players can build, or make it tedious to build on large scale (or don't make it a simulation). We won't do either.

Yes you do. There is an end of the map. (And its buggy btw. surface stops before the map.) Was there. . You could also drop tics like other games do. There are a lot of possibilities.

It's neat games can do fancy hair simulations and simulate millions of particles nowadays, but those are just an eye candy and don't actually have any effect on what is happening in the game world.

Dosn't change that they can.

The fact your PC can run a game, that came out 7-5 years ago and was originally developed for 15 year old HW (Xbox360 and PS3) and is not all that relevant.

The Game came out 4 years ago – less than a year before factorio and has on a technical perspective little in common with the discontinued Xbox360 version. I used it because it is still the typical measurement since there are not many more demanding games out there.
I think there is not a single common game out there that can't run smoothly on that CPU on "normal" gameplay. – Or like I said: Is factorio now the most demanding Game out there? And again what CPU does it need? I would be really interested in a answer to this Question. Sadly I can't upgrade on the AM3+-Socket. But sooner or later new hardware will come anyway. Is an i7-9700 enough or has it to be an Ryzen 9 3950X? Or are server CPUs the only option since they have much bigger memory bandwidth. What is the hardware you are currently playing on? Would it be worth trying on my office PC (i5-6500) which has better singlecore performance but worse mulitcore stats and only 1x16GiB@2133MHz RAM. (So no dual channel.)

[posila]

I think there is not a single common game out there that can't run smoothly on that CPU on "normal" gameplay. – Or like I said: Is factorio now the most demanding Game out there? And again what CPU does it need? I would be really interested in a answer to this Question.

What CPU does it need? Core 2 Duo, whichever model had first 64bit instruction set. If you pair it with half-decent dedicated GPU ... let's say GTX 460, you'll be able to play through the game and finish it (launch the rocket) without any slowdowns. Heck, take GTX 660 and you can play it with high-resolution sprites too. That's what's required for normal gameplay. (Disclaimer: we don't actually test on Core 2 Duo, the oldest CPU we test on is the first gen mobile Core i3 (Westmere architecture): Core i3-330M; but apparently there are decent number of people playing on Core 2 Duo)

I have no idea what the maps look like on servers you are trying to connect to look like, but 20 players can build lot of stuff in 50 hours, so I would not call that "normal" gameplay.

We don't run any servers ourselves at the moment, so I don't know what is typical server HW. However, memory bandwidth won't help you, we don't usually hit bandwidth limits, latency is an issue for the most part (as you pointed out, our memory access patterns are pretty poor).

The Game came out 4 years ago – less than a year before factorio and has on a technical perspective little in common with the discontinued Xbox360 version. I used it because it is still the typical measurement since there are not many more demanding games out there.

It's still comparing apples to moon rocks. One game spends almost all the time on rendering itself as beautifly as possible (yes, even the CPU time), the other game spends all the time on simulating activity of many individual machines. Out of your list of the games only Anno 1800 is at least the same domain of problem, so I must ask - how well do 20 player 50 hour matches run for you in Anno 1800?

[BlueTemplar]

I didn't play Factorio for a while. But in ~0.15 I was able to join smaller multiplayer sessions with my laptop with an i5-2520M/8G-DDR3.
Now I can't join most of the online games with my PC (FX-8370/GTX-960/16G-DDR4).

I have had some similar experiences recently with my FX-8320E.

Recommended (not minimal) requirements is a "Quad core 3GHz+-CPU". This is a Eight core CPU with 4.4GHz. So this is wrong now and the recommended hardware is by far not able to run the late game?

That's for single player. It doesn't matter for MP. See the end of this post.

And like I said. It can run all mayor games including GTA V, Anno 1800, Whicher III and RDR II. So factorio is now the single most demanding game now? (For the CPU)

[...]
1. Making the game state deterministic avoids bugs. But it makes performance optimizing a lot more complicated. An serve which would be able to override the gamestate of the clients like it is common in every other game would have made thing much easier. But would ultimately lead to glitches. So I support this decision. But it is not the most performant.
2. Solving conflicts by giving actors a strict total order makes the game defacto singlethreaded wasting all the cache of the other cores. Instead of using a queue of actors they could have used an array of movable objects/stacks would have made things parallelizable. (With exception of the user interaction. But having this sequential afterwards wouldn't take much time.) – Of course this would kill all optimizations they made until now. Since all look from the actor viewpoint instead from an place-viewpoint. So I don't think anybody is willing to do this change. But with this design you could even run the simulation on a GPU with much faster memory and hundreds of shaders. I made a simulation like this with CUDA once. It works.
[...]

It's not only a few k objects, in large bases there are millions of items and objects which can influence each other.
A big difference to other games is that Factorio simulates always the whole world (or even multiple worlds with mods). Most games simulate the world only in a certain area around the player. For example in Minecraft only 10 chunks around the player are loaded and machines will only work while the player is nearby.

1. Making the game state deterministic avoids bugs.

Determinism is mainly needed for multiplayer in Factorio because otherwise the amount of data the server would need to send would be huge.

I don't know why this is surprising to you. I don't know about Anno 1800, but games tending towards the simulation of large numbers of entities, dependent on each other, tends to be very hard on the single-core CPU performance.
I'm not aware of a single game like that that does NOT use the same "client-server with sequential simulation" model, and none that would have managed to multi-thread its simulation.
See this thread :
Parrallel processing in games & applications
(However, see caveat at the end of this post.)

And sorry the [FX-8370] might not be high-end but it is not below average. – At least not for causal gamers.

Well, if you only look at people with below average PCs, then yeah, it's isn't below *that* average. But people likely to play Factorio MP will have *above* average PCs, like you (and I) "discovered". (I also seem to have had better performance in 0.18, but this doesn't matter, see at the very end.)
Otherwise :
FX-8370 PassMark single-thread rating : 1537
i5-2520M PassMark single-thread rating : 1492
(Note that in a few years since you've played MP on 0.15, the average is likely to have risen quite a bit.)
Steam Hardware survey indicates the most popular category to be : "Intel 3.3 Ghz to 3.69 Ghz"
PassMark's "CPU Mark-worst" "Common CPU" in this category is the Intel Core i3-2120 with a single-thread rating of 1691.
PassMark's "CPU Mark-best" "Common CPU" in this category is the Intel Core i9-9900K with a single-thread rating of 2891.

AFAIK, the end of the map had to be introduced because some players played with fire eldritch majick trying to teleport to places way, waaaay above what your typical coordinates stored in typical number formats could handle ?

I have no idea what the maps look like on servers you are trying to connect to look like

Try to join 50+ players Biter Battles sometimes.
Though I wouldn't be surprised if a good part of the issues came from their own un-optimized Lua scripts. I feel that they could scale even more by instead using 1 surface per team, and having the C++ Factorio side to handle pollution and biter spawning ?
Or maybe there is some overhead that doesn't just come from more entities, but strictly from more players ?
(I wouldn't bet on it though...)

Ok, so, finally, what I feel is the crux of the issue : (And what has already been raised, but maybe not dwelled on enough ?)
The way how in current Factorio the players have to adjust to the server, rather than vice-versa.
It also would seem that this has not always been the case :
Multiplayer adaptive game speed (slow down for slow clients)
I'm aware of two ways of dealing with this issue :
- As expected, with games with expectations of not many players in MP, they tend to pick the "server slows down to keep up with slowest player" rule (Sword of the Stars 1, as an example. And, weirdly enough, Supreme Commander 1 : Forged Alliance (Forever) ? ).
- While for games with lots of players, where one single lagging player is going to slow everyone else down, well, they tend to follow the same rules as current Factorio. (See Spring games as an example.)
But one can imagine Factorio as being played in either of these "modes" : low (mostly private) or high (mostly public) player count...
So perhaps the best way would be to let the server owner to configure this behavior ?

[...]
My solution has been to manually set the game speed down via console command when certain players want to join, limiting the server to (say) 50 UPS so that a player that can barely do 55 UPS can both keep up and get a decent FPS.
However, this process is highly annoying and error-prone (I'll often forget to set game speed back up, so everything takes longer even if no-one is logged in).

Unfortunately I can't simply give console command access to my players even though they're trusted, because they can't even fully connect to the server (can't catch up) so they have no way to send a console command to slow the server down.

What I'd like to see is an optional feature that a server admin can enable, where the server will attempt to adapt its game speed value to match the slowest connected client. Then it can speed up again when the slow client disconnects.
Obviously this can be abused by a client that reports itself being unplayably slow, DoS-ing the server, so this option would only be appropriate for trusted clients (or you could cap the slowdown to some minimum value, say 0.5).

Also, funnily, in the current "server won't slow down" mode, eventually, the only people that are going to be able to keep up, would be those that have even faster hardware than the server ! And in the case of dedicated servers, only those that can compensate for the extra computing power needed to run the graphics !!
(Also, going back again to the "factorio is now the single most demanding game" - well duh, of course it is : there's no restriction on the number of entities !)

[Jap2.0]

I use a core i7 950, a 10+ year old CPU that allegedly has a somewhat lower passmark CPU score (single and multi threaded). I haven't run into any (even the larger) servers that I can't run. I haven't tried biter battles (I've been meaning to), but Redmew also has some decent softmodding and I haven't run into issues there.

[BlueTemplar]

https://www.cpubenchmark.net/cpu.php?cp ... GHz&id=837
Intel Core i7-950 @ 3.07GHz
Single Thread Rating: 1328

Well crap, there goes my theory...
Maybe it's another AMD Bulldozer issue, like the too slow (DDR3) RAM ? Or cache ?

Yes, please do try !
Maximum game "stage" and amount of players that you have seen ?
I just tried joining a 6h20 Biter Battles (0.18.2) with 25+ players - managed to somehow catch up, but the game quickly became unplayable after that, and I ended up getting dropped after a few minutes...
(As another datapoint, that 400+ players Spiffing game ran without issues for me, but I guess it's the server that was kind of weak ?)

[wanne]

PassMark's "CPU Mark-worst" "Common CPU" in this category is the Intel Core i3-2120 with a single-thread rating of 1691.
PassMark's "CPU Mark-best" "Common CPU" in this category is the Intel Core i9-9900K with a single-thread rating of 2891.

No. If you except the Top 15 CPUs (of 100) every AMD-CPU scores less than mine in this benchmark.

Steam Hardware survey indicates the most popular category to be : "Intel 3.3 Ghz to 3.69 Ghz"

OK I have an AMD with 4.4GHz. Problem is: This says nothing. The i9-9900K surely outperforms my CPU but it has only 3.6GHz.

See this thread :
Parrallel processing in games & applications
(However, see caveat at the end of this post.)

You are just linking to the thread. Not to a single post. And I red it. (Or at least the beginning few sites.) There are a few people who have ideas. Does not change that most games tend to do things sequential. (While everybody else learned to do multi threading. Even the stupid JS-One-Thread-Model.) As long as there is fast enough SC-Harware out there, they use it.

I have no idea what the maps look like on servers you are trying to connect to look like, but 20 players can build lot of stuff in 50 hours, so I would not call that "normal" gameplay.

Playercount was between 1 and three. But for sure. It is quite more than a normal game. This was the reason why I wanted to know what other multiplayer players use.

as you pointed out, our memory access patterns are pretty poor

This is the reason why multi threading would really help: You get 8 or 16 times the amount of cache. If you don't target anyway more targets help... But like I said: I assume this option is gone. factorio won't change its architecture any more.

[Jap2.0]

Well crap, there goes my theory...
Maybe it's another AMD Bulldozer issue, like the too slow (DDR3) RAM ? Or cache ?

I forgot to mention, DDR3 too. Granted, that still leaves a lot of room for variance (I don't remember the speed, and have no idea what the CL is, off hand). I've also heard that the CPU itself on AMD can add as much as twice the latency as on Intel, so that could be it, although I'm pretty sure that was for Ryzen so I don't know if that applies to older AMD CPUs as well.

[BlueTemplar]

PassMark's "CPU Mark-worst" "Common CPU" in this category is the Intel Core i3-2120 with a single-thread rating of 1691.
PassMark's "CPU Mark-best" "Common CPU" in this category is the Intel Core i9-9900K with a single-thread rating of 2891.

No. If you except the Top 15 CPUs (of 100) every AMD-CPU scores less than mine in this benchmark.

Steam Hardware survey indicates the most popular category to be : "Intel 3.3 Ghz to 3.69 Ghz"

OK I have an AMD with 4.4GHz. Problem is: This says nothing. The i9-9900K surely outperforms my CPU but it has only 3.6GHz.

I don't care about AMD CPUs there. I only wanted to have an idea of what a "common CPU" might be. And the Intels in that Ghz spectrum are the most common ones. (Sadly, the Steam Hardware survey is not more detailed.)
And hopefully these Intels didn't have all over the place Single/MultiThreaded performance ratios for similar Ghz...

[wanne]

Here some updates:
I started pinning factorio on a CMT module.
What works for me:
* Reduce rendering threads to 3.
* Set the whole Process on 2 CMT modules (ex. mask 15)
* Set the first spawned tread to a one CMT module of the both above. (ex. mask 3)
Now I can catch up with every server. Redmew still makes no fun to play. ~6FPS/20UPS. Most other games work fine with 60/60.
If some devs reading here (@posila?): Maybe you want to do this always: Set the Processor affinity (pthread_setaffinity_np() ) of the "main" thread to the first (or the first two) cores. This seems to help a lot for older AMD-CPUs and it shouldn't hurt.

I don't care about AMD CPUs there. I only wanted to have an idea of what a "common CPU" might be.

Your ranking is bullshit. The i3-2120 is by far not the worst single scoring CPU in the common category. Even not if you exclude AMDs wich score terrible in this benchmark. (Letting the one year older Mobile processor i5-2520M of half the price and a sixth of the TDP score almost as high as the FXes ones is a joke. AMD made not everything perfect but this is hilarious.) there are tons Intel CPUs that score much worse:
Core2 Duo E7500: 1201
i5-520M: 1061
Core2 Quad Q6600: 924
...
Even with the bad AMD scores from there my CPU would be above the average of "common" CPUs. (Roughly around place 40 of 100.)

[Loewchen]

Since help is not what seem to be looking for I will close this.

[posila]

The thread starts to be interesting. Unlocked, renamed and moved to Ideas and Suggestions.

I need to find where did our Bulldozer PC end up in the office so I can experiment ... Also, I might not get properly to it until May or so.

Reduce number of rendering threads doesn't make sense to me, though. The threads are collecting render data (which we "prepare render" stage), at which point both update and render are not running, and when that finishes next update and render (as in pushing data updates and draw commands to GPU) are launched in parallel. So number of prepare threads should not have any effect on catching up in multiplayer (as that just loops through updates as fast as possible)

Let me check if something else uses max render threads as a limit.

What's your UPS in this save? (you need to update to 0.18 experimental, if you haven't already): https://www.dropbox.com/s/hkqbnymgpl211 ... 8.zip?dl=0