[14.3] Unable to "Catch Up" with capable hardware

[Lilly]

This stack overflow answer was a pretty massive eye opener to me, as far as how much effect the branch predictor can have.

For me as well. Though, somehow I don't seem to encounter situations where improving branch prediction makes an important difference.

However, dynamic function calls are also branches. Those almost certainly lead to branch prediction failures. Meaning that object oriented programming (OOP) might not be such a smart idea if you're writing game logic that needs to be efficient/fast (i.e. executed >100.000 per second). This chapter about data locality describes an interesting alternative to OOP, which eliminate the whole virtual function calls, while also being cache friendly. It seems that Factorio actually went into this direction with their new implementation of the circuit network: FFF-138 (from a design point of view at least, I don't know about the implementation).

[kogimus]

by Sunder1977 » Thu Sep 15, 2016 2:02 am

Apparently OP and friends have no idea what anecdotal evidence is. Here, I'll help.
https://en.wikipedia.org/wiki/Anecdotal_evidence

Yes, your "edge case" is the very definition of anecdotal evidence.

I've been out of town, just catching up on the thread (yay for Fall Ridein. Great time. Iron Butterfly was actually surprisingly cool) and saw this....

You know.. if you read your own link :

In science, definitions of anecdotal evidence include:

"casual observations or indications rather than rigorous or scientific analysis"
"information passed along by word-of-mouth but not documented scientifically"

We have provided all available logging of the incident (thus, as rigorous of an analysis as the tools provided allowed), as well as the server savegame that allows this particular set of circumstances to be repeated (thus, providing repeatability of the experiment) .

That the circumstances that give rise to it are in the situation of an edge case for usage (but not, mind you, outside what appears to be intended scope) do not inherently make the issue "anecdotal"...if we weren't able to repeat it, provide logging, the gamestate and had only "one time, when we did this, thing happens", then sure.
Anecdotal.
But..uh.. that's not the case here.


Anyway, all that said, the whole "maybe this is another way it could be done?" conversation is pretty awesome, and I'm learning a bunch of stuff that I didn't know. I deal in a different part of the computer world myself, mostly mid tier networking issues, though I used to be a dba (completely not my idea. i hated that), a VoIP sysadmin, and am more of a generalist.

[inetknght]

How the different compilers handled the test code in that stackoverflow.com answer was interesting.

I deal with so many "developers" on a daily basis that remain willfully ignorant of how the hardware they're developing on actually works (they think modern abstraction absolves them of the need) that it brings a smile to my face every time I see discussions by developers that have a clue.

Indeed. It's been interesting learning the performance differences between GCC, LLVM, and ICC, even differences in minor point-releases between the three.

However, dynamic function calls are also branches. Those almost certainly lead to branch prediction failures.

Are you talking about virtual functions? That's not a branch prediction failure. That's typically just a cache miss. However it's typically a very expensive cache miss because it ends up being multiple levels of cache misses. It gets especially more expensive as your inheritance tree grows.

Suppose you have one virtual function overridden? That's once that you have to grab the virtual table for the object (probably a data cache miss) and then jump to the function (almost guaranteed an instruction cache miss). If your overridden function ends up calling anywhere else in its own inheritance tree, good news: the virtual table might still be in cache, depending on how much data you've worked with. If it's not, well, there's another data cache miss. And, of course, another instruction cache miss, unless the compiler's able to inline the virtual function call (haha, I'm not sure I know any that do).

A single cache miss is more expensive than a single branch prediction failure (with roughly current hardware). Both get to be extremely costly when you have lots of them. I highly recommend watching the talk that Chandler Carruth gave at cppcon 2014 about performance related to data locality and "big-O algorithmic performance". I'm not a big fan of the craze for looking only at big-O notation for this specific reason.

Speaking of cppcon... I'm at cppcon 2016 right now. Anyone else?

This chapter about data locality describes an interesting alternative to OOP, which eliminate the whole virtual function calls, while also being cache friendly. It seems that Factorio actually went into this direction with their new implementation of the circuit network: FFF-138 (from a design point of view at least, I don't know about the implementation).

Both of those are good reads

[Yehn]

Hi,

I was linked here from another thread and just wanted to share a little feedback.

As a server grows, fewer and fewer people are able to keep up and play on it. It would be nice if there was an option to have the server gradually scale back simulation speed to a specified minimum so those there in the early stages can keep playing on the map. It seems there's a lot of players out there that do not run Factorio on high end hardware.

Thank you.

[Deadly-Bagel]

The problem with Factorio is you look at it and it comes across as an indie game or something like FTL or Papers Please, that is just simple graphics and lightweight code. Therefore it should run on pretty much anything right? Minecraft runs on pretty much anything, and it's 3D! With the Tekkit mod pack it's even a bit similar to Factorio.

It's only when (if) you dig into exactly what the game does that you realise it's not that simple. Minecraft only loads a small area around the player, move away and it stops processing that area (basically stopping time). Factorio does not. Those fifteen outposts scattered across the map may not be rendering but all their miners are still mining, transporting and loading all that ore/oil. There are countless biters roaming the map, especially late game, 1,000 bots to a factory is a small number to have floating around and don't even get started on the circuit network.

So while Minecraft does quite a lot of pretty rendering, it's only processing a few hundred entities per tick, maybe a few thousand with some mods. Factorio is processing hundreds of thousands. It's incredible what the devs have accomplished, but to then synchronise this with 400 players??

When you think of it that way, is it still realistic to expect this game to just work on any old junk?

[LotA]

I'm really eager to see what multi-threading can do about catching up.

I guess that if done properly, it reduce tremendously the catching up time, thus negating the problem.

[Chriz]

Hello i know this thread is a bit old, but we have the same problem with unable to catching up at a map with about 25mb, i dont get it why, the server have very good hardware and our pc's too.

if you want i can post the server specs and pc specs.
(we are just 2 players on the server)

regards
Chriz

edit: there was something wrong with the mods.rar useless things in there

[Avezo]

My experience with multiplayer 'catch up' is that it has very little to do with your own hardware, much more with your internet bandwich and/or server capacity.

Results in such cases seems somewhat unexpected - you'd read everywhere that full-bot factory is better for UPS for your factory, because bots don't check for collision-boxes like belts do and whatnot and you would see it's actually true in megafactories in singleplayer mode, but when you actually log into random multiplayer map... You will notice that factory size actually matters less than amount of bots flying around and you'll wish it was belts all around.

[Chriz]

We have about 50k robbots but only 5-10k are doing something. The thing is i dont have the catch up problem and have a internet connection about 16mbits, my friend have 100mbits and have the catching up problem. The dedicated server have a 1gbits connection. When i cancel some robboter actions like transfering stuff he is able to connect sucessfully.

Regards Chriz

[orzelek]

In Factorio performance matters for catch-up.
And from few threads like this it seems that multiplayer is a bit more demanding then single player.
Can your friends PC run the save alone in single at 60 UPS?

If he is barely at 60 or close to it and your server has better hardware then his PC then his client will be unable to catch-up.

One of currently recommended tests (short time solutions potentially) is to use this command:

Code: Select all

/c game.speed = 0.9

You can change the number to lower if 0.9 is not enough. This will slow down the game for all parties involved and will allow for easier catch-up if performance of client is the problem.