Very poor performance on headless server

[tmdag]

I've been playing with friends from other side of the globe for a while now. Usually I was running server from my client.
This time I wanted to create a headless server that we can all connect to. It was running on my home network as my client, but just on a separate machine.

Machine is not crazy powerful but I would expect to be enough for the game:
OS: Fedora Linux 37 (MATE-Compiz) x86_64
Kernel: 6.5.12-100.fc37.x86_64
CPU: Intel Xeon X5690 (12) @ 3.47GHz
GPU: 06:04.0 Matrox Electronics Systems Ltd. MGA G200eW WPCM450
Memory: 8545MiB / 15967MiB

So I have followed instructions (https://wiki.factorio.com/Multiplayer), downloaded latest (https://www.factorio.com/download) created a system service that runs factorio - all *works* as expected.

My network is decent, its a 3Gbps synchronous, headless machine utilises it via SFP+ 10Gb network adapter, which I've tested many times.

Unfortunately when 3 people connect to the server,connection performance is very poor, lagging - something that we did not experience that much when running server via client on the same network.

I've checked CPU and Memory, nothing is capping or swapping so I am not sure where that issue comes from. After googling around I found multiple posts with people complaining that headless server has overall very bad performance (https://www.reddit.com/r/factorio/comme ... with_fast/), (https://www.reddit.com/r/factorio/comme ... delay_bad/)

I was wondering if maybe someone could guide me and help getting to the bottom of this. I did turn on debugging info within the game, observing whenever there are any packets lost or skipped frames but couldn't see anything out of the ordinary. Also I might just not know where to look and therefore I was hoping that some smart people on this forum and/or developers could help fixing this ongoing issue.

[Rseding91]

Thank you for the report however this is not a bug. The issue is likely the CPU you’re using on the server. It is incredibly weak compared to modern desktop processors getting only 30% the performance of desktop processors released 3 years ago.

[tmdag]

Thanks for reply!

That would make more sense. Is there any specific element to take into consideration when choosing good hardware for headless server?
Like if i would switch to "Intel Xeon E-2336 Rocket Lake 2.9 GHz" would that be still considered as too slow for the game ?

Thanks!

[tmdag]

Also, I have checked CPU usage of factorio, and It does not justify what you are suggesting. I am barely getting to 20% of CPU usage
Is there any debug mode that would confirm your theory that CPU is to blame here ?

Code: Select all

05:31:01 PM   UID       PID    %usr %system  %guest   %wait    %CPU   CPU  Command
05:31:06 PM  1003      4799   12.40    4.20    0.00    0.20   16.60     1  factorio
05:31:11 PM  1003      4799   12.60    4.40    0.00    0.00   17.00     3  factorio
05:31:16 PM  1003      4799   12.60    4.00    0.00    0.00   16.60     1  factorio
05:31:21 PM  1003      4799   12.57    4.19    0.00    0.20   16.77     6  factorio
05:31:26 PM  1003      4799   12.40    4.40    0.00    0.00   16.80    10  factorio
05:31:31 PM  1003      4799   13.20    3.40    0.00    0.00   16.60     8  factorio
05:31:36 PM  1003      4799   13.40    4.40    0.00    0.00   17.80     8  factorio
05:31:41 PM  1003      4799   13.40    4.40    0.00    0.20   17.80     4  factorio
05:31:46 PM  1003      4799   13.60    4.20    0.00    0.20   17.80     0  factorio
05:31:51 PM  1003      4799   16.40    4.00    0.00    0.00   20.40     8  factorio
05:31:56 PM  1003      4799   12.80    5.20    0.00    0.00   18.00     6  factorio
05:32:01 PM  1003      4799   13.60    3.80    0.00    0.20   17.40     6  factorio
05:32:06 PM  1003      4799   12.40    5.20    0.00    0.00   17.60     0  factorio
05:32:11 PM  1003      4799   12.60    4.20    0.00    0.20   16.80     7  factorio

Average:     1003      4799   13.14    4.29    0.00    0.09   17.43     -  factorio

Code: Select all

top - 17:36:21 up 20:47,  5 users,  load average: 0.81, 1.18, 1.17
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.1 us,  0.6 sy,  0.0 ni, 95.0 id,  2.2 wa,  0.6 hi,  0.6 si,  0.0 st
MiB Mem :  15967.7 total,   3755.6 free,   6393.5 used,   5818.5 buff/cache
MiB Swap:  16272.0 total,  16270.7 free,      1.2 used.   9772.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   4799 factorio  20   0 2267640 553164  36224 S  20.0   3.4   9:37.92 factorio

top - 17:36:26 up 20:47,  5 users,  load average: 0.74, 1.16, 1.17
Tasks:   1 total,   0 running,   1 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.4 us,  0.6 sy,  0.0 ni, 97.9 id,  0.0 wa,  0.1 hi,  0.1 si,  0.0 st
MiB Mem :  15967.7 total,   3754.4 free,   6394.6 used,   5818.7 buff/cache
MiB Swap:  16272.0 total,  16270.7 free,      1.2 used.   9771.4 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   4799 factorio  20   0 2267640 553164  36224 S  15.8   3.4   9:38.71 factorio

Code: Select all

Game FPS/UPS is around 54/61 to 60/60
latency 11ticks to 13ticks
buffer 4/2/0 to 6/4/2
jumps 0

[Rseding91]

CPU usage doesn't really mean anything when it comes to program performance. Virtually no CPUs are made equal and virtually no games can utilize all CPUs on a processor.

Just like growing a baby, what takes 1 woman 9 months to do can't be done in 1 month with 9 women. Most game logic can not utilize multiple CPU cores and so you will typically see 1-2 CPU cores at 100%.

[Rseding91]

The relative CPU performance of the server CPU you're using compared to a modern desktop CPU: https://www.cpubenchmark.net/compare/13 ... en-9-9900X

[tmdag]

Thanks, what I understand that the game is mostly single threaded, and therefore only “20% usage” is observed.
So we are just talking about the performance per thread.

So what would be the ideal spec for the headless server here?

[clarity]

You folk are not reading the CPU percentage correctly. When top shows 100% CPU usage, that is a single core fully utilized. If a program utilizes multiple cores, the usage will go over 100%.

Below 20% means factorio is barely using the CPU.

[tmdag]

Thank you @clarity ! For a moment I thought i am going crazy here. You are indeed right. 20% is just 20% of a single core, not a 20% of a total CPU.

Therefore headless server indeed is utilising only 20% of a single core within the CPU and statement that CPU is too slow is not justified.

Also still would love to hear - what CPU you guys are suggesting that would indeed be expected for headless server. I don't really understand this pushback considering that i am not the only one with headless server issues.


Would love to get a professional support rather than comparing my cpu to a pregnancy...

[tmdag]

Is there any possibility of getting some support on this topic?
At least some more in-depth information for what processor is actually considered OK for the headless server that can guarantee stability over the game?

I would love to know which specific aspects of CPU we should be looking for. This could help others with a similar issues as pointed to external links online.

Thank you!

[BlueTemplar]

Have you managed to pinpoint when it starts lagging ? (Stage of the game ? How many and which players connect ?)

Have you tried slowing down game speed ?

P.S.: How fast is your RAM ? AFAIK that can be a bottleneck for Factorio too.