Variable but crippling mulitplayer lag at 70+ hrs

[Talrich]

We have a 3 to 6 player vanilla map (no mods) which had undetectable lag at ~55 hours. We're now 78 hours in and the lag is often crippling for everyone except the server.

The system specs for our default host are:

Motherboard: Gigabyte GA-990FXA-UD3 AM3+
Processor: AMD Phenom II x6 1100T Black Edition, 3.3 GHz.
Video: Dual ATI Radeon HD 5770 1GB GDDR5 PCIe x16 in CrossFireX mode.
RAM: 16GB Corsair Vengeance 240 pin DDR3 1600 (Timing 9-9-9-24)
Network: FIOS Quantum Gateway, 125 Mbps service.
Network Adapter: 1 Gbps on Motherboard
The game is installed on the primary boot drive which is a 480GB AData Premier SP550 SATA III Solid State Drive.

Short Description from the host:

My system is normally the host for the game. When I am hosting I usually run in the 40-50 FPS/UPS range. For reference I generally get up to 60 FPS/UPS while running solo. Though I have experienced slow movement at times while others are complaining of lag, as the host I generally encounter few if any problems, as would be expected.

If someone else runs as the host my FPS/UPS numbers drop under 10, many times under 5. Latency numbers go up to around 35 and a number called "jumps" continuously climbs well above 3000. I am often unable to even move. Trains that go by on my screen will "teleport" rather than move smoothly. All of this assumes I can even get into the game. There were times where upon loading up the game it would say it was "catching up" but the progress bar would move backwards until an error would arise stating that my computer was too slow to be able to play.

Player #2 (alternate host)
Motherboard: MSI X99A Raider
Processor: Intel Core i7 6800K 6-core 3.4GHz/3.6GHz Turbo 15MB L3 Cache with HyperThreading
Video: NVIDIA GeForce GTX 1070 8GB GDDR5
RAM: 16GB HyperX FURYT DDR4- 2666
Network: FIOS
Network Adapter: 1 Gbps on Motherboard

Player #3 (playing on a toaster)
Intel i3 CPU M370 @2.4GHz 4GB RAM 64bit system
Network is FiOS 100Mbit service (usual speeds averaging around 30Mbit at peak times)

We'll be glad to share logs, but aren't sure where to find them. Please advise.

Shifting our ore operations to underground belts seemed to help for awhile, but only a bit, and the night-to-night variation in performance seems to be pretty extreme.

I hope someone can help. We all love the game and several of us look forward to these group games all week.

Best,

[DaveMcW]

The game is really bad at optimizing itself for big/slow multiplayer maps. The developers would rather focus on making everything run faster than solve this problem. (Little do they know that people will just use their better engine to make a bigger map...)

Anyway, use this command to optimize it yourself:

Code: Select all

/c game.speed = 0.8
/c game.speed = 0.5
/c game.speed = 0.3

[Rseding91]

The game is really bad at optimizing itself for big/slow multiplayer maps. The developers would rather focus on making everything run faster than solve this problem. (Little do they know that people will just use their better engine to make a bigger map...)

Anyway, use this command to optimize it yourself:

Code: Select all

/c game.speed = 0.8
/c game.speed = 0.5
/c game.speed = 0.3

What? The relative performance of the game on a small map or a big map is the same. Simply a bigger map has more to process so it takes longer.

[Talrich]

We will try modifying the game speed.

Given that player #1 sometimes cannot even join the alternate host (#2) even when the game world is paused, it seems like something more than game speed.

[can00336]

The computational requirements for MP are higher than that of SP:
https://www.reddit.com/r/factorio/comme ... ?context=1

Rseding91: Single player doesn't have the other player to simulate, the CRC checks each tick, or the latency state to calculate + the overhead of sending/receiving network packets.

You will need to lower game speed until the person with the slowest computer can keep up.
Also, don't open the train viewer window if you have a lot of trains and are on the edge of being able to keep up. You'll be stuck at 5 UPS unable to close the window or even exit the game without alt+F4 (assuming you aren't the server).

[garec76]

We'll try altering the game speed sometime over the weekend. To date though, I've never noticed any effect any time I try to use any of the commands. We'll see I guess!

[Caennanu]

Maybe this helps in finding the root cause.

A while back i had exactly the same issue, but this wasn't computer related.
When a friend of mine would host on his pc (i7 32gb, sli setup) the game initially ran fine.
then the map started getting bigger and bigger, eventually i was unable to keep up with the server and was having the same issues as you are stating down here.
Thats when i decided to host it on my machine (amd 8320 4ghz, crossfire 280x, 24gb ram) theoretically a lesser pc than my friend.
This however, ran fine for me and all the other players.

Thats when we decided to host it on a third friend's pc, it worked for a little while than the exact same issues appeared unless i was hosting it.
The difference between the 3? Internet upload speeds!

First host had roughly 3Mbit upload, the third host had 5Mbit, however i am on a fiber connection with 500Mbit upload.
So as a matter of test, i decided to limit my upload thru my router, and what do you know? the same issues started to occur for the other players as soon as i got close to matching their upload speeds.

[Talrich]

Thanks for the thought re: upload speeds. I'm glad to have another lead to chase down.

As player/host #2, I'm at ~100 Mbps download and ~120 Mbps upload. Host #1 cannot join me when the game is paused.

Player #4 reported that he's at ~200 Mbps download and ~20 Mbps upload.

Before posting, we tried speed restrictions and the command didn't seem to be having any effect. We tried it just now and things seemed stable at 0.5. We'll see if that works with the full crew on.

[kstsauveur]

My experiences strictly as a player of the map in question are consistent with OP's original post.

Net speeds: ~30-35Mbps (wireless, both up and down): 38-43 down, 30-35 up (wired; both up and down)

I've noticed (via random spot checks) the upload speeds drop down to their lower levels at "peak usage" hours more so than my downloads speeds.

Thanks to everyone for their suggestions.

[Caennanu]

you're welcome.

--- edit --

If you encounter issues with not keeping up, jump rising and fps dropping.
See my other post. Seems you need to downclock the server. dont downclock too far else youll get server not responding issues.

[Talrich]

Here's the update. The main host overclocked his CPU, and we got some decent performance out of that with the gamespeed set to 75% for several hours until a blue screen of death situation. New CPU for the host is on order and we'll see how much that helps.

[Caennanu]

so in your case, the host is too slow?
Or are you hosting on the pc you're gaming on? Then it would make sence.

[HOSH]

This is similar to what I have seen as well. The limiting factor if you have all computers running from a dedicated host is the slowest player (the toaster player), but reverse if the host is too fast all the others cannot keep up in factorio. Yes the latter is opposite logic if most games as normally you want the host to be the fastest machine of the group, see below for more info. Also the host and end users download/upload speed are a factor as you want the host at the location with the most upload speed.

Current resolution some of the community has found is to run a headless server on Windows or Linux. Preferably on Linux (I used Ubuntu Server LTS 16.06) as it takes up less resources (no GUI, just CLI) and is stable. Host changed to my location as I had the fastest upload and we had to put it on a machine with the clock speed of 2.5-2.9Ghz I left speed boost on my Ubuntu VM. Caennanu has found his runs good at 2.5Ghz on his last post. https://www.reddit.com/r/factorio/comme ... _on_linux/ is what I used for a rough reference.