Multiplayer latency feels high, but reports very low

[Aoz]

I have a dedicated server that I play on, and the latency feels extremely bad (super noticeable when driving a car)
but everything that exists says that it's very good. I would say it's closer to 500ms latency than what it reports,
judging by how it feels to turn the car. This problem is not apparent in my Single Player game. It also feels bad
when I take things chests as it seems to take a "while" (short time really but it's very noticeable).

Clients are all 0.14.22 (pretty reasonable gaming PCs, all have tons of ram, GF 770GTX or above with i7s)

About the server:
- Linode, lots of bandwidth, about ~700 mb of ram free + swap free.
- 28 ms ping to it via ping command line utility (0% loss over many packets)
- Checked disk, plenty of gigabytes remaining
- Checked 24/hr graphs, nothing is strained during play time, cpu never over 22%, network barely utilized, disk tame
- Speed from computer to server: 1.4 MB/s up (scp transfer up)
- Speed from server to computer: 5.9 MB/s down (caddy serving a file over http) (map also transfers very quickly)

Game server info:
- 0.14.22, no mods
- 11-17 ms latency in debug menu (28ms in the server browser)
- UPS/FPS always at 60.0 in debug menu
- 3 Players max, but problem is apparent with only me on the server
- Problem occurred immediately when starting the map, but the car only made me notice it enough to try to dig in

So I guess my question is, with latency like this, and server and client performance like this, why when I turn my car
or take some Iron Plates out of a crate does it feel like I'm pulling a computing muscle? Is this just a limitation of the game?

[DaveMcW]

Factorio latency is measured in ticks. Each tick is 17 ms on normal speed, or more on a slow map.

I agree it is very confusing.

[Aoz]

Okay, so the debug menu latency which reported 11 or 17, is actually not the number. But my real ping to the server is still 28ms, and that shows on the Server Browser as well which I assume couldn't measure in ticks. But even if that's the case, it wouldn't explain the extremely laggy movements while in vehicles which is what I'm more concerned about.

[torne]

Sounds like your latency is actually fairly high all the time. The game hides the latency of moving your character around by doing it predictively on the client side. This doesn't apply to interacting with items or driving, which need a reply from the server.

If the debug menu says your latency is 11 or 17 that's in ticks as was mentioned, and 17 ticks is 283 milliseconds, which would make driving pretty challenging indeed

Maybe the server (or you) are getting throttled somehow and it's dropping packets? I think resending them will increase the effective latency even if the actual round trip time is low.

[Klonan]

Okay, so the debug menu latency which reported 11 or 17, is actually not the number. But my real ping to the server is still 28ms, and that shows on the Server Browser as well which I assume couldn't measure in ticks. But even if that's the case, it wouldn't explain the extremely laggy movements while in vehicles which is what I'm more concerned about.

Vehicle movement isn't part of the latency state / latency hiding

[Aoz]

I measured the packet loss and it's 0. At least with ping, maybe there's something going on on the network side. I guess I'd ask if others have experienced this. Is it just me that feels like multiplayer vehicle handling is hard as heck to deal with, even at low pings (I consider 28ms to be pretty low)?

Also other ideas on how to troubleshoot?
I'm not even sure how I'd measure packet loss other than using ping.
I could TCP dump both sides and write some code to ensure that they all came? Seems painful tho lol.

[Rseding91]

I measured the packet loss and it's 0. At least with ping, maybe there's something going on on the network side. I guess I'd ask if others have experienced this. Is it just me that feels like multiplayer vehicle handling is hard as heck to deal with, even at low pings (I consider 28ms to be pretty low)?

Also other ideas on how to troubleshoot?
I'm not even sure how I'd measure packet loss other than using ping.
I could TCP dump both sides and write some code to ensure that they all came? Seems painful tho lol.

11-17 latency in the debug menu means 176-272 MS ping not 28 MS. Also latency is not a pure measurement of ping - it's your computers ability to keep up with the server and the stability of your connection to/from the server.

[Aoz]

11-17 latency in the debug menu means 176-272 MS ping not 28 MS. Also latency is not a pure measurement of ping - it's your computers ability to keep up with the server and the stability of your connection to/from the server.

I'm actually shocked to see a latency of 176-272ms given the systems I'm dealing with. So given that, how can I work on getting this lower? Is there any information or steps I can take to determine where the bottlenecks are using the game's tools to see if something's bogging down during the simulation (though I see 60/60 FPS/UPS at the top right)?

I'll attempt to benchmark the UDP loss between my two systems and see how that goes. That way we can decide if the network is truly a problem. I'll also get some "top" measurements going on to see what resource utilization on the machines is.

Thanks for the help so far everyone. I'll keep prodding.

[Aoz]

I tested my home connection against the server hosting factorio as well as an AWS m4.xlarge, they both fell over hard with total loss around 16 Mbits but were perfectly fine around 8 Mbits. 0 loss for quite some time, I only show the beginning of the test because it doesn't really change. I -do- notice a few packets scramble around here and there. You can see sometimes one extra dgram will arrive or one less will be sent. Maybe this is the cause? It seemed ever so slightly more stable with the AWS instance in terms of dgrams each tick of this program.

Do we know how much bandwidth Factorio uses? If it's over 1 MB/s I might actually have an issue on my hand.

Client:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  4]   0.00-1.00   sec   880 KBytes  7.21 Mbits/sec  110  
[  4]   1.00-2.00   sec   976 KBytes  8.00 Mbits/sec  122  
[  4]   2.00-3.00   sec   984 KBytes  8.06 Mbits/sec  123  
[  4]   3.00-4.00   sec   976 KBytes  8.00 Mbits/sec  122  
[  4]   4.00-5.00   sec   976 KBytes  8.00 Mbits/sec  122  
[  4]   5.00-6.00   sec   976 KBytes  8.00 Mbits/sec  122  
[  4]   6.00-7.00   sec   976 KBytes  8.00 Mbits/sec  122  
[  4]   7.00-8.00   sec   976 KBytes  8.00 Mbits/sec  122
...

Server:

Code: Select all

[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec   840 KBytes  6.88 Mbits/sec  9.911 ms  0/105 (0%)  
[  5]   1.00-2.00   sec   976 KBytes  8.00 Mbits/sec  9.939 ms  0/122 (0%)  
[  5]   2.00-3.00   sec   976 KBytes  8.00 Mbits/sec  9.930 ms  0/122 (0%)  
[  5]   3.00-4.00   sec   976 KBytes  8.00 Mbits/sec  9.932 ms  0/122 (0%)  
[  5]   4.00-5.00   sec   976 KBytes  8.00 Mbits/sec  9.949 ms  0/122 (0%)  
[  5]   5.00-6.00   sec   984 KBytes  8.06 Mbits/sec  10.063 ms  0/123 (0%)  
[  5]   6.00-7.00   sec   976 KBytes  8.00 Mbits/sec  10.015 ms  0/122 (0%)  
[  5]   7.00-8.00   sec   976 KBytes  8.00 Mbits/sec  10.025 ms  0/122 (0%)
...

I mean I'm demonstrating 0 loss with very low ping. So I wouldn't think it's the network that's contributing to my 280ms latency. I'll try monitoring resource usage next. Any support on where to look for problems would be appreciated.

As Klonan mentioned, cars aren't part of the latency hiding thing, so obviously I'm just feeling the 280ms latency I somehow have. Just trying to get to the bottom of why I have that high latency now.

[Rseding91]

It depends on the amount of players and what they're doing as to how much bandwidth the game uses.

I've looked over our network code and there's nothing that limits throughput or latency unless you have the option enabled and even then it's only when downloading the map from the server. One other area is the server minimum-latency setting which can impact peer latency (forces a minimum as the name implies).

We process network messages as soon as the operating system gives them to us and send a response as soon as the code execution reaches that point. The latency is calculated based off how long it takes for the peers messages to finally be received after the requests for the messages are sent from the server (round trip time).

Something between those two is slowing/delaying Factorio data. Normal latency or something else I couldn't say but at this point I can't see anything in how we handle network communication that would cause someone to have a higher latency than required.

[edsaperia]

Did you ever resolve this issue?

I'm experiencing a similar problem - fast computers, low round-trip-time to the server (10ms), negligible packet loss, but my game experiences latency of 50 - 150 ticks, making multiplayer gaming absolutely impossible