[0.17.74] Multiplayer latency under ideal conditions is abysmal

[Hornwitser]

When connecting over a gigabit lossless sub-millisecond latency link the game reports a jaw dropping 7-9 ticks of latency between the server and the client (this is looking at the show-multiplayer-statistics measure enabled from the debug menu.) At first glance it might not sound like much but it works out to a latency between 117-150ms which is more than what a typical trans Atlantic connection will give you in ping. And this is a parasitic latency that all multiplayer connection starts with and they can only become worse than this.

This issue has been reported before in this post where you state that you have latency hiding so it's not a problem. This is objectively false. Both driving vehicles and combat does not have any latency hiding, trying to do either of these with a minimal latency that's worse than a trans Atlantic connection is a terrible experience. Things like multiplayer combat with shotguns or car races are simply impossible with the 150ms+ delay. And that's just the tip of the iceberg. The editor doesn't have latency hiding for most actions making it slugging in multiplayer. Mod interfaces do not have latency hiding, so all mod interfaces have a sluggish feel that just nails them as being second class interfaces in multiplayer. Desync reports inherit this delay making them harder to interpret. And if you have this cool dexterity based mod idea you can forget about it working in multiplayer.

Looking at a packet dump of the connection it's easy to see that approximately 2-4 ticks of this latency is caused by the bunching up of tick closures in groups of two and sending packets out at a rate of 30 per second. Both the client and the server has to wait a full tick before it can send out the result because of this. Why do you do this? What is the motivation behind this? If you count IP and UPD headers an empty ClientToServerHeartbeat with 2 tick closures is 47 bytes, for ServerToClientHeartbeat it's 65 bytes. If you send 1 tick on each you get 43 bytes and 49 bytes respectively. This works out to rates of 26.8 kbit/s total for bunched in 2 ticks and 44.1kbit/s for sending each ticks by themselves. Which in turn works out to 12 MB/hour vs 20 MB/hour or a saving of 8 MB/hour of data an hour. With the typical multiplayer map size being 20-50MB this saving of 8 MB/hour does not make any sense and certainly doesn't weigh up to the parasitic 30-60ms of latency that it introduces.

Another thing a packet dump reveals is that there is a heck of a lot of delay in the network handling of the client. Under typical conditions the client would send for example inputs for tick 9 and 10 while the server sends at the same time the closures of tick 6 and 7. Somehow the client manages to waste 60ms of time in its processing of inputs and network packets. Thinking of this issue I don't see any reason why the latency has to be more than network latency + jitter factor + server tick computation time, which under the right condition could be less than half a tick.

[Klonan]

I will have a chat with our lead developer about it, whether it is worth changing it or adding some config (as he has mentioned before)

But overall this isn't a bug, so doesn't belong in bug reports

[Hornwitser]

I agree that the 2-4 ticks of latency introduced by bunching up ticks into pairs is by design, but thinking about this more there's something about those 4 or so unaccounted ticks of latency in the system here that doesn't make sense. There seems to be parts of the client that is holding on to tick closures for much longer than it should. So I spent some time making a diagram of the ticks as sent over the wire based on a packet capture (time flows to the right):

tick-delay.png (30.54 KiB) Viewed 1086 times

Observations I can make from this:
- The server is sending tick closures approximately once every 34ms regardless of when it actually completed computing a tick. This introduces a delay that's on average half a tick. You should change this so that the server checks after completing a tick if some requisite amount of time has passed since last sending out tick closures and if that is the case immediately send them out.
- The time it takes from the client sends inputs for tick 6 and to the server sends tick 6 back there's 73ms / ~4.4 ticks of delay. Even rounding up to 5 there's two whole ticks of delay unaccounted for in the client.
- There's no synchronization between the server and the client, which adds another delay averaging half a tick.

If I set the game speed to 0.1 the latency is still ends up at 5-7 while the real network latency becomes less than one tick, what are those 4-6 ticks spent on?

[TruePikachu]

Perhaps the server is waiting for a constant period of time for all the client inputs for the tick to arrive, before it sends the finalized tick out to everyone? If that's the case, while this would strictly still be NaB, it would be an improvement to have that time be a config option, as well as potentially letting the server send once it receives all inputs (if this would still allow the netcode to be stable in terms of timing, ofc).

I don't have experience with netcode, so I can't come up with any better ideas.