[Optimization] Improve map download netcode

[luziferius]

The network code used to download multiplayer maps can be optimized, resulting in generally faster map downloads.

Here in Germany, many ISPs offer highly asymmetric DSL lines, like 16MBit/s down, 0.5MBit/s up. Because the factorio network code uses about 4 times the number of packages in upload direction than needed, factorio quickly saturates the upload bandwidth available. For me, this leads to degraded map download speeds, even if the server is hosted on an uncongested symmetrical Gigabit connection.

To show the current state, I’ve created two screenshots of my system monitor, while it displays network metrics:

Purely as a reference, this is my current download speed when loading a regular, big file somewhere from the internet. From top to bottom it is data rate, packet rate, error rate. Here, the download line is saturated, while the upload line is not.

Data and packet rate of a regular file download.

Data rate regular download.png (99.21 KiB) Viewed 3445 times

The same situation, but using factorio to download a larger multiplayer map.

Data and packet rate during factorio map download.

Data rate factorio map download.png (2.31 MiB) Viewed 3445 times

For me, the download rate is capped at about 660 KiB/s, regardless of the used server. I have never seen higher download rates in factorio.

When you compare the packet rates, you can see that a regular download uses about 540-580 packets/s for TCP ACKs. Factorio somehow takes over 1800 packets/s, which is even more than the received package rate.

My suggestion is to reduce the number of acknowledgement packages sent, if you can control this. This will lead to less upload congestion and in turn, improve the map download speed for everyone with such asymmetrical network connections.

[weaknespase]

Probably too small payload for each transmitted packet. 660K/1800 ~ 320 bytes of payload per packet, i'm not sure they even using TCP for bulk data transfer, and I think they should.

[luziferius]

Probably too small payload for each transmitted packet. 660K/1800 ~ 320 bytes of payload per packet, i'm not sure they even using TCP for bulk data transfer, and I think they should.

Its actually 1300 packages/s incoming, and 1800 packages/s sent.
While downloading maps, Factorio sends more packages than it receives.

[SyncViews]

Probably too small payload for each transmitted packet. 660K/1800 ~ 320 bytes of payload per packet, i'm not sure they even using TCP for bulk data transfer, and I think they should.

If I recall they said previously that they don't use TCP because it is another thing for firewalls and NAT to break when not using port forwarded / public IP servers.

It has been some years since I programmed mixed TCP/UDP, but I also recall there are issues with TCP sending more and more until it causes packet loss, probably for a home connection on the servers upload side. You dont want packet loss there as it also impacts the UDP for all the players already connected.

[luziferius]

It has been some years since I programmed mixed TCP/UDP, but I also recall there are issues with TCP sending more and more until it causes packet loss […]

Yeah, thats the TCP congestion control algorithm (https://en.wikipedia.org/wiki/TCP_congestion_control). Many algorithms use packet loss as an indicator.
Newer TCP stacks tend to use algorithms that quickly saturate the line and don’t back off that far, if they hit packet loss conditions.
If they break UDP streams, that may be bad for servers with low upload bandwidth.

----

Increasing the overall packet size for incoming packets should improve the situation here. Currently, the packet size is about 500 bytes (660KiB/1300 packets). A normal network MTU is about 1500 bytes (including headers). so there is at least some room for improvement. Going too large may introduce packet fragmentation during the transport. Especially, if there is some network tunneling involved.

----

But I think, Factorio shouldn’t send that many requests per second, at all. TCP can acknowledge the data using a quarter of the packages Factorio uses.

[SyncViews]

Increasing the overall packet size for incoming packets should improve the situation here. Currently, the packet size is about 500 bytes (660KiB/1300 packets). A normal network MTU is about 1500 bytes (including headers). so there is at least some room for improvement. Going too large may introduce packet fragmentation during the transport. Especially, if there is some network tunneling involved.

1500 is for Ethernet, so LAN. WiFi I think has even more. Finding the MTU over the internet is more tricky, I recall that it is at least 500 and something bytes for IPv4 internet, but not sure on what the current state is.

But I think, Factorio shouldn’t send that many requests per second, at all. TCP can acknowledge the data using a quarter of the packages Factorio uses.

Yes this seems to be the bottleneck. Not looked at Factorio under say Wireshark (normally dealing either with regular stuff like HTTP, or audio/video streams), but would expect it to combine ACK packets

Not that TCP is perfect, a lot of the "optimisations" can end up seeming counter productive in the protocols I have worked with, at least without being careful anyway.

[diederTheBeater]

I have had a look into wireshark recently

[NotRexButCaesar]

I have had a look into wireshark recently

Is this in the correct topic? The post you quoted does not exist, and the fact that you have had a look into wireshark recently does not seem to be related to the discussion here.