[1.0.0] Can't download map on headless server over 1.8MB

[bigmond]

I cannot play a game on a headless server whenever the map exceeds 2MB, roughly.

The map starts to download and gets to just under 2Mb and then stops. The amount vaires a little but typically 1.8 M. It then drops me from the game.
I have tested this on a wide variety of servers.

M45A
M45C
M45D
M45G
Factorio.one NaEast-A
My own server
Many many more

I can connect just fine to a player hosted game. It is only headless servers that fail to connect.
Since I tried hosting the server myself I tried connecting to it via WAN and via LAN. I can connect just fine over LAN.
Playeers connecting to my PC when I host the game can connect just fine. But players connecting to my server externally cannot.
This bug is consitent. I have attempted connection multiple times.
I am not th eonly person having this issue. A member of my discord community cannot connect to any server either.

This can't be a firewall issue, it wouldnt even start the download. Infact if I forward
I have triple checked the firewall config on the server itself and on the router. Ports are open.

I have completely reinstalled factorio several times, including clearing the config in %appdata%.

As a work around I found out that if I limit the upload speed on the server to 500kbps then I can connect. It still stutters at 2MB but it then resumes. This is not a practical solution. It takes 15 min to get into a game an dthi sonly works for my own server. Obviously I cannot change server settings for other servers.

Log file attached x3. 2Vanilla, one modded.

Previous post discussing the issue
viewtopic.php?f=49&t=85747
viewtopic.php?f=49&t=86315

[bigmond]

Some additional info to add to this.
When the connection drops I cannot connect to the server again for 60 seconds. Connections to the server are blocked. After 60 seconds I can connect again.

We have been doing these tests with the server firewall disabled.

Connecting to a player hosted game does work, but...
After downloading 2MB, the server says server not responding and the download stops. This lasts apoproximately 10 seconds and then it resumes downloading, downloads approximately 2MB and repeats.

[Rseding91]

Thanks for the report however I have no way to reproduce this and as such I have no way to debug what's going wrong and or try to fix it.

Until there's some way for me to reproduce it; there's nothing I can do.

[Loewchen]

In the past disconnections when downloading a certain amount of the save file were caused a firewall getting triggered by the large amount of UDP packets.

[bigmond]

Hi guys.

I have found the culprit. It is the UDP flood defence on the router.

Disabling flood defence allows download of the map. I tried increasing the limit but I had to put it to 4500p/s in order to not trigger the flood defence.

I have tried a couple of servers and get into them just fine with udp flood defence disabled.

I'm inclined to call this a better work around rather than a fix since this is a security issue to disable and most users wont have access to chage this (although I do feel a little dumb I didnt at least try this earlier).

This also leaves the question as to why a headless server triggers the flood defence and a player hosted game does not, this is why I'm still enclined to call this a bug. Is there something in a player hosted game that works differently with regards to the networking? Is it throttled maybe?

I am amazed that factorio is even capable of triggering the flood defence, I have been hosting several games for years which require large downloads and have never managed to trigger it.

In any case I think at a minimum we need some sort of work around that does not open a security hole and can be done by an end user. Perhaps at a suggestion if a connection is dropped waith 60 seconds and retry using a throttled connection, this isnt exactly elegant but its a start

I'll have a word with the other players on my server who have the same problem, if we come up with anythign new I'll let you know. Is there anything else I can help with?

[Loewchen]

You can set the maximum map upload speed in the server settings if you rather constrict that than your firewall. NaB.

[bigmond]

This is a bug.

Restricting max upload speed takes 30 min to get into a game right now.

Plus you cant restrict it to that on every server, your solution does not allow me to play on servers I don't host myself

That is not a solution.

[bigmond]

Right I've been doing some analysis of what packets are sent and wow!

Factorio sent 735,713 packets just to download the map.
I have changed some of the settings on some of the games I own to try and increase network traffic and I have not managed to get any other game to 1000.

56.68% of the packets sent are under 64bytes in size (which includes the 42 byte header) and most of those only contain 5 bytes of data. The larger packets are 508 bytes (550 with header)

It makes absolutely no sense to send 5 byte packets as a map download, let alone the majority of packets. I really recommend you up this to at least 1000. You have 1500 to play with, some broadband networks are a little lower than this, of the 3 I just checked 1442, 1472, 1492. For the sake of making sure its reliable 1400 should be a safe size and would give a more efficient download size.

This would really explain why even with the UDP flood defence disabled the download speed is extremely slow. With just 5 data bytes in a onl y10% of the transmission is actual data.

This said why use 5 byte packets in the first place. The advantage of small packets is lower latency, but even games that are very latency dependent often use packets much larger than this minimum because of how inefficient it is.

Add to this the fact that many poor quality ISP provided routers struggle when sent a large number of packets quickly, this would perhaps explain some of the reported latency issues that so far have gone unexplained.

You guys really need to at a minimum find a way to throttle the packets so as not to trigger common DOS protections and you need to question why so many packets during map download are so small. Even the larger packets could do with being 2.5x larger.

[Rseding91]

Most of those small packets is the game requesting the next map block be sent or confirming it got the one it requested.

So for each requested block 2 packets get sent to the server and 1 comes from the server.

As for the packet size: https://en.wikipedia.org/wiki/IPv4#:~:t ... %20packets and issues with fragmentation causing map downloads to fail in the past: 25178

[bigmond]

OK how about you set headless servers to use the same mechanism that player hosted games does.

[Rseding91]

OK how about you set headless servers to use the same mechanism that player hosted games does.

They do. What are you referring to?

[bigmond]

I am refering to the fact that the issue above occurs on headless servers only, it does not occur on player hosted games.

[Rseding91]

I am refering to the fact that the issue above occurs on headless servers only, it does not occur on player hosted games.

The network code for both games is identical; as in we don't have 2 different sets of code. It's the same.

The only difference I can think of is if the packets get routed through steam or not since you can't host a dedicated steam-networking enabled server. But that difference is 'send packet data to steam' vs 'send packet data to the operating system socket'. The actual data sent, frequency of the packets, and number are identical.