Very slow downloads from the website

[Crysesa]

Yes, downloads in-game cancel after a while for large mods.

download.PNG (16.69 KiB) Viewed 3262 times

But! Using a dedicated downloader like jdownloader I get somewhat faster speeds, around 300 kbps which is still way slower than my actual connection speed, and I can sometimes successfully download mods.

[Crysesa]

as i'm not the only person living in BC, I imagine there's more people who experience this

BC as in British Columbia? To troubleshoot this further I would like you to do some network path testing with a tool like https://en.wikipedia.org/wiki/MTR_(software) or https://en.wikipedia.org/wiki/PathPing

Could you please do traces for us1.factorio.com, eu1.factorio.com, jp1.factorio.com, mods-data.factorio.com and cdn.factorio.com

Reports of these tools are a bit verbose and contain somewhat private information (IPs of your home network, ISP gateway, etc) so feel free to send them to support@factorio.com instead of posting them here.

Using winmtr I tried to test mods-data.factorio.com. I am not a network specialist, but I believe the packet loss rate is quite high?

winmtr_mods-data_downl.png (15.2 KiB) Viewed 3244 times

[orzelek]

as i'm not the only person living in BC, I imagine there's more people who experience this

BC as in British Columbia? To troubleshoot this further I would like you to do some network path testing with a tool like https://en.wikipedia.org/wiki/MTR_(software) or https://en.wikipedia.org/wiki/PathPing

Could you please do traces for us1.factorio.com, eu1.factorio.com, jp1.factorio.com, mods-data.factorio.com and cdn.factorio.com

Reports of these tools are a bit verbose and contain somewhat private information (IPs of your home network, ISP gateway, etc) so feel free to send them to support@factorio.com instead of posting them here.

Using winmtr I tried to test mods-data.factorio.com. I am not a network specialist, but I believe the packet loss rate is quite high?


winmtr_mods-data_downl.png

That first 87% loss is strange since packets seem to pass further up anyway.
But those losses near end that can cut up to 50% of packets will wreak havoc with factorio transfer mechanics (and will badly mess even tcp/ip) since transfer acknowledges and packets will get lost a lot and it might get stuck trying to retransmit packets. So in the end it will be doing lots of transmit/retransmit and your speed will plummet.

From what I recall I had issues in online games when I had 5-10% drops on one stpe and after reporting that to ISP (when I managed to reach properl technical person) it was considered a problem and traced to defective network equipment.

For comparison my trace to mods url doesn't go through twelve99 at all:

image_2022-01-16_205234.png (39.91 KiB) Viewed 3221 times

[ptx0]

your link doesn't go through twelve99 because your ISP peers with Cogent, which is a far better ISP than GTT or Twelve99.

[donny]

I also just tried downloading mods via the ingame downloader. It takes minutes until it shows the modlist and minutes until the download button can be seen. the download itself didn't take long. I'm located in Germany and use Vodafone DSL. Do you guys still have problems?
*Update* I read somewhere that Avast causes problems. This solved my problem. But one mod stubbornly doesn't want to be downloaded: https://mods.factorio.com/mod/rewire-tool

[ptx0]

I also just tried downloading mods via the ingame downloader. It takes minutes until it shows the modlist and minutes until the download button can be seen. the download itself didn't take long. I'm located in Germany and use Vodafone DSL. Do you guys still have problems?
*Update* I read somewhere that Avast causes problems. This solved my problem. But one mod stubbornly doesn't want to be downloaded: https://mods.factorio.com/mod/rewire-tool

try it again. some changes hadn't fully gone through yet, i think. it works perfectly for me now.

[nuhll]

might i ask why use a cdn instead of just getting like 5 servers in different regions?

e.g. germany for europe, something asian, some australian, us left, us right... something like this? (i mean you see better where ppl are from)

that way you can easier switch and or control how your stuff gets served? (and if you dont want to fiddle around with "auto select correct server" just make it adjustable, you could preselct via language tho)

i also had crazy problems with your downloads, but thankfully never since you changed the first time (atleast not that bad i expiericnced it at that time)

[BlueTemplar]

just getting like 5 servers in different regions

Which would then be a CDN ?

[ptx0]

sorry for the necro-bump. i've got follow-up information from investigating issues this morning.

first off, i'm so sorry for forcing Wube to waste time on something I should have spent more time validating. even opening up Wireshark would have shown me a few questionable details.

secondly, i have a fever. I might not be coherent. however, this morning I had the idea to investigate TCP Window Scaling and its function in my life, after having to restart a download every 30 seconds so that it completed.

basically, disabling TCP Window Scaling forces (on Linux, anyway) the window size to a fixed 65535 which means we send as much data as a TCP packet naturally allows, before considering the 'scaling factor'. This results in a CONSTANT throughput of 10mbps, but it can't go any higher on a single TCP session. Not with my current RTT latency, anyway. For the same Window size on different latency connections, you can get varying throughput. 10mbps is somewhat ideal, i've had 5mbps this way to further distant networks.

but then I went into Windows to figure out whether it's a Linux-specific problem. Something to note here, is, if I'm tethered to my phone that uses the same LTE provider, it works fine! My theory is there's something wrong with the modem they provide, that I can not change. I've put it into Bridge mode and turned off any kind of filtering/QoS and alas, nothing changes this behaviour...

except...

putting these values into sysctl.conf:

Code: Select all

net.core.wmem_max = 4194304
net.core.rmem_max = 12582912
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_wmem = 4096 87380 4194304

and reenabling TCP Window Scaling. Now, my downloads start out "slow" and gain speed as they go, instead of beginning deceptively fast, and cliff-diving.

I should note that using `iperf3` to test the Window size is what led me to searching how to increase these values, as, the default max window size on Linux, is a mere 250kB

For what it's worth, the limits are the same on Windows.

anyone with a very fast network connection (pretty much any modern ISP) should revisit their TCP tunings, because despite praise that things "Just Work" a lot more often these days, sometimes, you still have to biff it.

[nuhll]

just getting like 5 servers in different regions

Which would then be a CDN ?

No.
Static servers in different regions. Like steam does. (i know each region is not only one server)

But the speed problems never came back after they fixed it, so im happy anyway.