[0.14.8] [Twinsen/cube] Downloading map hangs at 100%

[Cathillach]

When my friend hosts the server I often get the bug where once the eta for the map download reaches 0.0s that the game just hangs. The download speed declines slowly to 0. Eventually, the game gives me the option to quit the game or reconnect.

When I host the server, my friend gets this bug every time he tries to join my game.

This is frustrating as every now and then I get disconnected from the server suddenly. While playing normally I have no lag and the game runs fine. Then I try and reconnect to the game and this bug pops up often causing me to need to restart factorio and quite often that does not even fix the bug.

While the map is downloading, the download speed is also extremely slow. My friend has Fibre internet with speeds of 30Mbps/10Mbps. While I can easily get 2.0 Mbps downloads through steam. Yet in factorio I download the map with a maximum speed of 150kbps.

Trying to connect to the game starts at line 948 in log file.

Factorio-Error-Log.txt

Log File

(401.72 KiB) Downloaded 217 times

Save File.

[Klonan]

Thanks for the report,

Can you attach the mods file you used?
Can you check if the game loads fine without the mods installed?

[Twinsen]

Looks like you might have a really bad route between yourselves.

Can you try to use a network testing tool to measure the connection? For example https://sourceforge.net/projects/netiogui/
Compare the speeds you get in that tool to what you see in Factorio.

Also it seems that the game is trying to get as high speed as possible, clogging your connection. Try setting "Maximum upload speed" to about 5-10% lower than the speed between yourselves(not advertised speed of the ISP).

[Cathillach]

We have run the NetIO-GUI with my friend as the server and me as the client.

NetIO_GUI results


As my friend can never get into the game when I am hosting on this map he tried to connect to my game. I limited the upload speed to 200 KB/s, 150 KB/s, and 100 KB/s and he could not join any of those times either.

My Factorio Log from hosting those games.

Factorio-Error-Log.txt

(17.46 KiB) Downloaded 199 times

I tried hosting a new game with the same mods and he could join fine although the spawn point sort of ended the game right there.


List of Mods we are running:
core 0.0.0
base 0.14.9
bobconfig 0.14.0
bobinserters 0.14.0
boblibrary 0.14.2
Nucular 1.0.4
bobenemies 0.14.0
bobores 0.14.0
bobtech 0.14.0
bobplates 0.14.0
angelsrefining 0.3.2
bobassembly 0.14.0
bobelectronics 0.14.0
bobgreenhouse 0.14.0
boblogistics 0.14.2
bobmining 0.14.0
bobpower 0.14.0
bobrevamp 0.14.1
angelsaddons-oresilos 0.2.2
angelsinfiniteores 0.4.1
bobmodules 0.14.0
bobwarfare 0.14.1
bobconfig 0.14.0
bobinserters 0.14.0
bobenemies 0.14.0
bobores 0.14.0
bobtech 0.14.0
rso-mod 2.2.2
bobplates 0.14.0
angelsrefining 0.3.2
bobassembly 0.14.0
bobelectronics 0.14.0
bobgreenhouse 0.14.0
boblogistics 0.14.2
bobmining 0.14.0
bobpower 0.14.0
bobrevamp 0.14.1
bobmodules 0.14.0
bobwarfare 0.14.1
bobelectronics 0.14.0
angelsinfiniteores 0.4.1

[Riverchimp]

We played a map on 0.13 (no mods) with Cathillach hosting and we had some troubles then as well. At first it was fine, but as the map file got larger (>15MB) we starting seeing this 100% loaded hang. It would not always happen but as the file got larger the chance it happened increased. Our final save was 27 MB and I would be able to connect about 1/3 of the time.

Our current map file is 13 MB and I cannot connect at all. (Because of this I have been hosting it so we could play).
When I try to connect I always get two "TIMEOUT!" messages within the first couple of seconds of the download. This is sometimes (not always) followed by one or more "Received block <some number in 1900-2200 range> that we didn't request" messages.
Normally when I download the map it goes at about 400-500 KB/s so Cathillach limited the upload to 400 KB/s and that did not work. So we tried 100 KB/s and that also did not work but it seemed like the limit did not always kick in immediately. Sometimes the speed would still be ~400 KB/s for a while before dropping to 100 KB/s.
We can connect fine with a new map with and without mods.

I have two videos of me trying to connect. For both of these videos we used on upload cap of 100 KB/s, on the second video the cap does not seem to be used until the download is at 80% already.
https://www.dropbox.com/s/3kw0go0d0tz0c ... 1.flv?dl=0
https://www.dropbox.com/s/7o8gecuvdqkkn ... 2.flv?dl=0
Attached are the logs for only the second video. (I lost the logs from the first )

[Rippie]

Trying to play on the Factorio EU MMO today.
Definitely tried about 10-15 times.
Managed to connect once or twice.
I got numerous times where the map fully downloads, and stays on 100%, ETA on 0.0s, and the download speed gradually decreasing to 0.

Tried joining other multiplayer games, every single one fully downloads the map to 100%, after which it simply stops and does nothing.
For me, it has been happening since 0.12.xx.

Since the "downloading map" window never closes / times out, there is no error / crash and I have to disconnect manually.
According to the log file, it's just a huge timeout, even though I managed to download the map with several MB's per second.

Also, after disconnecting, the following line pops up in the log:

Code: Select all

308.694 Info ClientSynchronizer.cpp:191: nextHeartbeatSequenceNumber(1468312821) cannot process synchronizer action(MapDownloadingProgressUpdate)

Maybe it has something to do with the bug?

[Twinsen]

The speed you see in the downloading window is actually broken since it shows the speed at witch you are requesting map packets, not the speed at witch you are receiving them. Based on what I see, I believe the client is requesting too many map packets, completely blocking the communication of game packets on your connection, making almost all the game packets get dropped.

It's still very tricky to know what is happening on the network. I might need to communicate and experiment with you over a few releases, since it only seems to happen on your specific connection.

Thanks for the provided screenshots and information. I was also very interested in the pings of the connection. Can you post a screenshot with both speed and ping?

[Cathillach]

We ran the program again and managed to get the pings to work, however, the got an error stating "no response from NetIO server".



While playing this evening I didn't have the connection hang error as each time I was dropped I managed to quit the server before I was disconnected. This seemed to help, but may be unrelated as I only get the 100% hang every 3rd time or so. However, I was dropped about 6-7 times in 5 hours of game time. The drops come with no warning and there were no lag spikes. It also seems to get more common when the map gets larger.

We also added some new mods to improve our next playthrough and started a new game:
TheFatController 1.0.0
Crafting_Speed_Research 0.1.4
EvoGUI 0.4.108
RailTanker 1.4.0
Warehousing 0.0.11
Orbital Ion Cannon 1.4.4

Factorio-Error-Log.txt

(603.84 KiB) Downloaded 198 times

Mods.zip

(11.57 MiB) Downloaded 187 times

[Rippie]

For me this issue is reproducable for any server that reaches over a few MB/s download speed.
I managed to join a server in the first go where the download speed crawled up to 970KB/s (see log file @ 167.579). (ping was ~30ms)
I did not manage to join a server in two attempts that had a ping of 11ms (see screenshot).
See attached screenshots(.rar) and log file.

It feels like this bug shows up when the map download finishes with a speed of >2-4 MB/s. At this speed the log entry saying "Finished download" is not triggered.

@Twinsen: I can easily reproduce this issue, since multiplayer has basically been a no-go for me because of this bug. If there is any way to increase the verbosity of the log file? Or if you can send me a version of Factorio with extra logging / debug mode, I would gladly test this out for you. I'm a software engineer by profession (we write in C/C++), so feel free to ask me anything to diagnose this issue.

Is the downloading of the map and the parsing of the downloaded bytes (chunks?) multithreaded / handled in separate threads by any chance? My gut tells me there's a race condition leading to a deadlock where the "Finished download" is not triggered due to lack of synchronization between the threads.

[Twinsen]

Thanks Rippie.
Now I see that is happening. The game packets communicate just fine, just the the map download stops.

I'll look into it.

[Twinsen]

Unfortunately I can't seem to find any obvious problem. Also tried to reproduce it. I noticed some slowdown when the map is reaching the end(which is normal) but nothing as extreme as your case.

Rippie, did you try multiplayer prior to 0.14? Did you have the same problem? If not, can you try it in 0.13.x?
After the 0.14.10 release I can make a special exe that does verbose logging for map download/map upload logic.

[flonussi]

Hey,
I have this issue too...
We are playing on a map that by now uses around 35mb of space.
The host uses Linux and I (as a Client) am on a Windows machine.

I'll attach the log file, maybe it useful for you!

[Rippie]

Also tried to reproduce it. I noticed some slowdown when the map is reaching the end(which is normal) but nothing as extreme as your case.

The download speed decreases after reaching 100% download speed. But my guess is that the displayed download speed is an average over a certain time window, say the past 5 20 seconds. When the download is at 100%, the average will slowly decrease, until the average over 5 20 seconds is 0 KB/s.
Edit: The 0.14.10 changelog says it was 20 seconds, so my guess was right.

Rippie, did you try multiplayer prior to 0.14? Did you have the same problem?

Yes and yes. Back in 0.12.x I hosted a server, and friends could not connect. Same the other way around. FYI, my internet connection is 100MB/s up and down, so map downloading also went with several MB/s. Back then we blamed the peer-to-peer connection thing. After the 0.14.x multiplayer rewrite, I expected all would be fixed. Sadly, it is not.

After the 0.14.10 release I can make a special exe that does verbose logging for map download/map upload logic.

Sounds good, let me know. Nothing pleases me more than helping to improve this wonderful game.

[Rippie]

The MMO Test 2 with Steejo is currently going on.
Managed to join the first time, but got dropped from the game.
Can not reconnect because of this bug again.
See log file attached.

Edit: Later the server started dropping and all. This bug happened when the server was alive, as confirmed by Steejo's stream.

[nexyon]

We can confirm the same problem and same log output with 0.14.9 and 0.14.10. We have tried different setups:

• Linux server (headless and normal client) with Windows client
• Windows server with Linux client

[flonussi]

I just tested a few server - client configurations. And what I found out is:

Linux Client - Linux Server -> is working normally.
Mac Client - Linux Server -> is working normally.
Windows Client - Linux Server -> NOT WORKING.

Linux Client - Windows Server -> Sometimes not working

And it appears only on maps >10mb

[Twinsen]

The MMO Test 2 with Steejo is currently going on.
Managed to join the first time, but got dropped from the game.
Can not reconnect because of this bug again.
See log file attached.
Edit: Later the server started dropping and all. This bug happened when the server was alive, as confirmed by Steejo's stream.

During the event the server's internet connection was doing very well 99% of the time. All the big drops were caused by a bug that was causing the server to freze, which we just fixed for 0.14.11.
I'm even thinking your ISP is doing some silly stuff like throttling or even blocking these packets. Can I ask what ISP from where are you using?
Same question for anyone else in this topic with this problem.

[Rippie]

I'm even thinking your ISP is doing some silly stuff like throttling or even blocking these packets. Can I ask what ISP from where are you using?

I highly doubt it has anything to do with my ISP, Edutel in the Netherlands. It's a fibre wire connection.
I have never had any issues with my ISP.

[flonussi]

Personally, I don't think that my ISP is blocking something. If I use my Macbook, I can connect to our server without an issue. If I use my Notebook with Linux on it, I can connect to our server, but if I use my Windows machine, it gets stuck at 100%, 100% of the time.

And a few of my friends experience the same. Anyone with a linux machine doesn't have this issue. Only the windows users.

EDIT: The 14.11 Patch didn't change anything, the bug is still there

But I would be happy to help in resolving this issue! It makes the game unplayable in multiplayer for me

[Riverchimp]

Can I ask what ISP from where are you using?

Vodafone New Zealand is my ISP.