Map download never finishes [14.5] headless windows

Anything that prevents you from playing the game properly. Do you have issues playing for the game, downloading it or successfully running it on your computer? Let us know here.
Post Reply
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

dadymax wrote: Client cannot download map at all (hangs at 0 byte)
I believe you forgot to update the executable on the server. The new executable dumps all the network data in the log and it's not happening in the server logs.
I also double checked and it works locally for me.
Top
drunya13
Manual Inserter
Manual Inserter
Posts: 1
Joined: Fri Jan 13, 2017 10:13 am
Contact:
Contact drunya13

Re: Map download never finishes [14.5] headless windows

Post by drunya13 »

hello guys, have the same problem. Cant test with Factorio_0.14.21 Fix attempt+TransferTargetSource Logging 2.zip because have ubuntu headless dedicated server.
Maybe someone can help my method. As Twinsen said someone (maybe ISP or firewall) is blocking or modifying packets.
I have installed vpn pptpd server on my ubuntu server, my friend connect to factorio server through vpn and the problem disappeared
This partly confirms the words that said Twinsen.
good luck
Top
dadymax
Burner Inserter
Burner Inserter
Posts: 11
Joined: Thu Jan 05, 2017 12:28 pm
Contact:
Contact dadymax

Re: Map download never finishes [14.5] headless windows

Post by dadymax »

Twinsen wrote: I believe you forgot to update the executable on the server. The new executable dumps all the network data in the log and it's not happening in the server logs.
I also double checked and it works locally for me.
You're right. On server side admin forgot change executable. Will try later good combination.

By the time I try to plug my PC direct to ISP cable (w/o router) and try to connect with clean client to clean server and from 3 tries all three was successfull. May be lucky, may be there is a root of evil. Later I will try again and also try to get logs from spec-executable for better understanding - what goes on.

But. When test server was on "bad" executable I was connecting with standart client to standart server trought my router, get this bug (speed drop to 0 at the end of map download) and launch sniffer on UDP protocol on my PC. Some packets with sizes ~30-60 and ~500 was sniffed - this mean that packet get trought router but for some reason doesn't accepted by factorio?.. Uh... Remote debug with out debugger is a pain :)
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

dadymax wrote: But. When test server was on "bad" executable I was connecting with standart client to standart server trought my router, get this bug (speed drop to 0 at the end of map download) and launch sniffer on UDP protocol on my PC. Some packets with sizes ~30-60 and ~500 was sniffed - this mean that packet get trought router but for some reason doesn't accepted by factorio?.. Uh... Remote debug with out debugger is a pain :)
Interesting. Could be worth looking into, maybe you can show the packets(+ the detailed logs from the last version) that are sniffed after the map the map download stalls completely at 100%.
The game will continue to communicate the status and actions in the game as you are downloading. These are sent 30 times per second and are probably around 50 bytes like you saw.
Map packets have a size of 507 bytes and they are requested every half second or so in case of a stall. If you are receiving these packets in the sniffer then they don't reach the Factorio executable or are probably ignored because of tampering.

Maybe look in your router for "advanced routing" or any kind of "smart" port/traffic management, those maybe mess up the ports or addresses.
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

Any info on this? Logs with packet data would be very useful for us.
Top
Rippie
Burner Inserter
Burner Inserter
Posts: 14
Joined: Sun Sep 25, 2016 3:10 pm
Contact:
Contact Rippie

Re: Map download never finishes [14.5] headless windows

Post by Rippie »

I have been able to log three failed attempts of connecting to a friend using the executable found in "Factorio_0.14.21 Fix attempt+TransferTargetSource Logging 2.zip".
The logging entries are attached.

The normal executable downloads the map and then hangs on 100% map download.
The executable with "Fix attempt" does not download the map at all and stays idle on 0B and 0B/s transfer.
Sometimes, after 30 seconds or so, it manages to download approx 502 bytes.

Hope this helps.
Attachments
Factorio_logging.zip
(493.83 KiB) Downloaded 173 times
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

Rippie wrote:I have been able to log three failed attempts of connecting to a friend using the executable found in "Factorio_0.14.21 Fix attempt+TransferTargetSource Logging 2.zip".
The logging entries are attached.

The normal executable downloads the map and then hangs on 100% map download.
The executable with "Fix attempt" does not download the map at all and stays idle on 0B and 0B/s transfer.
Sometimes, after 30 seconds or so, it manages to download approx 502 bytes.

Hope this helps.
The fix changed the map transfer protocol so you need to replace both the client and the server executable with the new one for it to work.
Top
admalledd
Burner Inserter
Burner Inserter
Posts: 8
Joined: Thu Oct 06, 2016 4:05 am
Contact:
Contact admalledd

Re: Map download never finishes [14.5] headless windows

Post by admalledd »

Me and a friend are able to reproduce with me hosting from a linux box. Chances of a linux debug build? I only have access to linux machines, he has a mix of windows and linux.

Might be a day or three though before we have time at each of our respective houses to try again.

Would you also like a full wireshark capture from both ends? (and if so, any filtering we should do besides target/source IP to reduce file sizes?)

EDIT: PCAP BPF syntax seems to work fine and log everything of note with "(udp dst port 34197 or udp src port 34197)"

From looking at the PCAPs, the packets *are* making it to both sides... No idea of the factorio internals though (would need those linux debug builds for me to run)

Uploading now to my private server, let me know if you want the PCAPs and I can send a link via PM. They are a bit large.
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

admalledd wrote:Would you also like a full wireshark capture from both ends? (and if so, any filtering we should do besides target/source IP to reduce file sizes?)
Uploading now to my private server, let me know if you want the PCAPs and I can send a link via PM. They are a bit large.
The filter you mentioned seems ok.
Just the wireshark captures would be hard to interpret, but sent them anyway as they will surely come in handy later.
Ideally I would need the logs from the latest test executable and the wireshark capture. I don't mind the large file sizes.

Here is the test executable for linux(full game, not headless). Either use this executable on both client and server or you can use it with the windows executable from "Factorio_0.14.21 Fix attempt+TransferTargetSource Logging 2.zip" a few posts back.
Factorio_0.14.21 Fix attempt+TransferTargetSource Logging 2 linux.zip
(18.76 MiB) Downloaded 170 times
Thanks for the help.
Top
admalledd
Burner Inserter
Burner Inserter
Posts: 8
Joined: Thu Oct 06, 2016 4:05 am
Contact:
Contact admalledd

Re: Map download never finishes [14.5] headless windows

Post by admalledd »

Had to go to bed and let it upload over night, so writing this quick in the morning before coffee. Let me know if I confuse words and I can re-write this later to be more clear in the afternoon after work.

With the fix we were unable to reproduce the map download problems. Note that we were having UPS and dropping problems, but I can at least say that is likely related to the huge amounts of file IO going on, I know that while running this build and wireshark my disk IO was at 80-100% the full time. So that is progress!

Linky to all files: http://www.admalledd.com/dl/prv/files/f ... g_pcaps.7z

Important file stuffs:

We recorded some PCAPs with the original executables that night while we were waiting for a linux build. Those are under the "orig_executable" directory. The tests with the Fix+Detailed logs are in the other folder, those include PCAPs as well.

Of note from the original executable tests: we tested with a brand new vanilla map (no mods), forced the render out to get a map size of around 40MB. The connection problems seem to be probabilistic in nature and that larger map sizes make it more and more likely. The second thing about the test is that from a first glance packets *are* getting back and forth at the end just fine and seem to match up. This makes me wonder if factorio for some reason is dropping perfectly good packets? Or some such if I am understanding the theory on what might be going wrong here...

More wild mass guessing available later if desired, just wanted to get rough comments and those files to you before I had to leave for work where factorio is blocked :C
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

If the problem didn't occur then the logs won't be of much use.

But the wireshark logs ended up being very conclusive. Some specific packets are being filtered somewhere.
The packets you see being sent correctly are game packets. I first filter out game heartbeat packets (!data.data[0] == 06 and !data.data[0] == 07) and i'm left with mostly map download packets.
Packets start with this byte based on type:
ClientToServerHeartbeat = 6, //0x06
ServerToClientHeartbeat = 7, //0x07
TransferBlockRequest = 12, //0x0c
TransferBlock = 13, //0x0d

First test: (BattleshipBrotemkin windows client, admalledd linux host)
1. The client continuously requests Map package 12124(Data: 0c5c2f0000), marked in cyan on the screenshot and also continuously requests Map package 21623(Data: 0c77540000), marked in green on the screenshot.
2. The requests reach the server(green and cyan)
3. The server sends the correct responses (marked red and pink)
4. Those responses never reach the client(on client side all you see is sent requests for 15 seconds)

I included a screenshot and I also included the packets that are being filtered, so maybe you can ask your ISP a few questions of why they are filtering those specific packets.
Note that the you can see in the map near the scroll bar that one of the packets started getting filtered quite early.
Data

Second test: (BattleshipBrotemkin windows host, admalledd linux client)
1. The client sends the same request over and over for map packet 45003 (Data: 0ccbaf0000)
2. That specific request never reaches the server(all other requests work fine)
Data
It's interesting that such a small packet gets filtered.


We can't know witch one of you have a fishy ISP, router or firewall.
If you want to test further, I guess it's just trial and error: try to connect without the router, try to connect to different clients or try using a different computer.
The radom byte I added in the fix attempt will probably work, but now I'm really curious what is happening.
Can I ask what ISP both of you are using?
Many thanks for the help.
Top
admalledd
Burner Inserter
Burner Inserter
Posts: 8
Joined: Thu Oct 06, 2016 4:05 am
Contact:
Contact admalledd

Re: Map download never finishes [14.5] headless windows

Post by admalledd »

Hrm... We are both on Comcast, and in fact within the same routing loop here. The Full IP/layer 3 route between us is:

1. Admalledd-Desktop (no packet mangling firewall rules in place, linux 4.4)
2. Admalledd-Router (Archer C7v2 3.14.3 Build 150427 Rel.36706n)
3. Comcast-gateway "XE-9"
4. Comcast-gateway "TE-4"
5. Brotemkin-Router (TPLink TL-WDR3600 3.13.31)
6. Brotemkin-Desktop (Windows 10, firewall disabled during tests)


We both have SPI firewall disabled on our routers...

Tested to my server (admalledd.com) sending those lost packets in a manually crafted way, from both our networks, all four were received at the server.

Although while trying to use my own google-fu, I see your networkengineering.stackexchange post and bugger all else.

Nothing I can think of would be filtering or dropping those packets. Comcast at least in this area to my previous tests when I had access from college stuff didn't do any real packet shaping or filtering until outside the local loops. Been about three years now since those tests though.

With respect to FFF-136, there is no way to force a TCP mode? Or no simple build switch or such?

Extended testing with network topology changes is probably going to have to wait for monday though. As for calling Comcast and asking why they are blocking the packets: I can already sadly say I won't ever be able to get an answer from them on that front.
Attachments
udp_pythonTests.7z
In case I goofed on the dead simple test scripts, attached.
(1.13 KiB) Downloaded 141 times
Top
Twinsen
Factorio Staff
Factorio Staff
Posts: 1330
Joined: Tue Sep 23, 2014 7:10 am
Contact:
Contact Twinsen

Re: Map download never finishes [14.5] headless windows

Post by Twinsen »

Switching to TCP would require many code changes, NAT punching wont work and I see no reason for it.

I could be that some packets are blocked in the context of the other packets. Your guess is as good as mine. It's all up to you to debug it unfortunately. Let me know anything you find out, or if you need any help.

To help with debugging:
Map packets look like this:
Transfer Block Request, 5 byte message: 0c <block number as 4 byte little endian unsigned integer>
Transfer Block, 508 byte message: 0d <block number as 4 byte little endian unsigned integer> < 503 bytes of map data, padded with zeros if it's the last one>

In the "Fix attempt 2" executable they look like this:
Transfer Block Request, 6 byte message: 0c <random byte> <block number as 4 byte little endian unsigned integer>
Transfer Block, 508 byte message: 0d <random byte> <block number as 4 byte little endian unsigned integer> < 502 bytes of map data, padded with zeros if it's the last one>
Top
pieppiep
Fast Inserter
Fast Inserter
Posts: 170
Joined: Mon Mar 14, 2016 8:52 am
Contact:
Contact pieppiep

Re: Map download never finishes [14.5] headless windows

Post by pieppiep »

At work I'm a programmer for 'some kind of email-like system'.
We use soap to send messages and base64 encoding to send binaries via soap.
Some years ago one customer could send most messages, but some get strange errors.
It took some time, but we finally found out the ISP was a 'cristian-ISP' with a filter for all http with 'forbidden' word.
I opened the base64 encoded binary in a texteditor and searched for some words. They where there and so those attachments get blocked.
I took a look at the other attachments and found no naughty words in there.
In this case we asked the ISP to not filter the specific url for the soap server and it was fixed.

In this case for factorio it doesn't seem to be naughty words, but maybe there's a virusscanner or something like that with virus signatures.
The signatures can look like a few random bytes.

If the implementation is something like

send datapacket
wait some time
while not confirmationreceived
{
resend datapacket
wait some time
}

Maybe you can change it a little to

add dummy byte to datapacket
encrypt/hash/whatever the dummy byte to the datapacket
send datapacket
wait some time
while not confirmationreceived
{
change dummy byte
re-encrypt/hash/whatever the dummy byte to the datapacket
resend datapacket
wait some time
}
Top
Nezzam
Manual Inserter
Manual Inserter
Posts: 1
Joined: Fri Jan 27, 2017 6:48 pm
Contact:
Contact Nezzam

Re: Map download never finishes [14.5] headless windows

Post by Nezzam »

Another thing to check for is if the padding with 0's is what's tripping the packets to filter.

If that's the case, then maybe the last packet sent by the server instead of padding with 0s it's <map_data><end_of_map_data_key><random_data><end>.

If we replay the same packet through this environment, does it always fail? And does it fail in both directions?
Top
sillyfly
Smart Inserter
Smart Inserter
Posts: 1099
Joined: Sun May 04, 2014 11:29 am
Contact:
Contact sillyfly

Re: Map download never finishes [14.5] headless windows

Post by sillyfly »

Not sure it helps, but I have a guess as to the reason - it may have to do with UDP checksum and tunneling through IPv6. Basically, UDP checksum is optional in IPv4, but not in IPv6, and a wrong or missing (all zeros) UDP checksum will result in a packet being dropped.

What I suspect/guess might be happening is that all dropped packets have a calculated checksum of zero (which the UDP RFC says should be replaced with all ones, but I'm guessing is not in some implementations?), and when such packets tunnel through IPv6 they get dropped. There is no real way of checking if this is indeed the problem, as the clients are behind NAT and so the real source IP is not available (it is used to calculate the UDP checksum), but suffice to say the payloads of both outgoing messages have UDP checksum of 0x5DC5 (or possibly the bitwise negation of this value, i.e. 0xA23A), so they should also have the same total UDP checksum.

Sadly, I don't think there is a real way of actually verifying this theory, unless all router logs along the way are available. That the addition of one random byte solves this issue is consistent with this explanation, though, and should probably suffice to solve it for good (assuming this byte is different in consequent attempts, of course).

Edit: Actually, if someone encountering this problem is very resourceful and curious they may be able to further corroborate my guess, by sending additional messages with the same UDP checksum. If anyone previously affected by this wants to give this a go but needs some technical help - you are welcome to ask.
Top
hatterson
Manual Inserter
Manual Inserter
Posts: 2
Joined: Fri Apr 15, 2016 2:53 pm
Contact:
Contact hatterson

Re: Map download never finishes [14.5] headless windows

Post by hatterson »

sillyfly wrote:Not sure it helps, but I have a guess as to the reason - it may have to do with UDP checksum and tunneling through IPv6. Basically, UDP checksum is optional in IPv4, but not in IPv6, and a wrong or missing (all zeros) UDP checksum will result in a packet being dropped.

What I suspect/guess might be happening is that all dropped packets have a calculated checksum of zero (which the UDP RFC says should be replaced with all ones, but I'm guessing is not in some implementations?), and when such packets tunnel through IPv6 they get dropped. There is no real way of checking if this is indeed the problem, as the clients are behind NAT and so the real source IP is not available (it is used to calculate the UDP checksum), but suffice to say the payloads of both outgoing messages have UDP checksum of 0x5DC5 (or possibly the bitwise negation of this value, i.e. 0xA23A), so they should also have the same total UDP checksum.

Sadly, I don't think there is a real way of actually verifying this theory, unless all router logs along the way are available. That the addition of one random byte solves this issue is consistent with this explanation, though, and should probably suffice to solve it for good (assuming this byte is different in consequent attempts, of course).

Edit: Actually, if someone encountering this problem is very resourceful and curious they may be able to further corroborate my guess, by sending additional messages with the same UDP checksum. If anyone previously affected by this wants to give this a go but needs some technical help - you are welcome to ask.
If this is indeed the issue it would seem to imply that it lies with a router in between the two computers. Based on what admalledd said earlier there were only 4 hops in between which limits it to either one of their two routers the couple of local hops that comcast has it take.

The actual packet traces don't look like they're still available or I'd do this myself, but wireshark makes it pretty easy to filter packets based on checksum. Just right click on the checksum of the packet that was getting dropped and select Apply As Filter -> Selected. It'll give you a filter like udp.checksum == 0x155e. That will let you know if any other packets happened to share the same checksum by fluke. If there are any, and they got through, it seems to disprove the idea that it's a checksum. If there aren't any, they you're left with doing something like sillyfly suggested and generating some UDP packets with that checksum (but different data) to see if it's truly an inspection thing triggering it, or if somewhere along the lines the recalculations during routing end up with a 0 checksum and the specific router isn't properly flipping that.

If it is the checksum, it would explain why it doesn't show up in other areas as most are going to send dynamic data and 1 failing checksum is much lower than you'd expect overall packloss to be on the open internet anyway so it wouldn't trip any alarm bells.
Top
fregate84
Fast Inserter
Fast Inserter
Posts: 233
Joined: Sun Jun 22, 2014 10:56 am
Contact:
Contact fregate84

Re: Map download never finishes [14.5] headless windows

Post by fregate84 »

you can post your question on https://www.nanog.org/

It's a technical mailing list (for part you are interest for) of all ISP in the US.
It's can help (or not) but lots of Engineer ISP are on this mailing list (including me).
You should also share pcap.
Top
User avatar
hansinator
Fast Inserter
Fast Inserter
Posts: 160
Joined: Sat Sep 10, 2016 10:42 pm
Contact:
Contact hansinator

Re: Map download never finishes [14.5] headless windows

Post by hansinator »

Hi, I read about this in the FFF.

Given that the 5-byte packet that Twinsen posted on Stackexchange is being dropped I doubt that it is any intentional filtering. If it was, the structure that is filtered must be something like 0[C|D]xxxx0000. To me this looks too generic.
admalledd wrote:As for calling Comcast and asking why they are blocking the packets: I can already sadly say I won't ever be able to get an answer from them on that front.
I'd say contacting Comcast is the best option you have. Don't give up so fast. Trying out various things to nail down the problem is probably too time consuming.

@Twinsen:
Maybe you can contact them "officially" and ask them to have a look at the issue. My experience with contacting ISPs about probably network-related problems is limited to ISDN, but I can say that the companies we reached out to had a technician have a look at the issues and replied.

It would be interesting to know the provider and router brand+model of all who are affected by this. Maybe you could get some more insight if the debug log contained a traceroute to the server by default.
Top
Vxsote
Inserter
Inserter
Posts: 38
Joined: Sat Oct 01, 2016 12:51 am
Contact:
Contact Vxsote

Re: Map download never finishes [14.5] headless windows

Post by Vxsote »

I apologize in advance if this was explained but I missed it while reading through briefly, but... were the packet captures made on the hosts themselves, or from an intermediate point on the network? When I run into odd network behavior I often find it helpful to examine the traffic somewhere in the middle to isolate true network problems from problems with the hosts themselves.

If either of your routers (I'm not familiar with those models) has the ability to do a packet capture on the router itself, it can be a huge help. pfSense is great for this, if you happen to be familiar with it and have the ability to use at least temporarily. Otherwise, a switch that supports port mirroring (or an old hub) is also an option, although you still have potential OS issues with whatever box you plug in to do the capture. You mentioned possible future testing with network topology changes, so perhaps you already had this in mind. Also, for anyone who is interested, Netgear GS105E and GS108E switches are supposed to support port mirroring and are fairly inexpensive (and also support VLANs, which is handy if you want to use pfSense on a box with a single NIC, but that is getting a bit off topic).
Top
Post Reply

Return to “Technical Help”