[15.35] issues with RCON over TCP

[derdude]

Hey developers

I was programming some RCON client to connect to a factorio server's remote console.
The program didn't write RCON-packets at one go but rather piecewise to the TCP buffer. Subsequently the server writes something along 'Couldn't parse RCON data: Couldn't read from input file. File could be corrupted.' to the log, even though my programm hasn't disconnected.
It probably expect the TCP buffer to contain at least the amount of data it wants to read, which isn't guaranteed by TCP. The TCP stack may arbitrarily transfer data from one peer to another. Thus, server should perform reads until the requested amount of data has been received.

I temporarily fixed the problem by writing packets at once.

[JohnyDL]

isn't this a bug not an idea/suggestion? you might want those log files available to the devs too

[Rseding91]

I'm not sure what you mean by "Thus, server should perform reads until the requested amount of data has been received." - the server requests nothing from a RCON connection - it simply listens to what gets sent to it. If you send garbage data it's going to get garbage results?

[derdude]

isn't this a bug not an idea/suggestion?

My bad, I somehow overlooked the bug-subforum.

you might want those log files available to the devs too

The log file merely contains "Couldn't parse RCON data: Couldn't read from input file. File could be corrupted."
Edit: and something along "client from address ... connected." preceding the above. I don't have the log files anymore because I don't own the server, but I'll try to recreate them.

the server requests nothing from a RCON connection - it simply listens to what gets sent to it. If you send garbage data it's going to get garbage results?

Program writes complete packet into the TCP stream -> Server accepts packet as intended.
Program writes exactly the same packet into the TCP stream in pieces -> Server emits above message.

Both methods are and should be equivalent as TCP is not packet oriented. It may scatter that one write into the stream into multiple pieces on it's own, so the server can't fully read what's been sent in one go. In this case, I "manually" (and not intended, which gave me some great debug session) scattered one chunk of data.

My guess is that the server performs a single read operation on each request of data. When that read returns less bytes than requested, the server stops processing and complains. This is incorrect behaviour, as it should perform reads and accumulate data until the requested amount of bytes has been received.

[JohnyDL]

There are factorio server logs and more data in the log files than one line by uploading everything the devs can evaluate what lead up to the event and try to recreate it

In your case since you're deliberately doing something a typical a video of what you did how and why and what you think the results should be would be helpful too

[derdude]

This is the logging output when connecting and sending the RCON authentication-packet in pieces of two bytes at maximum:

Code: Select all

52.586 Info RemoteCommandProcessor.cpp:230: New RCON connection from <IPv4-Address>:49055
52.587 Error RemoteCommandProcessor.cpp:322: Couldn't parse RCON data: Couldn't read from input file. File could be corrupted.

As I said, sending the packet's bytes all at once doesn't give any problem and sending them with multiple sending calls should not too.

[JohnyDL]

put it this way I don't get what you're doing I don't understand it and I don't get what you're saying in a way that's interpretable to anything I know or understand within TCPIP or networking

for example to me the TCP level would assemble the message before forwarding it to the next level in the chain, but what you're saying seems to be you've arranged it so TCP is sending each packet up the chain regardless of if it's a complete message and it broke. Well of course it broke that's not how TCP works. If it was HTTP over TCP rather than RCON then the TCP layer doesn't pass individual packets to the browser, at the very least it partially collates them (like live video where whole files make no sense because the file is still in the process of being made) but most of the time the application layer of TCP gets given chunks of files in a way it can comprehend, though I'm sure 99% of HTTP is whole files, if there are lost or missing data it attempts recovery before the application layer is even alerted and if necessary repeat requests any missing data. So sending half a message to RCON sounds like nonsense.

So explain it, explain it slowly, explain it fully and explain it with all the relevant information and logs required to put it all together and produce duplicable results on other machines, algorithmically if necessary, batch or bash code, etc.. not vagities that really really don't translate into comprehensible computer science let alone useful information for tracking down the problem you think you're having.

[derdude]

Okay, let's explicate

I am programming with C++ using the WinSock API on Windows or POSIX sockets on Linux for communication over TCP. To use a convenient stream-interface, I decided to create a strembuffer which can be written to or read from. The streambuffer manages to, well, buffer outgoing data and then using WinSock's/POSIX' send() function it writes all data into the TCP stream. Latter happens when the buffer is full or the stream is flushed.

The RCON-client uses that streambuffer to read and write it's packets like the initial one for authentication. Since I hadn't set the buffer to be large enough, the call to send() happens every second byte. The first packet's size is 10 bytes plus length of password, so the buffer is flushed to send() 9 times in total before that packet is finally written to the TCP stream.

Note that I do not say that 9 TCP-packets are sent; TCP is stream-, not packet oriented. send() merely puts the data into an internal sending buffer. The TCP driver will then grab some data (not neccessarily all) and transfers it into the server's receiving buffer. The server can fetch data from that buffer using WinSock/POSIX recv().

regardless of if it's a complete message and it broke

The RCON packet is still transmitted completely, though that doesn't happen within one call to send(), but with 9 send()s. Thus it's probable that the server's receiving buffer will fill up 'slowly' until everything (in this case, the authentication RCON-data) has been transmitted. Factorio's server will - that is my guess - subsequently notice that data is available and tries to read a full RCON packet.

As such packet hasn't been fully transferred into the receiving buffer yet, recv() will return less data than the server expected. This is is not a failure because the rest of the RCON packet will arrive soon. Factorio's server must read from the receiving buffer until a full RCON-packet is available.

One can test this circumstance with usual TCP-clients or /dev/tcp/<host>/<port> on Linux. First send a RCON packet in one go, then send it as multiple pieces.

[Oxyd]

The RCON-client uses that streambuffer to read and write it's packets like the initial one for authentication. Since I hadn't set the buffer to be large enough, the call to send() happens every second byte. The first packet's size is 10 bytes plus length of password, so the buffer is flushed to send() 9 times in total before that packet is finally written to the TCP stream.

That's not how send works. send sends a packet. You are sending 9 2-byte packets – none of which is a valid RCON message.

If you want to buffer things, you'll have to implement a buffer yourself, and then pass the buffer to send when you have a complete packet in it.

[derdude]

That's not how send works. You are sending 9 2-byte packets

TCP is stream oriented. https://linux.die.net/man/7/tcp

TCP does not preserve record boundaries.

Multiple calls to send() may be split or merged. https://stackoverflow.com/questions/174 ... dp-message

There is no preservation of message boundaries, TCP can bundle up data from many send() calls into one segment, or it could break down data from one send() call into many segments - but that's transparent to applications sitting on top of TCP, and recv() just gives you back data, with no relation to how many send() calls produced the data you get back.

A single RCON packet may be transferred using multiple calls to send().

[JohnyDL]

I'm pretty sure you're conflating things

TCP doesn't preserve boundaries between different messages but if it can split up a message it has to recombine them or if it combines them it has to be able to separate them that back or it's not transparent. Here's a demonstration, if you send the following strings to the TCP layer and the messages are going to the same address, I've denoted messages denoted by <> and you send the message to the TCP layer as send(<message>)

<ABCD><EFGH><IJKL>

Then the TCP does it's thing sending packets denoted by []

[<AB][C][D><EFGH><IJK][L>]

As you can see the packets strictly do not care about the boundaries of the messages and it cannot care about order either as it arrives as

[C][<AB][D><EFGH><IJK]

Now it can't send that up to to the application layer because that would be A nonsense and B not transparent so it needs to reassembles as

<ABCD><EFGH><IJK

It might even send the <ABCD> and <EFGH> up to the application layer in a different order but it would never send the "<IJK" as a message up to the application because that's not transparent, and so it sends a message asking for the lost packets with the <IJKL> message.

RCON will always make a single comprehensible message for itself to send to the TCP layer and then send(<full message>) to be fractionated or combined by the TCP layer, it doesn't make sense for it to deal with sending half a message itself and then the other half because it cannot reassemble them.

[derdude]

TCP is not message/packet/record oriented. Data can be written at one end into the stream and read from another, like pipes. See https://tools.ietf.org/html/rfc1180#page-24.

It might even send the <ABCD> and <EFGH> up to the application layer in a different order

That's not correct.

[JohnyDL]

<ABCD> and <EFGH> are effectively different streams and it doesn't matter what order they're discharged from the TCP layer to the application layer.

if I send(<Request for HTML document>) and send(<Request for image>) to the same server it doesn't matter if the server gets them in that order. The stream isn't "<Request for HTML document><Request for image>" but the TCP Layer can treat it as one message because it's going to the same place.

the TCP layer also doesn't strictly act like a pipe for streams of data either usually you'll send(<THIS ID, DATA>) and the stream would be assembled by the application layer and assume a sequential ID, if you have ID 3, 4, 5 and 7, for a video stream, then a good video player would be able to handle that frame jump, and if 8, 9, 10, 6, 11 comes in that same video player would be able to insert frame 6 in the right place.

Code: Select all

send(<if>)send(<you>)send(<sent>)send(<a>)send(<message>)send(<like>)send(<this>)send(<to>)send(<a>)send(<server>)send(<it>)send(<would>)send(<have>)send(<no>)send(<way>)send(<of>)send(<reassembling>)send(<it>)send(<coherently>)send(<as>)send(<messages>)send(<can>)send(<get>)send(<reordered>)send(<and>)send(<it>)send(<would>)send(<have>)send(<no>)send(<clue>)send(<as>)send(<to>)send(<how>)send(<they>)send(<are>)send(<ordered>)

You sent nine 2 byte packets in place of one 18 byte package by the sound of it and had no way of reassembling

[derdude]

<ABCD> and <EFGH> are effectively different streams and it doesn't matter what order they're discharged from the TCP layer to the application layer.

Your < and > don't make sense as TCP does not operate with packets (See above link to RFC). ABCD will always be sent and received before EFGH.

if I send(<Request for HTML document>) and send(<Request for image>) to the same server it doesn't matter if the server gets them in that order. The stream isn't "<Request for HTML document><Request for image>" but the TCP Layer can treat it as one message because it's going to the same place.

The server will get them in the same order. The stream contains "Request for HTML documentRequest for image". Note the absence of < and >. Yes, the TCP layer may put everything in one IP-packet or split the data into more than two, but that's transparent to the application layer.

the TCP layer also doesn't strictly act like a pipe for streams of data either usually you'll send(<THIS ID, DATA>) and the stream would be assembled by the application layer and assume a sequential ID, if you have ID 3, 4, 5 and 7, for a video stream, then a good video player would be able to handle that frame jump, and if 8, 9, 10, 6, 11 comes in that same video player would be able to insert frame 6 in the right place.

Like streams usually behave, the bytes retain their order with transmission.

Code: Select all

send(<if>)send(<you>)send(<sent>)send(<a>)send(<message>)send(<like>)send(<this>)send(<to>)send(<a>)send(<server>)send(<it>)send(<would>)send(<have>)send(<no>)send(<way>)send(<of>)send(<reassembling>)send(<it>)send(<coherently>)send(<as>)send(<messages>)send(<can>)send(<get>)send(<reordered>)send(<and>)send(<it>)send(<would>)send(<have>)send(<no>)send(<clue>)send(<as>)send(<to>)send(<how>)send(<they>)send(<are>)send(<ordered>)

Again: TCP is stream-oriented. No messages, for the application it's a stream of sequential bytes. Not packets. The TCP-stack may send "ifyousentamessagelikethistoaserveritwouldhavenowayofreassemblingitcoherentlyasmessagescangetreorderedanditwouldhavenoclueastohowtheyareordered" at once and the server can successfully read it at once. The server could also just read six characters ("ifyous") and then read the rest. However it can fail to do so, because just three characters have arrived yet. Then recv will only return "ify". In this case, the server must wait until more characters are available.

You sent nine 2 byte packets in place of one 18 byte package by the sound of it and had no way of reassembling

You might confuse TCP with UDP? With UDP a peer indeed transfers messages/packets/records, which may be reordered or get lost. With TCP a peer transfers bytes over a stream.

[JohnyDL]

Yeah I actually talked with someone who I trust knows a lot more than me about this after the fact, he said the same thing about TCP and UDP, so my brain is fried from not actually doing the study for 6 years and I need to re-read everything. I understood that TCP/IP was how the internet worked and the internet packets were mixed up, and the messages would be assembled and passed up discretely, not as a stream.

He also mentioned if you were sending text with C it's null terminated so the message may be full of /0s and that according to the spec splitting up app packets doesn't work. But I'm going to freely admit I'm out of my depth with C.

[derdude]

So we agree that the server has to read from the TCP stream until it can assemble a valid RCON-packet?

[JohnyDL]

maybe, but if you're not constructing valid app packets (and I don't know the ins and outs of what send() pads the app packets with) then it'll break, even if it 'should' come through as a stream.

[derdude]

The program sends valid RCON packets over the TCP stream.

[JohnyDL]

The program sends valid RCON packets over the TCP stream.

Only if you put valid data into it.

[derdude]

Well, that's what I said.