[15.35] issues with RCON over TCP

[JohnyDL]

But is the data still a valid RCON packet if you break it up like that, do you accidentally include artefacts that don't appear in normal use for example Da/0ta/0li/0ke/th/0is rather than datalikethis or is the password included in each send or other issues not with how Factorio handles the data but with the initial formatting

[derdude]

No, there are no additional characters.

[JohnyDL]

Are you sure? Like I say I'm not familiar with C specifics but some languages terminate strings with a null character, some deal with it in other ways like adding integers at the beginning of the string to say how long it is.

Have you tried wiresharking the TCP packets and assembling the two messages side by side and looking for discrepancies you may have introduced? especially ones that wouldn't appear in a standard text editor.

Do you need to add something to your buffer that posts at the proper length to RCON because it just discards messages that're too short even if there's more to come?

I'm honestly just throwing ideas at the wall now hoping something sticks, I'm not sure I can help, I'm clearly out of my depth.

[derdude]

I printed the data in hex before send()ing them.
The data is definitely correct and the bug can always be reproduced by changing the buffer's size of my streambuf-object, in which the bytes are stored before send()ing them, to something very small. I'm pretty sure it is a bug in Factorio.
In case someone wishes, I can write some simple program for demontration.

[Rseding91]

You have to send the entire set of data at once - you can't call send() on half of the data and then send() the other half if you want it to work.

The underlying mechanics may fragment the data while sending and receiving but you still have to send it all in one go.

We have no plans to change this. If you can show me concrete wording in some standard that says it's acceptable to just not send all of the data you're meant to send then I might look into it but so far all I'm seeing is you're sending incomplete data and it fails to parse on the other end because you're not supposed to do that.

[sillyfly]

This sounds like a more general stream-handling philosophy - my understanding is that one shouldn't rely on streams alone for length of data (or completeness of data), regardless of the underlying mechanism. As I see it, the user (developer) may never know if more data will be available on the stream.

That being said, the TCP RFC says this:

The data that flows on a connection may be thought of as a stream of
octets. The sending user indicates in each SEND call whether the data
in that call (and any preceeding calls) should be immediately pushed
through to the receiving user by the setting of the PUSH flag.

A sending TCP is allowed to collect data from the sending user and to
send that data in segments at its own convenience, until the push
function is signaled, then it must send all unsent data. When a
receiving TCP sees the PUSH flag, it must not wait for more data from
the sending TCP before passing the data to the receiving process.

There is no necessary relationship between push functions and segment
boundaries. The data in any particular segment may be the result of a
single SEND call, in whole or part, or of multiple SEND calls.

The purpose of push function and the PUSH flag is to push data through
from the sending user to the receiving user. It does not provide a
record service.

There is a coupling between the push function and the use of buffers
of data that cross the TCP/user interface. Each time a PUSH flag is
associated with data placed into the receiving user's buffer, the
buffer is returned to the user for processing even if the buffer is
not filled. If data arrives that fills the user's buffer before a
PUSH is seen, the data is passed to the user in buffer size units.

(From https://www.ietf.org/rfc/rfc793.txt , chapter 2.8).

I understand it to mean the sender is given a promise that a PUSH flag will cause the immediate sending and subsequent notification of the receiver, but the receiver may see data even before a PUSH, if the local buffer is full or other higher-level mechanisms dictate so.

Edit: The RFC goes in a bit more detail on page 48, in the description of the Receive command:

If enough data arrive to fill the buffer before a PUSH is seen,
the PUSH flag will not be set in the response to the RECEIVE.
The buffer will be filled with as much data as it can hold. If
a PUSH is seen before the buffer is filled the buffer will be
returned partially filled and PUSH indicated.

This seems to explicitly say the receiver cannot assume complete data upon Receive, unless the PUSH flag is also set. So the question is, does the implementation of Send used implicitly set the PUSH flag with every call?

[derdude]

So the question is, does the implementation of Send used implicitly set the PUSH flag with every call?

Most implementations set the PUSH flag on each segment after coalescing data supplied by send(). However, this flag is not intended for marking the end of records, as stated within the RFC and applications should not rely on such implementation defined circumstances.

the user (developer) may never know if more data will be available on the stream.

That's why the server has to wait until either the connection closes or new data arrives. The number of bytes supplied to send() do not correlate to the number of bytes recv() returns when new data is available.

[derdude]

Any other thought?
I did not open this topic because I want my program to work, but because the part of the server which processes the RCON-data might be seriously flawed.

[malcom2073]

While I don't understand how Factorio is handling receiving data, TCP is indeed a stream protocol, not datagram based. That being said, Factorio seems to complain if you either send parts of the packet too far apart, or ignores packets completely if you send them too close together, as if it is expecting each read() to contain a whole, and only one whole packet.

Amusingly I just stumbled upon this thread topic in the opposite way: When sending multiple messages, if I don't flush the buffer between calls (to force a psh, causing the receiver to get the packet as is) then Factorio seems to ignores everything after the first packet at least until an amount of time had passed. I've had to basically wrap my write() calls in buffer flushes, to ensure that Factorio actually reads them properly.

I've found a workaround at least, I've implemented retries with a timeout to ensure that Factorio receives and parses all the messages sent.