[0.18.1] Invalid packet (type ConnectionRequestReply)

[Hornwitser]

The Big Community Games server broke and while trying to connect to it I noticed the following error in the client log:

Code: Select all

 211.926 Info UnparsedNetworkMessage.cpp:91: Invalid packet (type ConnectionRequestReply) received from IP ADDR:({116.202.35.156:34197}): invalid state (WaitingForAccept) for packet type

The admins posted a snippet from the server log in their Discord showing a spam of messages of the form

Code: Select all

18632.428 Warning TransmissionControlHelper.cpp:317: Message 1847 still not confirmed!

I made a packet capture of trying to connect to the server while it was broken. The server is very slow to respond to the handshake and the client sends multiple handshakes while waiting. When the server does respond it sends a response for each of the handshakes the client sent and starts multiple connection negotiations in parallel (see the attached screenshot), it appears that the negotiations fails due to the messages printed in the client log but the server still expects the client to acknowledge the messages it sent.

[Ext3h]

There was the suspicion that we brought the server into that state by use of excessively sized blueprints.

An issue I could observe as player was that using an upgrade planer to upgrade more than 256 entities (all of them same type) in a single drag&drop action resulted in instant disconnect for the performing player. Nothing in the log, except for a generic (WaitingForUserToSaveOrQuitAfterServerLeft), and the corresponding modal dialog.
Except the server didn't quit (yet). Not a desync either, but a hard connection drop.

As the issue occurred for the first time, one of the other players was experimenting with huge blueprints too. I don't have a record of which blueprint they used exactly, but by repeating that experiment the server could be brought in that state again immediately. In total we got 3 or 4 rounds of server reboot -> breaking it.

[Gerkiz]

Hello,

It seems that we have this issue as well. Although Hornwitser does not specify how large their .log file was - ours is at rougly 420MB.

Attached is the 7zip file that is ultra compressed with lzma2.

/Gerkiz

[Gerkiz]

Also .pcap-output from the weird connection.

Note, the player does not even connect, he just starts to send this weird randomness of data and it makes the server stutter like it has a lot to-do.

/Gerkiz

[Hornwitser]

This was determined to be a handshake flood attack at BCG, and Gerkiz looks to be under the same kind of attack. This is more or less a script kiddie level attack for which there is no way to defend against.

[Gerkiz]

I see that this was moved to "Technical Help" - may I ask why?

/Gerkiz

[Mactorio]

This doesn't seem like an issue with the game/server software. It's an issue with someone doing some kind of denial of service attack on the server. Doesn't seem to be much you can do until they get bored and knock it off (or you block every single IP they're using, which can be a monumental amount of work if it's a DDoS attack), but that doesn't necessarily mean you have to give them a target to hit. They already have the IP so there's not much you can do about that (unless you request a change from the provider), but you can change the server port, make it a private server and only give out the new port # to people you're pretty sure aren't responsible for the attack. After a few days the problem should go away on it's own.

[Gerkiz]

Thank you for your reply. This information is already something that I know.

Since the server is sending unlimited information/packets to the clients I am certain that the Factorio devs could do something to limit this. As of now the client initiates a handshake flood and the server responds unlimited to the client until the client is blocked either via blackhole or some other sort of deny.

This is a flaw since the attacker is sending a payload that the server recognizes and responds the client.

If there is nothing to be done, then what could the “hoster” possibly do? Give up and stop hosting?

As of now it seems that Big Community and Comfy has had these sort of attacks and it saddens me that it’s not prioritized.

/Gerkiz

[Gerkiz]

This is now the third day. To think that this is something that should be possible today is kind of disgusting.

A DDoS? This is something that I can understand that the Factorio devs can’t do anything about. But when it’s their software that is having this flaw? It feels like we’re prioritized to the ground since “no one else are having these issues”.

Eh don’t even know why I’m writing this since no one of the devs will reply.

Over and out
/Gerkiz

[Mactorio]

I don't think public multiplayer games are a priority in general. Factorio is a very popular game for it's type (last I heard there were maybe around 2 million copies sold?), but look at the multiplayer server list and there's maybe 100 people in public servers and another 50 in passworded games on a good day. (Looking at steam right now, 10000 people are "in game" but there's < 100 in public servers). Most of the game is designed around a single player experience, or a private multiplayer experience with friends, hence why there's not a ton of anti-grief tools that are well suited for public games.

I assumed you were being ddosed since attacks from a single source are usually easily blocked once you realize something's up. I guess you have someone that's persistent if they're exploiting the game client itself and is avoiding your blocks.

Ultimately making the game client/server have protection against potential attacks is a lot of work to serve < 1% of the players, and could potentially cause problems with denying legitimate connections, not to mention it's not necessarily effective. It's always going to cost the server more CPU cycles to analyze and process data than it costs the attacker to generate bogus packets and send them out. Send enough data to any program and it'll start stalling. It's one thing if someone was going down the list and hammering every single server, but this seems to be an isolated issue (which you unfortunately are the one to get).

It might be a lot of work to make the server private and give the port number to the regulars, but that's probably the best solution if someone's been targeting your server for multiple days.