[0.12.26] [cube] Headless dedicated server shuts itself down

[Umcookies]

Hello there!
I appreciate you taking your time to read this.

To keep it short and sweet I've setup a headless server on a Vultr VPS (kvm, 756mb ram, 15.10Ubuntu X64) and much to my surprise it started immediately, nothing more to download or install, just the factorio files and map file and the server was working perfectly. Or at least appeared to be.

After XX minutes the server simply shuts itself down, no crashing (as far as I can tell) no intervention from me. It just kills itself and I have to restart it manually. Which isn't terrible, the server starts incredibly quickly and we can rejoin immediately however it's pain to have to baby sit the server. I would love to be able to just leave it going overnight.


As usual, the log file http://pastebin.com/0p4jS1c2
(Ignore the first ~4 seconds of the log, that's me restarting the server quickly while we were waiting for the server to time out)

and the map file down below.

[kovarex]

The problem is, that the server is bombarded by random packets from at least 2 sources that eventually crash it.

Cube is working on some improvements that could solve it. (So it starts ignore IP addresses that sent more invalid packets).

[Umcookies]

The problem is, that the server is bombarded by random packets from at least 2 sources that eventually crash it.

Cube is working on some improvements that could solve it. (So it starts ignore IP addresses that sent more invalid packets).

Fantastic, glad to know that it's a known issue. Also explains why the time it takes to shutdown is inconsistent.

Is there anything we could do to the machine in the meantime to avoid this? IPTables or firewalls for instance?

[kovarex]

Code: Select all

 0.676 Info Router.cpp:368: Received packet type(Heartbeat) heartbeat(networkTick(108613) closedTick(false) tickClosures((tick(1240449) inputActions() ) (tick(1240450) inputActions() ) (tick(1240451) inputActions((SelectedEntityChanged) ) ) ) ) requestsForHeartbeats((requestPeerID(0) requestNetworkTick(108610) ) (requestPeerID(2) requestNetworkTick(108607) ) (requestPeerID(2) requestNetworkTick(108608) ) (requestPeerID(2) requestNetworkTick(108609) ) (requestPeerID(2) requestNetworkTick(108610) ) )  with invalid sessionMagic(259) from address(42.241.5.236:34197) our sessionMagic(931)
   0.676 Info Router.cpp:368: Received packet type(Heartbeat) heartbeat(networkTick(108614) closedTick(false) tickClosures((tick(1240451) inputActions() ) (tick(1240452) inputActions() ) ) ) requestsForHeartbeats((requestPeerID(0) requestNetworkTick(108610) ) (requestPeerID(0) requestNetworkTick(108611) ) (requestPeerID(1) requestNetworkTick(108608) ) (requestPeerID(1) requestNetworkTick(108609) ) (requestPeerID(1) requestNetworkTick(108610) ) (requestPeerID(1) requestNetworkTick(108611) ) )  with invalid sessionMagic(259) from address(58.170.216.248:34197) our sessionMagic(931)

From this (repeated) part of the log, it is apperent, that these two IP. addresses were doing it:
42.241.5.236
58.170.216.248
So blocking these two, or other addresses that will be mentioned in the log with the wrong sessionMagic will probably solve it.

[daniel34]

Code: Select all

 0.676 Info Router.cpp:368: Received packet type(Heartbeat) heartbeat(networkTick(108613) closedTick(false) tickClosures((tick(1240449) inputActions() ) (tick(1240450) inputActions() ) (tick(1240451) inputActions((SelectedEntityChanged) ) ) ) ) requestsForHeartbeats((requestPeerID(0) requestNetworkTick(108610) ) (requestPeerID(2) requestNetworkTick(108607) ) (requestPeerID(2) requestNetworkTick(108608) ) (requestPeerID(2) requestNetworkTick(108609) ) (requestPeerID(2) requestNetworkTick(108610) ) )  with invalid sessionMagic(259) from address(42.241.5.236:34197) our sessionMagic(931)
   0.676 Info Router.cpp:368: Received packet type(Heartbeat) heartbeat(networkTick(108614) closedTick(false) tickClosures((tick(1240451) inputActions() ) (tick(1240452) inputActions() ) ) ) requestsForHeartbeats((requestPeerID(0) requestNetworkTick(108610) ) (requestPeerID(0) requestNetworkTick(108611) ) (requestPeerID(1) requestNetworkTick(108608) ) (requestPeerID(1) requestNetworkTick(108609) ) (requestPeerID(1) requestNetworkTick(108610) ) (requestPeerID(1) requestNetworkTick(108611) ) )  with invalid sessionMagic(259) from address(58.170.216.248:34197) our sessionMagic(931)

From this (repeated) part of the log, it is apperent, that these two IP. addresses were doing it:
42.241.5.236
58.170.216.248
So blocking these two, or other addresses that will be mentioned in the log with the wrong sessionMagic will probably solve it.

58.170.216.248 is Umcookies IP, so blocking it will prevent him from joining his own server, probably even lock him completely out of the server if SSH is affected too.

[Umcookies]

From this (repeated) part of the log, it is apperent, that these two IP. addresses were doing it:
42.241.5.236
58.170.216.248
So blocking these two, or other addresses that will be mentioned in the log with the wrong sessionMagic will probably solve it.

Ah my apologies, as I pointed out in the original post the first ~4 seconds of the log are due to me restarting the server while the two clients that were originally connected were still attempting to reestablish a connection with the server that had just stopped itself.

Basically:
Server running for 20+ minutes and stops running >
Client's lose connection to server and the progress bar appears waiting for the connection to reestablish >
While those progress bar's are still there I alt tab out to the SSH window and start the server again >
That's what causes those first ~4 seconds of spam in the log.

I need to stress that the server ran completely fine for almost an hour as you can see at the end of the log file despite that 'connection spam' at the beginning of the log. I personally don't believe that is what is causing the server to stop itself.

[cube]

So your server is not bombarded by anyone, we just read the log wrong :-)

[Umcookies]

So your server is not bombarded by anyone, we just read the log wrong

My mistake really, I've got a couple sets of logs and i chose to upload the messiest one for whatever reason.

Now the fact my server isnt being bomb barded is both reassuring and worrying because now i have utterly no idea why the server keeps killing itself.

I need to head to bed now (factorio is making it real hard to wake up in the mornings ) but if theres anything i can send to help diagnose this issue I'm more than willing to send it. Just let me know how i can help.

[cube]

The only real problem I see in the log file is the Network unreachable error 10 lines from the end. Basically sending packet failed and Factorio just gave up and turned off. Did you have any network connectivity problems or changes in network configuration while playing?

[Umcookies]

I'll setup some monitoring and report back.

Would it be connection issues from a cilent or for the server causing this?

[cube]

I don't think so, these would just cause lost packets and we can handle those. Your problem looks like a network interface on the VM suddenly going missing. I found this blog post that talks about the same error message caused by bad routing table, but the fact that you can restart the server afterwards confuses me.

[Umcookies]

Just popping back in to let you know I haven't abandoned this yet, key word being yet.

So far I've discovered that it ~may~ be an OS issue, I spun up a new VPS and tried using CentOS 7. So far it's gone 3 days strong and is still running as I type this.

[cube]

Any news on this?