Factorio Multiplayer Protocol

[BBBBSammich]

TL/DR
I want to know a few details about the multiplayer joining protocol

WHY
I want to make a proxy that automatically launches a VM and gives interactive feedback to the player. I have done this before for Minecraft, it worked, and my peers have requested that I do it for Factorio.

MORE
I would like to make a proxy for Factorio which serves to redirect joining players to a server which is not necessarily running. It’s expensive for me to keep a factorion server running 24/7 (when paid by the hour) and it makes financial sense to only run the server when someone is playing. I have done this for Minecraft, and it worked great. I believe Minecraft’s multiplayer protocol is reverse engineered by the community https://minecraft.wiki/w/Java_Edition_protocol. But I do not have the capacity to do so for Factorio, some attempts were made in the past, but not to a significant degree. It would be great if some insight could be given to the workings of the factorion multiplayer protocol, (specifically the server joining part). I know that maintaining documentation about the full protocol is impossible because of the way the game works.

[Rseding91]

The Factorio server auto paused by default when no one is online, so in that sense it will use virtually no CPU. Is that not the goal?

[Tertius]

Is it really necessary to write a whole proxy service?
What do you need to start on demand, is it just a service (the Factorio server process) or do you need to start up a whole virtual machine?

With systemd, it's possible to listen on a port (in this case the Factorio udp port) with a *.socket unit and start a *.service unit that's supposed to serve this port. So the service isn't running as default, and if systemd detects activity on the port, it starts the associated *.service unit.

This *.service doesn't need to be the factorio server service, it can be any *.sh script. I guess you're able to start your vm from within, and if it is up, add an iptables rule to forward the local udp port to the vm udp port. This is actually your proxy: a corresponding iptables NAT rule for udp traffic on the given port. Don't need to look inside the packets.

[BBBBSammich]

Yes, it does have to do with starting a virtual machine. In my country at least, it is a lot cheaper to run a virtual machine on demand rather than using a hosting service or other solutions. I have so far written a discord bot where players can send a command to start the vm, but it is more convenient for players to start the vm by trying to join the server. I am a uni student hosting a game for my class, and it worked great with minecraft. Setting up the proxy was actually easier than the discord bot, as the edge cases were actually a lot fewer. I just had to look at the active TCP connections to see if anyone was online for instance. I know it's a bit different for UDP but I believe the principles stay the same.

I'd only have to detect when a player joins, maybe send some sort of message and kick them off. The vm is then started and any other connection attempts will be deferred until the server has successfully started. Then any further connection attempts will route all further messages directly between server and client. So I just need to know how to accept a connection and then immediately kick the player. A ping would also be useful to check if the server is online.

[BBBBSammich]

Is it really necessary to write a whole proxy service?
What do you need to start on demand, is it just a service (the Factorio server process) or do you need to start up a whole virtual machine?

With systemd, it's possible to listen on a port (in this case the Factorio udp port) with a *.socket unit and start a *.service unit that's supposed to serve this port. So the service isn't running as default, and if systemd detects activity on the port, it starts the associated *.service unit.

This *.service doesn't need to be the factorio server service, it can be any *.sh script. I guess you're able to start your vm from within, and if it is up, add an iptables rule to forward the local udp port to the vm udp port. This is actually your proxy: a corresponding iptables NAT rule for udp traffic on the given port. Don't need to look inside the packets.

This could work, I'd just have to set up a subprocess wrapper that reads the console logs, and sends an api call to stop the vm when all players leave. Or I can maybe check if the network activity ceases from the "proxy" machine, which would also be a failsafe if the server fails to start. It would just miss the interactive element that gives feedback to the player whether the server is responding and actually starting or not.

So if it is possible to do what my previous comment suggests with the existing factorio protocol, then that would still work better.

[eugenekay]

Just because you CAN do something does not mean that you SHOULD, or that it is easy.

Factorio uses a custom UDP based protocol for multiplayer communication, which will be very difficult to setup an intelligent “parser” to handle the connection attempts and kick-off the orchestration described. It will then take at least a few seconds - possibly a minute or more - to start the Game’s Virtual Machine Environment, load and process the Game and Data, prepare the Savegame for loading, and only THEN will it be possible to attempt to process the Client’s login packet…. By which time the Client has already given-up and shown an Error to the User for Server Timeout.

In comparison, the existing default behavior of the headless server is to auto-Pause the game once all Clients have disconnected, and it has authentication built in to the network protocol so that garbage packets are ignored. Why reinvent a more complicated, less functional Wheel? You cannot "Save" much by stopping the Process by running, since the Proxy & Orchestrator would still need to run on the same Machine / IP Address. The Server does use some Memory (which can be Paged to disk by the OS if needed) and very little CPU while Paused.

[BBBBSammich]

Just because you CAN do something does not mean that you SHOULD, or that it is easy.

Factorio uses a custom UDP based protocol for multilayer communication, which will be very difficult to setup an intelligent “parser” to handle the connection attempts and kick-off the orchestration described. It will then take at least a few seconds - possibly a minute or more - to start the Game’s Virtual Machine Environment, load and process the Game and Data, prepare the Savegame for loading, and only THEN will it be possible to attempt to process the Client’s login packet…. By which time the Client has already given-up and shown an Error to the User for Server Timeout.

In comparison, the existing default behavior of the headless server is to auto-Pause the game once all Clients have disconnected, and it has authentication built in to the network protocol so that garbage packets are ignored. Why reinvent a more complicated, less functional Wheel?

when there's money at stake, nothing is overkill. Also please read my comment describing why the auto-pause is not an option, I do not want to be redundant. But I'll say this again: the intention is to kick the player with a message when they join to tell them that the server is starting. I know this system works because I have used it before with over 50 users all having near to no issues. They have asked that I implement the same system on factorio because they enjoyed not having to open discord to play online. It saves costs by 50% compared to other game hosting services, and even more to renting a VM.

[eugenekay]

In comparison, the existing default behavior of the headless server is to auto-Pause the game once all Clients have disconnected, and it has authentication built in to the network protocol so that garbage packets are ignored. Why reinvent a more complicated, less functional Wheel?

when there's money at stake, nothing is overkill. Also please read my comment describing why the auto-pause is not an option, I do not want to be redundant. But I'll say this again: the intention is to kick the player with a message when they join to tell them that the server is starting. I know this system works because I have used it before with over 50 users all having near to no issues. They have asked that I implement the same system on factorio because they enjoyed not having to open discord to play online. It saves costs by 50% compared to other game hosting services, and even more to renting a VM.

You have missed my point slightly: there are EASIER ways to achieve your goal than parsing an entire network protocol. RCON already exists to provide a communication channel with the Factorio Server itself.

If you are already planning to run two instances of Factorio side-by-side (aka a "Waiting Room" with a restricted map to save resources, and a "Main Game" which has your MegaBase), you don't need a Proxy at all. Use a firewall-level Port Redirect (eg, IPTables) rule to send traffic to your desired Instance.

If you want to Pause the Factorio Instance to save CPU/RAM you don't need a Virtual Machine - Linux can do Process Backgrounding by itself - see the man page for "fg".

[BBBBSammich]

In comparison, the existing default behavior of the headless server is to auto-Pause the game once all Clients have disconnected, and it has authentication built in to the network protocol so that garbage packets are ignored. Why reinvent a more complicated, less functional Wheel?

when there's money at stake, nothing is overkill. Also please read my comment describing why the auto-pause is not an option, I do not want to be redundant. But I'll say this again: the intention is to kick the player with a message when they join to tell them that the server is starting. I know this system works because I have used it before with over 50 users all having near to no issues. They have asked that I implement the same system on factorio because they enjoyed not having to open discord to play online. It saves costs by 50% compared to other game hosting services, and even more to renting a VM.

You have missed my point slightly: there are EASIER ways to achieve your goal than parsing an entire network protocol. RCON already exists to provide a communication channel with the Factorio Server itself.

If you are already planning to run two instances of Factorio side-by-side (aka a "Waiting Room" with a restricted map to save resources, and a "Main Game" which has your MegaBase), you don't need a Proxy at all. Use a firewall-level Port Redirect (eg, IPTables) rule to send traffic to your desired Instance.

If you want to Pause the Factorio Instance to save CPU/RAM you don't need a Virtual Machine - Linux can do Process Backgrounding by itself - see the man page for "fg".

I suppose so. With my discord bot I am currently controlling the server over RCON, I did not think to set up a waiting room server, but that could work. So that is a solution. I have just always felt that getting access to Factorio on a slightly more fundamental level could streamline it a bit.

I thought that if there was a simple way for the devs to release the required information I would happily slave away at the hard parts. For some reason I decided to code this bot in Java instead of C#, and I also decided to make use of docker, except the existing wrappers for the docker API are shit, so I wrote my own wrapper (excuse the language). But the point is that, I don't really care to dig deep into documentation etc. The proposed solution would work, especially if I write a custom scenario that would be able to communicate with my wrapper through rcon. So this is perhaps my final call to something that might be impossible but I would still maybe like some consideration.

Thank you all for the help I do really appreciate the new perspective gained. (Sorry for the verbose language, perhaps you can see part of my Factorio style )

[eugenekay]

I thought that if there was a simple way for the devs to release the required information I would happily slave away at the hard parts.

They already have - you have the Game Binaries - and can Capture the Network Packets - it's just a lot of work to figure out what is going on, and create a packet-compatible Proxy/Fake Server for Clients to connect to..... if you already have a RCON-connected bot it is a lot less code to listen on the Waiting Room instance for New Player connections --> fire up the MegaBase Instance / Wait for RCON to come up --> Print a Chat Message to the Waiting Room --> Shut Down Waiting Room & Swap IPtables rules.

except the existing wrappers for the docker API are shit, so I wrote my own wrapper (excuse the language).

No Disagreement here.