Friday Facts #147 - Multiplayer rewrite

[vtx]

To network "masters"

If you are trying to suggest using UDP you probably never developed any client-server apps. While when applying UDP to VoIP or streams you can accept packet loss even if it is severe, in games like Factorio data loss is not acceptable at all so TCP is the only valid option.
UDP can be used in multiplayer games only in case that you do transmit the full game state in every data chunk so if you dropped a chunk due to packet loss you might catch up when you recieve next chunk. This might apply to shooters for example when the map is static and all data you transfer from server is updated positions of players/projectiles. In Factorio only player actions are transmitted so dropping an action will surely result in a different game state on two ends.

The game allready use UDP only for multiplayer. They only remove the interconnection between player and placing the authority ( validation of action ) to only one place ( server ) instead of having everyone to validate every action.

In the end the choice of protocol depend on what you really need and how you are able to hide the latency.
TCP : you need 100% reliability of data but didn't care that much of the speed. ( ex : card game )
UDP : you need speed but didn't care that much about reliability. ( ex : realtime game )
You can't have both.

With the next factorio multiplayer the server will be the only one who have the "real" state of the game. So every player will adjust their game state to the server so you'll not have different game state on two ends.

[Engimage]

To network "masters"

If you are trying to suggest using UDP you probably never developed any client-server apps. While when applying UDP to VoIP or streams you can accept packet loss even if it is severe, in games like Factorio data loss is not acceptable at all so TCP is the only valid option.
UDP can be used in multiplayer games only in case that you do transmit the full game state in every data chunk so if you dropped a chunk due to packet loss you might catch up when you recieve next chunk. This might apply to shooters for example when the map is static and all data you transfer from server is updated positions of players/projectiles. In Factorio only player actions are transmitted so dropping an action will surely result in a different game state on two ends.

The game allready use UDP only for multiplayer. They only remove the interconnection between player and placing the authority ( validation of action ) to only one place ( server ) instead of having everyone to validate every action.

In the end the choice of protocol depend on what you really need and how you are able to hide the latency.
TCP : you need 100% reliability of data but didn't care that much of the speed. ( ex : card game )
UDP : you need speed but didn't care that much about reliability. ( ex : realtime game )
You can't have both.

With the next factorio multiplayer the server will be the only one who have the "real" state of the game. So every player will adjust their game state to the server so you'll not have different game state on two ends.

Factorio game state is not something you can sync with the server during game play cause the data size is too big. The main design idea in this case is having the identical game model on both server and client and only transmit actions + small data for object player is interracting with. In this scenario a missed action due to transmission failure can lead to snowballing desync of two models. For example a player turns a belt.

In this case you have to increase fault tolerance to a maximum + carefully select what parts of game state will need to be forcefully synced on a regular basis. In theory UDP might work here as it will not cause transmitting bottlenecks but it will majorly fail if:
1. Too much data needs to be synced on a regular basis
2. Player has connection issues.

So things that come into mind to sync:
- All train conditions through the whole map
- All entities player interracts with
- Map state sync

Anyways almost every player action in the model is vital so syncing actions is a complex thing to do. And one thing to consider - namely handling desyncs due to packet loss. Using pure UDP is possible and will smooth the game process but you should know exactly how you handle that desync that will be there all the time.

[Killavirus]

To OP
1. Server of servers While the first step is implementing a game server itself, I would suggest to do it thinking about superstructure. You might want to have a master server that can launch instances of game servers on request so one server could possibly serve multiple games simultaneously. While initially MP server was designed to run on one of players' machines, the best experience can be achieved when running it on some datacenter like cloud VPS or whatever. And the player has to get the means to control that server so it launches the game on its side. And why do I say multiple games simultaneously? Because a single server could probably handle multiple games at once and at some point hosting a server may become an additional (paid) service. Or imagine several friends just rent a VPS to handle their games.

2. Multithreading. About handling of transmitting long data. I see the game is saving for pretty long time. I doubt this is because it has to write much data (I have 30 Mb save for example and it saves 3 seconds long for me). The data is structured, compressed etc. I am sure you could make use of multithreading to help with this. You can just make a memory data snapshot (takes mostly no time) and then launch a separate thread which will compress date etc so this process will have near to no effect on players. Anyways with so many parallel processes I can't get why did you not think about making use of multithreading. I laughed hard on some people with cool Core i7s who said game only uses 15% of CPU. In the modern world you should really make use of this cool processors. And for i7 users - turn god damned Hyperthreading off in your BIOS.

3. Map entry points. I do not know where does a player spawn when he enters MP game? Well a master of the game has probably his position saved in the game save file but what about others? They all can surely spawn in the same position but I got an idea. Maybe you could let people choose say a train station where to spawn? And maybe the game start will have a special item like "crashed spaceship" (according to game lore) which could serve likek this point...

1. I would be very sad if all the servers were hosted by the developers. I enjoy having full control over the server, Although having a server manager would be pleasant so that we can manager our own instances.

2. Not sure why shadowcopy isn't utilised on windows systems, I am presuming its (no offense intended) laziness on the developers part due to having to recode map saving in multiple ways for different os systems. or for an unknown reason to me they are unable to access the API's

3. Would prefer the ability for admin to change the spawn location via command.

[jjcf89]

1. I would be very sad if all the servers were hosted by the developers. I enjoy having full control over the server, Although having a server manager would be pleasant so that we can manager our own instances.

You should check out https://github.com/MajorMJR/factorio-server-manager.

[Killavirus]

Many thanks !!!
well found

[striepan]

Factorio game state is not something you can sync with the server during game play cause the data size is too big. The main design idea in this case is having the identical game model on both server and client and only transmit actions + small data for object player is interracting with.

I'm with you

In this scenario a missed action due to transmission failure can lead to snowballing desync of two models.

Now it's just non sequitur.
The server is the authority. If the client didn't get the data from the server it can't compute the next tick.
And later you contradict yourself talking about syncing state that doesn't need to by synced. Only players actions must be synced.

So let me re-frame my pov on this. It's about isolating packet loss to the affected client only.

As I just said, in case of server -> client communication, the client has to stop to get the data from the server again, but with some spare cpu cycles it can catch up to the others. Other players should not be affected. It would be a reliable channel.
If you insist on reliable communication in case of client -> server communication then packet loss from a single player can affect everybody. If there is not enough time for a re-transmission, the server and everybody else has to wait for the one unlucky guy.
So let's make client -> server unreliable. Now, if you let the server continue the client has to retry it's actions at a later time. But in most cases it's just that the player's avatar stays in the same spot one tick longer. Who cares? Other interactions are already handled by the latency hiding layer on the user's side, so even the affected player might not notice a thing.

So yes, reliable server -> client matches TCP 1:1. But it would be nice to have the client -> server packet loss-tolerant. We just move on from one lost UDP packet instead of waiting for TCP to re-transmit.
Of course, as I am not the developers, I i might be misunderstanding something and this would not work. There are many details that make multiplayer a complex problem.

[Hicsy]

Yo don't forget you can just treat multiplayer synchronisation like a data backup. What you are sending is lots of incremental backups, but don't be afraid to send the occasional snapshot/keyframe which has the full status of things (or a bunch of things). Ie. when a new user connects you can start buffering up the difference, then send them a keyframe (shit will teleport a lot, and little things that you wont bother blending might disappear or seem to combust randomly, but the joining player knows this is just the catchup process).
You should also be prioritising EVERYTHING. I know you are mostly just sending tiny inputs anyway, and *hoping* things stay in sync (they wont) but dont be afraid to go "yo every 10ticks all players will send me their map location and ill blend that with what I know and redistribute in that keyframe"... "oh, and fluids, they can full-update every 100ticks and then blend...".

same thing if someone's cpu starts hacking out, they can just start dropping everything that isn't a snapshot/keyframe, and then they can at least have a jumpy world rather than getting dropped straight away.

From some of your other posts about "hiding lag" I assume you have read more than most of us on multiplayer - but if you havent seen them: ID did a really good writeup on their Quake syncro, Valve on lag-correction/prediction, and M$ on "hiding".

Sorry if I linked the wrong articles to any of those just google them instead

[kovarex]

Yo don't forget you can just treat multiplayer synchronisation like a data backup. What you are sending is lots of incremental backups, but don't be afraid to send the occasional snapshot/keyframe which has the full status of things (or a bunch of things). Ie. when a new user connects you can start buffering up the difference, then send them a keyframe (shit will teleport a lot, and little things that you wont bother blending might disappear or seem to combust randomly, but the joining player knows this is just the catchup process).
You should also be prioritising EVERYTHING. I know you are mostly just sending tiny inputs anyway, and *hoping* things stay in sync (they wont) but dont be afraid to go "yo every 10ticks all players will send me their map location and ill blend that with what I know and redistribute in that keyframe"... "oh, and fluids, they can full-update every 100ticks and then blend...".

same thing if someone's cpu starts hacking out, they can just start dropping everything that isn't a snapshot/keyframe, and then they can at least have a jumpy world rather than getting dropped straight away.

From some of your other posts about "hiding lag" I assume you have read more than most of us on multiplayer - but if you havent seen them: ID did a really good writeup on their Quake syncro, Valve on lag-correction/prediction, and M$ on "hiding".

Sorry if I linked the wrong articles to any of those just google them instead

All the things you are talking about might be relevant if the game states would be kilobytes in size. But in Factorio, the game states can be 10-100 megabytes easily when compressed. You can't just sync that every 10 ticks or so. Syncing only things on the screen wouldn't work as well, as when you zoom out, you can still see tens of thousands of objects, updating these regularly wouldn't be possible.
In other way, the lock-step needs to be kept, no doubt about it.

[provet]

Subject: Factorio development explained in gifs

"Hey I have a great idea that will make the game better, you should add multiplayer"

(sorry couldn't resist)