Friday Facts #147 - Multiplayer rewrite

[5thHorseman]

But sometimes, you just have to learn from your own mistakes.

That describes pretty much everything I've ever done in this game

[Killavirus]

Thank you SO MUCH for not sticking your head in the sand and shrilling but it works it works !!!!

Can't wait for the new server model, Super Excited, Im happy to chuck a VPS on a 6 year old laptop cover it in tin foil and hide it in the shed 15M away that you can use as a "realistic" test bed

In the meantime, is there any this we can do to optimize our servers ?

[icedevml]

TCP hold hostage data when detect packet lost or packet arriving before it's timeline resulting a delay before the software are aware of data. Also to speed up TCP some will tempted to turn off Naggle but the benefit is marginal as you'll need to create your own buffer system for data as the ratio Sent:receive will no longer match 1:1.

This is a lockstep architecture. If any peer would skip some action or perform two actions in a different order then the determinism is lost and the game is desynchronized. There is no way to proceed if somebody haven't collected all information about next tick. Knowing data for further ticks means nothing, because the game is unable to process these data at the moment.

So what would you do in case when no information is received and the retransmission is not supported at all (but redundancy is)? Kick this peer because he lost internet connection for one second and then force him to reconnect and download the map once again?

And also, you don't have to care about RTT with a single, stream, continously-open connection. These ACK packets control the connection state but are not stalling further packets from going out in any way. Thanks to ACK you are provided with retransmission features out of the box and I don't think that anybody would be able to implement such features with his own custom protocol.

[huwp]

TCP is a protocol designed for file transfer and is poorly suited to games. I won't go into details - there are plenty of publications about it - use of TCP in a game project tends to go the same way as p2p does:

"TCP is so much easier and robust, we'll just use that."

Months of unexplained bugs and unstable latency later...

"Nope nope nope nope... we should have used UDP from the start!"

There are open source libs for UDP packet management if needed. Reinventing the wheel isn't required.

[vtx]

This is a lockstep architecture. If any peer would skip some action or perform two actions in a different order then the determinism is lost and the game is desynchronized. There is no way to proceed if somebody haven't collected all information about next tick. Knowing data for further ticks means nothing, because the game is unable to process these data at the moment.

So what would you do in case when no information is received and the retransmission is not supported at all (but redundancy is)? Kick this peer because he lost internet connection for one second and then force him to reconnect and download the map once again?

And also, you don't have to care about RTT with a single, stream, continously-open connection. These ACK packets control the connection state but are not stalling further packets from going out in any way. Thanks to ACK you are provided with retransmission features out of the box and I don't think that anybody would be able to implement such features with his own custom protocol.

About RTT over TCP for every packet you send or receive the protocol need to ACK everything, that's the RTT I talking about. With UDP you don't need to ACK at all, I'll explain later.

You have to see it as a whole. As they said the game right now is P2P everyone need to open a connection to everyone and send data to everyone and receive from everyone before proceeding game tick. We can say the game and network are tighly coupling. What they will do is to decoupling the network from the game.

By centralizing the authority of the game to one computer aka the server.
By removing the connections between clients and only having one connection per client to the server.
Every client will run the game in their own bubble of time with the server.

Server send to client1 :
network_id:10[game_event_merged]
Client 1 :
never receive on the game tick / client will simply proceed without anydata from server and send is game_event to the server with the last network_id received 9.
server send to client 1 :
network_id:10[game_event_merged]:network_id:11[game_event_merged]
Client 1 :
received network_id:10[game_event_merged]:network_id:11[game_event_merged] proceed to the game and send back is game_event with last network_id received 11.
...
then client 1 receive network_id:10[game_event_merged] he simply drop the packet as it's allready processed.
....
Server received from client 1 is last id and remove the data from the buffer next packet send will look like
network_id:12[game_event_merged]

That's an simplified example for dealing with packet lost and unordered packet. As you can see we don't need retransmission of the packet lost and we allways drop data received after their timeline. It's exactly because this game use determinism that it's possible to do that. Every client will only mirroring what happening on the server.

here some great documentation you should read if you want :
http://gafferongames.com/networking-for ... dp-vs-tcp/
http://gafferongames.com/networked-phys ... -lockstep/

[icedevml]

Server send to client1 :
network_id:10[game_event_merged]
Client 1 :
never receive on the game tick / client will simply proceed without anydata from server and send is game_event to the server with the last network_id received 9.
server send to client 1 :
network_id:10[game_event_merged]:network_id:11[game_event_merged]
Client 1 :
received network_id:10[game_event_merged]:network_id:11[game_event_merged] proceed to the game and send back is game_event with last network_id received 11.
...
then client 1 receive network_id:10[game_event_merged] he simply drop the packet as it's allready processed.
....
Server received from client 1 is last id and remove the data from the buffer next packet send will look like
network_id:12[game_event_merged]

But you know that ACK is just a field in the "flags" bitfield of TCP segment? So in properly configured TCP connection this:

send back is game_event with last network_id received 11

means exactly one packet. There is also an extension for TCP for not in-order data acknowledgement:

https://tools.ietf.org/html/rfc2883

some other cool options also persist:

https://tools.ietf.org/html/rfc7323

I can't really belive that some custom protocol will be faster than well-tuned TCP.

here some great documentation you should read if you want :
http://gafferongames.com/networking-for ... dp-vs-tcp/

I know this article. I would like to bring up one quote:

For realtime game data like player input and state, only the most recent data is relevant, but for other types of data, say perhaps a sequence of commands sent from one machine to another, reliability and ordering can be very important.

The temptation then is to use UDP for player input and state, and TCP for the reliable ordered data.

From this quote it follows that neither TCP nor UDP is recommended by the author for the case of Factorio.

TCP is a protocol designed for file transfer and is poorly suited to games. I won't go into details - there are plenty of publications about it - use of TCP in a game project tends to go the same way as p2p does:

"TCP is so much easier and robust, we'll just use that."

Months of unexplained bugs and unstable latency later...

"Nope nope nope nope... we should have used UDP from the start!"

There are open source libs for UDP packet management if needed. Reinventing the wheel isn't required.

I will not even answer to this, because this post doesn't carry any argumentation.

[vtx]

I know this article. I would like to bring up one quote:

For realtime game data like player input and state, only the most recent data is relevant, but for other types of data, say perhaps a sequence of commands sent from one machine to another, reliability and ordering can be very important.

The temptation then is to use UDP for player input and state, and TCP for the reliable ordered data.

From this quote it follows that neither TCP nor UDP is recommended by the author for the case of Factorio.

Misunderstanding happens when you take a portion of text out of context. That portion of text are talking about using both TCP and UDP for a game as he explain was a very bad thing to do.

Using UDP for realtime game data like player input and state.
Using TCP for the reliable ordered data.

It's better to take one protocol over the other depending on what you need to most and how fast you need the data to travel for subsecond data it's better to take UDP as this game send data at 16ms interval.

[striepan]

As I understand it TCP is bad for factorio *because* of re-transmissions. In case of packet loss the server could be like: "Well bad luck for you, no player action in this tick. Send me this again and i can get you in some later tick". So the affected user might be a little unhappy, but every one else didn't notice (in case of re-transmission, if it takes too long, we get stutter) . Is this even possible with TCP where data should be ordered? Even if you managed it, it would be ugly and inefficient implementation of... a clean, custom UDP protocol.

[icedevml]

As I understand it TCP is bad for factorio *because* of re-transmissions. In case of packet loss the server could be like: "Well bad luck for you, no player action in this tick. Send me this again and i can get you in some later tick". So the affected user might be a little unhappy, but every one else didn't notice (in case of re-transmission, if it takes too long, we get stutter) . Is this even possible with TCP where data should be ordered? Even if you managed it, it would be ugly and inefficient implementation of... a clean, custom UDP protocol.

But it would mean rolling the game back for the slow peer (reverting his action), which is not possible with this engine as far as I know.

a clean, custom UDP protocol

You've already seen how this "clean, custom UDP protocol" works in reality, it was not even possible to perform map downloads with attainable speed because of mistakes. Such mistakes will obviously happen if you seriously don't have years of experience in networking and protocols.

[DaveMcW]

rolling the game back for the slow peer (reverting his action), which is not possible with this engine as far as I know.

It's not reverted, it's just applied really slowly (slow enough for a round trip to the server).

There are some lag-hiding layers like player position that can be reverted if required.

[striepan]

But it would mean rolling the game back for the slow peer (reverting his action), which is not possible with this engine as far as I know.

Slow peer? If he's too slow he can't play! If you mean the guy whose packet was lost/ didn't arrive in time I believe it works differently. Client's run simulation in lock-step, so they can't apply their local actions to it without everybody's knowledge(and approval). They apply some local actions immediately to latency hiding layer for player convenience and if there's conflict they can revert that. The clients must agree on actions to be performed. With the new multiplayer proposal it would be easier - the server tells everybody. So maybe he can tell everybody that you were just standing there as the biters attacked...

You've already seen how this "clean, custom UDP protocol" works in reality, it was not even possible to perform map downloads with attainable speed because of mistakes. Such mistakes will obviously happen if you seriously don't have years of experience in networking and protocols.

Map transfers are different. There, it really makes no sense to reinvent TCP badly. This should be straightforward. Or... i could see some bittorrent library for servers with slow upload, so every client works on map transfer to be faster.

[Chanz]

This might be interesting:
https://developer.valvesoftware.com/wik ... prediction
https://developer.valvesoftware.com/wiki/Interpolation

UDP will always be the holy grail in real time communications. Its used in Voip, Streams, First-person-shooter, etc. and should be used by Factorio. Sure there can be many errors, but there are concepts to migrate for mistakes in development.
I gave you the links to the valve developer wiki because its a great start to see how AAA game devs deal with it.

Basically how they solved the issues is by taking snapshots of the world state and transmit it to clients (similar to the third picture in the FFF#147). World Snapshots can be sent as a whole or in delta parts. Once a client gets out of sync (bad connection, mistakes in the code, ...) the client receives a new whole snapshot and is able to play again without being disconnected.

The only bad thing about world snapshots is the additional latency. Maybe in Factorio the gameplay will always be "slow" enough to hide the latency. But if that is not the case interpolation will be the keyword.

Best regards,
Chanz

PS: Thanks for this great game, you guys do a really great job!

[markheloking]

The only bad thing about world snapshots is the additional latency.

Not only that. The whole world snapshot would be the whole save. How are you going to send the whole world save (which can get over 20MB even on normal size bases, let alone mega bases) within fractions of a second (fast enough so you don't have to wait seconds or minutes every time a packet fails)? Having gigabit connections all around. Basically the outcome would be pretty much exactly how it's done now, as one also gets the world state (the whole save) sent again if they desync.

I don't think you quite get the concept of determinism of this game and how that affects network traffic, and especially why that means the difference between TCP and UDP. Packet loss is not acceptable for this application, THAT'S why UDP isn't and can't be used (without making it into some franken-build tcp-lookalike on the application layer).

UDP is used for all the applications you mentioned because on those applications only latency matters. A reasonable amount of packet loss can easily be accepted in those applications as that packet loss is not application-stalling or application-breaking.

[Fahrradkette]

UDP will always be the holy grail in real time communications. Its used in Voip, Streams, First-person-shooter, etc. and should be used by Factorio. Sure there can be many errors, but there are concepts to migrate for mistakes in development.

VoIP, Streams and FPS are very good examples for using UDP because if there's some packet loss it doesn't matter. On VoIP/Streams one hears/sees some stuttering while in a FPS, players simply walk into walls when they're having packet loss.

The point is, as long as one doesn't have to resend it, UDP is fine. As soon as one has to transfer a state, TCP is the protocol since _The Internet_ is aware of it.
I read that when a TCP packet is lost at an intermediate node (like somewhere in between my ISP and the server's ISP) my ISP (or his backbone provider) will resend the packet without any action from my computer or the game server.

What about using UDP where it doesn't matter when packets are lost(like player movement, hand-crafting) and using TCP for stuff where the gamestate matters, like inserting items in to a chest, placing entities, setting combinators or pasting blueprints?

I often had UDP congestion on my outgoing connection due to the need of resending lost UDP packets, it even made that sh*tty router reboot
Now I don't try to play multiplayer anymore, I don't want to annoy people. I might try out an UDP over TCP tunnel so the game doesn't have to be smart. But to do so I'd have to talk to someone who's hosting since he has to provide the other end. What do you guys think of this, is it worth a try?

Regards Kette

[icedevml]

The point is, as long as one doesn't have to resend it, UDP is fine. As soon as one has to transfer a state, TCP is the protocol since _The Internet_ is aware of it.
I read that when a TCP packet is lost at an intermediate node (like somewhere in between my ISP and the server's ISP) my ISP (or his backbone provider) will resend the packet without any action from my computer or the game server.

Especially for that reason I said before that I don't belive that any fancy UDP protocol would be faster than well-tweaked TCP in this case. TCP is well-understood by all network devices, while custom UDP protocol is end-to-end only. It doesn't matter on LAN but in more complicated case of course it *does*.

Basically some propositions of custom protocol were related to reinventing TCP with another terminology and in a little obfuscated way.

I often had UDP congestion on my outgoing connection due to the need of resending lost UDP packets, it even made that sh*tty router reboot

This is what happens right now with Factorio. My dedicated server running 0.13 already DDoSed few flats.

[Engimage]

To OP

I am really glad and exited to hear that you will improve the multiplayer. Sure thing p2p is not any good when you aim to play over internet. So client-server architecture is the only option here.

To network "masters"

If you are trying to suggest using UDP you probably never developed any client-server apps. While when applying UDP to VoIP or streams you can accept packet loss even if it is severe, in games like Factorio data loss is not acceptable at all so TCP is the only valid option.
UDP can be used in multiplayer games only in case that you do transmit the full game state in every data chunk so if you dropped a chunk due to packet loss you might catch up when you recieve next chunk. This might apply to shooters for example when the map is static and all data you transfer from server is updated positions of players/projectiles. In Factorio only player actions are transmitted so dropping an action will surely result in a different game state on two ends.

Now to my thoughts

1. Server of servers While the first step is implementing a game server itself, I would suggest to do it thinking about superstructure. You might want to have a master server that can launch instances of game servers on request so one server could possibly serve multiple games simultaneously. While initially MP server was designed to run on one of players' machines, the best experience can be achieved when running it on some datacenter like cloud VPS or whatever. And the player has to get the means to control that server so it launches the game on its side. And why do I say multiple games simultaneously? Because a single server could probably handle multiple games at once and at some point hosting a server may become an additional (paid) service. Or imagine several friends just rent a VPS to handle their games.

2. Multithreading. About handling of transmitting long data. I see the game is saving for pretty long time. I doubt this is because it has to write much data (I have 30 Mb save for example and it saves 3 seconds long for me). The data is structured, compressed etc. I am sure you could make use of multithreading to help with this. You can just make a memory data snapshot (takes mostly no time) and then launch a separate thread which will compress date etc so this process will have near to no effect on players. Anyways with so many parallel processes I can't get why did you not think about making use of multithreading. I laughed hard on some people with cool Core i7s who said game only uses 15% of CPU. In the modern world you should really make use of this cool processors. And for i7 users - turn god damned Hyperthreading off in your BIOS.

3. Map entry points. I do not know where does a player spawn when he enters MP game? Well a master of the game has probably his position saved in the game save file but what about others? They all can surely spawn in the same position but I got an idea. Maybe you could let people choose say a train station where to spawn? And maybe the game start will have a special item like "crashed spaceship" (according to game lore) which could serve likek this point...

Anyways. Thanks for the great game devs! I have watched quite a few streams/videos where players play togather and I am looking forward to do it myself when it is more convinient as it is REALLY FUN.

[Rseding91]

... 2. Multithreading. About handling of transmitting long data. I see the game is saving for pretty long time. I doubt this is because it has to write much data (I have 30 Mb save for example and it saves 3 seconds long for me). The data is structured, compressed etc. I am sure you could make use of multithreading to help with this. You can just make a memory data snapshot (takes mostly no time) and then launch a separate thread which will compress date etc so this process will have near to no effect on players. Anyways with so many parallel processes I can't get why did you not think about making use of multithreading. I laughed hard on some people with cool Core i7s who said game only uses 15% of CPU. In the modern world you should really make use of this cool processors. And for i7 users - turn god damned Hyperthreading off in your BIOS. ...

Factorio already threads the saving logic so it compresses and saves at the same time.

3 seconds for a 30 MB save file means it's waiting on your hardware. I'm able to save a 55 MB save file in 3.4 seconds on my computer.

[Fahrradkette]

My saves go way faster, because I'm loading the whole factorio folder into a ramdisk before starting it and save it back after it closes.
If I ever experience a power out, I'll tell myself: https://www.youtube.com/watch?v=xzpndHtdl9A

[steinio]

RAMDisk? It's so 90s. Never thought that it's still a think with SSD drives and so.

[Fahrradkette]

Oh, there is a ssd in my comp but it's occupied by whinedos for when I want to play some other games, it needs it

On my default OS, making a ramdisk is fairly easy, also factorio doesn't eat that much ram, so I have some to spare.

I love the fact that this great game runs native on a proper OS:)