Friday Facts #147 - Multiplayer rewrite

[iamwyza]

I seriously hope you manage to pull off some kind of re-write that would allow us to play together.

Server vs peer-to-peer is about 3+ players. If two people have a crappy connection, switching to server mode won't help.

Well, unless the server is located between them. Say setup in...India? Where your transit time to the server is 150ms. It may actually help then. Of course if either of the 2 clients is hosting the server locally, yes, it's not gonna help.

[Harkonnen604]

What would help in huge lag situations is if game tracks two distinct game states - one real which encompasses all other players actions in sync (and may lag behind) and one virtual that simulates the game from the point of last synced exchange considering other players made no actions whatsoever, and this virtual state is actually shown on the screen. Each time new packet from other players is received, recent player actions are replayed from last real point to update that virtual non-lagging gamestate. This is like in shooters - movement and rotating head appears to be instant though server-side position is a few milliseconds behind. Since Factorio does not assume two players doing things together in the same place, that should work, but will require hard optimization of update cycle cpu cost like 10 times to handle those instant replays (or make it less precise, e.g. just move items on belts, but do not operate inserters, do not perform bots routing, etc... - these are ok to be jumpy from laggy network).

[cswvna]

One thing about the simulation speed of the slowest machine in multiplayer: the graphic displayed for resource density in each tile changes as the resource is depleted (at least while that tile is on screen), but is it necessary for the slowest machine to check for that depletion? Or is that a burden that has already been considered, and shifted to faster machines?

[DaveMcW]

Yes, every machine checks every resource in the game. Because it would be worse to send all that information over the network.

[sabriath]

I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data. In a true server model, the server would act as FULL authority, clients would tell the server any changes they do individually, but only required data is sent to the clients from the server (so the packet should be extremely brief). The other issue is filtering/ghosting, if you want to cut down on bandwidth, then you wouldn't be sending "1+2+3" to both clients, it is unnecessary data for "1" to receive its own transmission changes, and the same for "2," these cases may be trivial on the small packets you do send, but you will soon find this method will exhaust a server with more people and in a lagspike NACK denial.

I have written the best method to control your networks in a previous FF post, it was pushed aside because it didn't conform to your "visual synchronization" problem, but players will have nothing to visually see if they can't keep a connection alive. Good luck.

[Shadowjonathan]

along with the subtitles for the trailer and other related media.

> Trailer

*coughs loudly*

[Rseding91]

What would help in huge lag situations is if game tracks two distinct game states - one real which encompasses all other players actions in sync (and may lag behind) and one virtual that simulates the game from the point of last synced exchange considering other players made no actions whatsoever, and this virtual state is actually shown on the screen. Each time new packet from other players is received, recent player actions are replayed from last real point to update that virtual non-lagging gamestate. This is like in shooters - movement and rotating head appears to be instant though server-side position is a few milliseconds behind. Since Factorio does not assume two players doing things together in the same place, that should work, but will require hard optimization of update cycle cpu cost like 10 times to handle those instant replays (or make it less precise, e.g. just move items on belts, but do not operate inserters, do not perform bots routing, etc... - these are ok to be jumpy from laggy network).

That would never work... you want to run the game *twice* on every computer would just make someone with a slow computer lag behind *even more*.

[Harkonnen604]

That would never work... you want to run the game *twice* on every computer would just make someone with a slow computer lag behind *even more*.

I mean situation when all players can handle 60+ UPS, but ping is just too high, e.g. two Dell Alienware playing Factorio between UK and Australia That local virtual branch does not have to simulate entire map, only chunk the player is in since it won't deviate by more than network delay (1 second at worst I think), so you can't run that far. Plus it will have to simulate all trains in case player goes to map. it's not twice but like +5% for big factory.

[vaderciya]

Perhaps one day, we will have oil tankers...

[Urth]

The new network model is looking more like what I've read about the Q3 network code. If you have not read this and are interested in networking models for games this is a must read!
http://fabiensanglard.net/quake3/network.php

[kovarex]

I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data. In a true server model, the server would act as FULL authority, clients would tell the server any changes they do individually, but only required data is sent to the clients from the server (so the packet should be extremely brief). The other issue is filtering/ghosting, if you want to cut down on bandwidth, then you wouldn't be sending "1+2+3" to both clients, it is unnecessary data for "1" to receive its own transmission changes, and the same for "2," these cases may be trivial on the small packets you do send, but you will soon find this method will exhaust a server with more people and in a lagspike NACK denial

Sending 2 back to 2 is some extra data, but without that, I the server wouldn't be the only authority, so it could never drop packets. Also the data amount is really tiny. Few bytes per user action, and users do few actions per second most.

[striepan]

I was thinking about a proposal to the way starting a game, connecting and saving works before this FFF. It does not seem to be intrusive to the new server ideas kovarex wrote about.

Proposal is to allow the players, if they coordinated, have a game without waiting for the map to be downloaded.

It goes like this: Players, before a game, meet in a lobby. When everybody joined in/after some deadline a map with known parameters is started. Each client generates game on its own - no need to send over a map. This could greatly speed things up in case of multiple players. After some time, on a chosen game tick everybody saves and goes to sleep or whatever. They could set a date and meet in a lobby again, to continue a game without any map transfers.

This can be a game changer for players with slow but reliable connection.

Now of course new/late players still need to download the map and annoy everybody with unanticipated pause. There could be a discipline option to disallow this and refuse connection from others mid-session. New players and people who missed the last gaming session could be allowed to download the map and join in the lobby at the beginning of a new session.

That would mean annoyance free multiplayer experience vs what we have today, showing greatest gains on big maps with multiple clients, where everybody has to re-download the map potentially over a slow connection. This also means the server does not need to have a connection with much upload bandwidth.

The admin could set the game session-end time or maybe it could be voted on. The voting system maybe to much or maybe fully polished multiplayer needs voting anyway for kicking people without admin present (or more advanced stuff).

[huwp]

I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data.

I believe that's intended to be the stepping stone to the full client server set up. If they were just to gut the existing code and start over it would an even bigger disaster (I've rewritten networking code professionally, it's a longwinded process)

[Muchaszewski]

Don't wait for upload

This feature could also be added. When someone joins the game, the server needs to save it, and others have to wait for it, this can't be removed, but once it starts uploading the game, other players (including the server) could just continue playing, while the server is providing the map. On top of that, the server would save all the actions made by the players in the meantime. Once the map is uploaded, the server would send these actions and the client would fast-forward to catch up. The only limitation is, that the client has to be able to run the map in faster speed, so he can catch up.

This is generaly a bad idea. This is Factorio and in Factorio you go BIG. Before patch I played on realy big map and only me and server could handle 60 (bearly) UPS, every other guys had at most 55 in worst case 30 or lower, so it would be not posibble to run it fast forward. Mutch better idea, but not very efficient (network wise) is to make this chunk based, where you send small area (like 8x8 chunks) in one packet at very begining. Next, every tick you would send a surrounding area that changed its state in every next tick around that "loaded" (player can handle simulation by himself) area, and everything that was already send would be considered "loaded" unless something in that chunk has beed changed, that client couldn't know about, eg. belt going through chunks, inserters putting items through border etc. This would require to send a lot of more data in longer period of time until player had downloaded whole map and can make his own simulation, but he could play along with other players, with no speeding up game thing of magic. Also if done right, You woudn't have to send whole chunk again while updating chunk state for client, only state of objects that client wouldn't know about in given tick again like only belts next to chunk border, or inserter placing an item over chunks.

Of course if client would have big latency, or packet drop he would have to recive every missed step, or even begin loading process from scratch. But in this scenario, there would be no saving for server, since he could always send current game state.

[EncryptingWolf]

Don't wait for upload

This feature could also be added. When someone joins the game, the server needs to save it, and others have to wait for it, this can't be removed, but once it starts uploading the game, other players (including the server) could just continue playing, while the server is providing the map. On top of that, the server would save all the actions made by the players in the meantime. Once the map is uploaded, the server would send these actions and the client would fast-forward to catch up. The only limitation is, that the client has to be able to run the map in faster speed, so he can catch up.

Won't it be fast enough if you just don't render anything during the fast forward? Since I believe rendering and animation takes quite a big chunk out of the cpu. He doesn't have to see the fast forward.

[Jürgen Erhard]

From O(n^2) to O(n)

This made me fan myself.

Also: hardly any mention of disconnects. They do happen, because sadly, the Internet is not (and won't ever be) a 100% reliable connection medium. So people do drop without the mentioned "discussion" being possible. Dealing with that is rather hard.

And IPv4/v6 is just mentioned... but is, for me at least, a big thing. Nope, haven't yet done any multiplayer, and this is one reason I'm wary of doing it: I'm on a DS-Lite connection, so I have a publicly routable IPv6 address, but only a CGNAT IPv4, that is not reachable from outside. For example, I cannot do multiplayer in Warframe because they only do IPv4.

[icedevml]

By the way, guys. If you realized all the stuff about multiplayer and that P2P is kinda utopic idea, can't you just use TCP right now? The only guy to open ports will be server, which is obvious. I can't really get why you try so hard to reinvent the wheel by implementing TCP over UDP. Such attempts were made hundreds of times already and this is a mess. TCP is handled properly by system network stacks (rather than applications itself) and some more advanced network devices. I mean - nothing will be optimized better than TCP and your concept is 200% matching to the idea of TCP. You can not allow to lose any packet and you can't mess the ordering, so why not?

As a guy with dedicated server (and Factorio on it) right now I could see that very strange things happen from time to time, sometimes the server is just flooding clients with packets. Implementing own "reliable layer" is a titanic work which will not bring a big profit and probably you will be fixing bugs till the end of universe.

[Panasonic1]

kovarex,

Where is MERLOT?

Did you received it? Don't torment us

[ske]

If you added a second (mirror) server following the main server and handle the saving of the game you could get rid of any interruptions when players join. Downside is you would have to run two server processes. I suppose once this works it could easily be made optional.

[vtx]

By the way, guys. If you realized all the stuff about multiplayer and that P2P is kinda utopic idea, can't you just use TCP right now? The only guy to open ports will be server, which is obvious. I can't really get why you try so hard to reinvent the wheel by implementing TCP over UDP. Such attempts were made hundreds of times already and this is a mess. TCP is handled properly by system network stacks (rather than applications itself) and some more advanced network devices. I mean - nothing will be optimized better than TCP and your concept is 200% matching to the idea of TCP. You can not allow to lose any packet and you can't mess the ordering, so why not?

As a guy with dedicated server (and Factorio on it) right now I could see that very strange things happen from time to time, sometimes the server is just flooding clients with packets. Implementing own "reliable layer" is a titanic work which will not bring a big profit and probably you will be fixing bugs till the end of universe.

You are wrong UDP is the optimal choice for this game. I know it's seem unintuitive to choose UDP over TCP but it's not. To have a reliable / ordered connection you have to pay a price ( lower speed & lag spike). TCP and UDP both have packet lost & unordered packet the first one deal with it internally the other one don't bother at all.

-TCP hold hostage data when detect packet lost or packet arriving before it's timeline resulting a delay before the software are aware of data. Also to speed up TCP some will tempted to turn off Naggle but the benefit is marginal as you'll need to create your own buffer system for data as the ratio Sent:receive will no longer match 1:1.

-UDP on other hand keep the ratio Sent:receive as 1:1. As UDP don't bother at all about packet loss and unordered message. It's faster, because you can include in the datagram the #id of network with a sending buffer both side you can deal with packet lost and unordered packet easily by redundancy of data.
Packet lost : the game didn't care as eventualy a packet with all packet lost will be received.
Unordered packet : the game will drop any packet with lower id than the current network he are.

Rightly implemented you also remove completely the RTT ( round trip time ) of TCP for each packets it's take 200ms for a ping of 200ms.

. To simply send once & forget, with UDP you'll be able to remove the RTT as it's no longer required will result in each packet to be receive in 100ms with a ping of 200ms.