Friday Facts #147 - Multiplayer rewrite

Regular reports on Factorio development.
iamwyza
Fast Inserter
Fast Inserter
Posts: 115
Joined: Tue Jun 07, 2016 2:59 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by iamwyza »

DaveMcW wrote:
bobingabout wrote:I seriously hope you manage to pull off some kind of re-write that would allow us to play together.
Server vs peer-to-peer is about 3+ players. If two people have a crappy connection, switching to server mode won't help.
Well, unless the server is located between them. Say setup in...India? Where your transit time to the server is 150ms. It may actually help then. Of course if either of the 2 clients is hosting the server locally, yes, it's not gonna help.

Harkonnen604
Filter Inserter
Filter Inserter
Posts: 285
Joined: Thu Jun 09, 2016 5:56 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Harkonnen604 »

What would help in huge lag situations is if game tracks two distinct game states - one real which encompasses all other players actions in sync (and may lag behind) and one virtual that simulates the game from the point of last synced exchange considering other players made no actions whatsoever, and this virtual state is actually shown on the screen. Each time new packet from other players is received, recent player actions are replayed from last real point to update that virtual non-lagging gamestate. This is like in shooters - movement and rotating head appears to be instant though server-side position is a few milliseconds behind. Since Factorio does not assume two players doing things together in the same place, that should work, but will require hard optimization of update cycle cpu cost like 10 times to handle those instant replays (or make it less precise, e.g. just move items on belts, but do not operate inserters, do not perform bots routing, etc... - these are ok to be jumpy from laggy network).

cswvna
Manual Inserter
Manual Inserter
Posts: 1
Joined: Sat Jul 16, 2016 2:10 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by cswvna »

One thing about the simulation speed of the slowest machine in multiplayer: the graphic displayed for resource density in each tile changes as the resource is depleted (at least while that tile is on screen), but is it necessary for the slowest machine to check for that depletion? Or is that a burden that has already been considered, and shifted to faster machines?

User avatar
DaveMcW
Smart Inserter
Smart Inserter
Posts: 3699
Joined: Tue May 13, 2014 11:06 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by DaveMcW »

Yes, every machine checks every resource in the game. Because it would be worse to send all that information over the network.

sabriath
Inserter
Inserter
Posts: 26
Joined: Fri Jan 10, 2014 6:49 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by sabriath »

I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data. In a true server model, the server would act as FULL authority, clients would tell the server any changes they do individually, but only required data is sent to the clients from the server (so the packet should be extremely brief). The other issue is filtering/ghosting, if you want to cut down on bandwidth, then you wouldn't be sending "1+2+3" to both clients, it is unnecessary data for "1" to receive its own transmission changes, and the same for "2," these cases may be trivial on the small packets you do send, but you will soon find this method will exhaust a server with more people and in a lagspike NACK denial.

I have written the best method to control your networks in a previous FF post, it was pushed aside because it didn't conform to your "visual synchronization" problem, but players will have nothing to visually see if they can't keep a connection alive. Good luck.

Shadowjonathan
Manual Inserter
Manual Inserter
Posts: 1
Joined: Sat Jul 16, 2016 8:17 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Shadowjonathan »

along with the subtitles for the trailer and other related media.
> Trailer

*coughs loudly*

Rseding91
Factorio Staff
Factorio Staff
Posts: 13171
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Rseding91 »

Harkonnen604 wrote:What would help in huge lag situations is if game tracks two distinct game states - one real which encompasses all other players actions in sync (and may lag behind) and one virtual that simulates the game from the point of last synced exchange considering other players made no actions whatsoever, and this virtual state is actually shown on the screen. Each time new packet from other players is received, recent player actions are replayed from last real point to update that virtual non-lagging gamestate. This is like in shooters - movement and rotating head appears to be instant though server-side position is a few milliseconds behind. Since Factorio does not assume two players doing things together in the same place, that should work, but will require hard optimization of update cycle cpu cost like 10 times to handle those instant replays (or make it less precise, e.g. just move items on belts, but do not operate inserters, do not perform bots routing, etc... - these are ok to be jumpy from laggy network).
That would never work... you want to run the game *twice* on every computer would just make someone with a slow computer lag behind *even more*.
If you want to get ahold of me I'm almost always on Discord.

Harkonnen604
Filter Inserter
Filter Inserter
Posts: 285
Joined: Thu Jun 09, 2016 5:56 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Harkonnen604 »

Rseding91 wrote:That would never work... you want to run the game *twice* on every computer would just make someone with a slow computer lag behind *even more*.
I mean situation when all players can handle 60+ UPS, but ping is just too high, e.g. two Dell Alienware playing Factorio between UK and Australia :) That local virtual branch does not have to simulate entire map, only chunk the player is in since it won't deviate by more than network delay (1 second at worst I think), so you can't run that far. Plus it will have to simulate all trains in case player goes to map. it's not twice but like +5% for big factory.

User avatar
vaderciya
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Nov 07, 2015 1:55 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by vaderciya »

Perhaps one day, we will have oil tankers...

Urth
Burner Inserter
Burner Inserter
Posts: 6
Joined: Sat Feb 27, 2016 11:56 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Urth »

The new network model is looking more like what I've read about the Q3 network code. If you have not read this and are interested in networking models for games this is a must read!
http://fabiensanglard.net/quake3/network.php

kovarex
Factorio Staff
Factorio Staff
Posts: 8078
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by kovarex »

sabriath wrote:I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data. In a true server model, the server would act as FULL authority, clients would tell the server any changes they do individually, but only required data is sent to the clients from the server (so the packet should be extremely brief). The other issue is filtering/ghosting, if you want to cut down on bandwidth, then you wouldn't be sending "1+2+3" to both clients, it is unnecessary data for "1" to receive its own transmission changes, and the same for "2," these cases may be trivial on the small packets you do send, but you will soon find this method will exhaust a server with more people and in a lagspike NACK denial
Sending 2 back to 2 is some extra data, but without that, I the server wouldn't be the only authority, so it could never drop packets. Also the data amount is really tiny. Few bytes per user action, and users do few actions per second most.

striepan
Inserter
Inserter
Posts: 23
Joined: Wed Jan 27, 2016 5:23 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by striepan »

I was thinking about a proposal to the way starting a game, connecting and saving works before this FFF. It does not seem to be intrusive to the new server ideas kovarex wrote about.

Proposal is to allow the players, if they coordinated, have a game without waiting for the map to be downloaded.

It goes like this: Players, before a game, meet in a lobby. When everybody joined in/after some deadline a map with known parameters is started. Each client generates game on its own - no need to send over a map. This could greatly speed things up in case of multiple players. After some time, on a chosen game tick everybody saves and goes to sleep or whatever. They could set a date and meet in a lobby again, to continue a game without any map transfers.

This can be a game changer for players with slow but reliable connection.

Now of course new/late players still need to download the map and annoy everybody with unanticipated pause. There could be a discipline option to disallow this and refuse connection from others mid-session. New players and people who missed the last gaming session could be allowed to download the map and join in the lobby at the beginning of a new session.

That would mean annoyance free multiplayer experience vs what we have today, showing greatest gains on big maps with multiple clients, where everybody has to re-download the map potentially over a slow connection. This also means the server does not need to have a connection with much upload bandwidth.

The admin could set the game session-end time or maybe it could be voted on. The voting system maybe to much or maybe fully polished multiplayer needs voting anyway for kicking people without admin present (or more advanced stuff).

huwp
Burner Inserter
Burner Inserter
Posts: 19
Joined: Mon Jun 13, 2016 10:01 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by huwp »

sabriath wrote:I can already foresee similar problems that you currently face with the new setup....you have solved nothing. You state prior that you took the inputs of the other clients through the server and routed them, but fail to see that's exactly what you are doing in the 1+2+3 model....you are still just rerouting data.
I believe that's intended to be the stepping stone to the full client server set up. If they were just to gut the existing code and start over it would an even bigger disaster (I've rewritten networking code professionally, it's a longwinded process)

Muchaszewski
Inserter
Inserter
Posts: 23
Joined: Wed Nov 13, 2013 11:17 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Muchaszewski »

Don't wait for upload

This feature could also be added. When someone joins the game, the server needs to save it, and others have to wait for it, this can't be removed, but once it starts uploading the game, other players (including the server) could just continue playing, while the server is providing the map. On top of that, the server would save all the actions made by the players in the meantime. Once the map is uploaded, the server would send these actions and the client would fast-forward to catch up. The only limitation is, that the client has to be able to run the map in faster speed, so he can catch up.
This is generaly a bad idea. This is Factorio and in Factorio you go BIG. Before patch I played on realy big map and only me and server could handle 60 (bearly) UPS, every other guys had at most 55 in worst case 30 or lower, so it would be not posibble to run it fast forward. Mutch better idea, but not very efficient (network wise) is to make this chunk based, where you send small area (like 8x8 chunks) in one packet at very begining. Next, every tick you would send a surrounding area that changed its state in every next tick around that "loaded" (player can handle simulation by himself) area, and everything that was already send would be considered "loaded" unless something in that chunk has beed changed, that client couldn't know about, eg. belt going through chunks, inserters putting items through border etc. This would require to send a lot of more data in longer period of time until player had downloaded whole map and can make his own simulation, but he could play along with other players, with no speeding up game thing of magic. Also if done right, You woudn't have to send whole chunk again while updating chunk state for client, only state of objects that client wouldn't know about in given tick again like only belts next to chunk border, or inserter placing an item over chunks.

Of course if client would have big latency, or packet drop he would have to recive every missed step, or even begin loading process from scratch. But in this scenario, there would be no saving for server, since he could always send current game state.
In game Lotny

EncryptingWolf
Burner Inserter
Burner Inserter
Posts: 5
Joined: Thu Jul 07, 2016 11:52 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by EncryptingWolf »

Don't wait for upload

This feature could also be added. When someone joins the game, the server needs to save it, and others have to wait for it, this can't be removed, but once it starts uploading the game, other players (including the server) could just continue playing, while the server is providing the map. On top of that, the server would save all the actions made by the players in the meantime. Once the map is uploaded, the server would send these actions and the client would fast-forward to catch up. The only limitation is, that the client has to be able to run the map in faster speed, so he can catch up.
Won't it be fast enough if you just don't render anything during the fast forward? Since I believe rendering and animation takes quite a big chunk out of the cpu. He doesn't have to see the fast forward.

Jürgen Erhard
Filter Inserter
Filter Inserter
Posts: 298
Joined: Sun Jun 12, 2016 11:29 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Jürgen Erhard »

From O(n^2) to O(n)

This made me fan myself. :D

Also: hardly any mention of disconnects. They do happen, because sadly, the Internet is not (and won't ever be) a 100% reliable connection medium. So people do drop without the mentioned "discussion" being possible. Dealing with that is rather hard.

And IPv4/v6 is just mentioned... but is, for me at least, a big thing. Nope, haven't yet done any multiplayer, and this is one reason I'm wary of doing it: I'm on a DS-Lite connection, so I have a publicly routable IPv6 address, but only a CGNAT IPv4, that is not reachable from outside. For example, I cannot do multiplayer in Warframe because they only do IPv4.

icedevml
Burner Inserter
Burner Inserter
Posts: 12
Joined: Sun Jul 03, 2016 1:13 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by icedevml »

By the way, guys. If you realized all the stuff about multiplayer and that P2P is kinda utopic idea, can't you just use TCP right now? The only guy to open ports will be server, which is obvious. I can't really get why you try so hard to reinvent the wheel by implementing TCP over UDP. Such attempts were made hundreds of times already and this is a mess. TCP is handled properly by system network stacks (rather than applications itself) and some more advanced network devices. I mean - nothing will be optimized better than TCP and your concept is 200% matching to the idea of TCP. You can not allow to lose any packet and you can't mess the ordering, so why not?

As a guy with dedicated server (and Factorio on it) right now I could see that very strange things happen from time to time, sometimes the server is just flooding clients with packets. Implementing own "reliable layer" is a titanic work which will not bring a big profit and probably you will be fixing bugs till the end of universe.

Panasonic1
Burner Inserter
Burner Inserter
Posts: 19
Joined: Fri May 27, 2016 6:56 pm
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by Panasonic1 »

kovarex,

Where is MERLOT? :mrgreen:

Did you received it? Don't torment us :D

ske
Filter Inserter
Filter Inserter
Posts: 411
Joined: Sat Oct 17, 2015 8:00 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by ske »

If you added a second (mirror) server following the main server and handle the saving of the game you could get rid of any interruptions when players join. Downside is you would have to run two server processes. I suppose once this works it could easily be made optional.

vtx
Fast Inserter
Fast Inserter
Posts: 150
Joined: Tue Jun 28, 2016 9:48 am
Contact:

Re: Friday Facts #147 - Multiplayer rewrite

Post by vtx »

icedevml wrote:By the way, guys. If you realized all the stuff about multiplayer and that P2P is kinda utopic idea, can't you just use TCP right now? The only guy to open ports will be server, which is obvious. I can't really get why you try so hard to reinvent the wheel by implementing TCP over UDP. Such attempts were made hundreds of times already and this is a mess. TCP is handled properly by system network stacks (rather than applications itself) and some more advanced network devices. I mean - nothing will be optimized better than TCP and your concept is 200% matching to the idea of TCP. You can not allow to lose any packet and you can't mess the ordering, so why not?

As a guy with dedicated server (and Factorio on it) right now I could see that very strange things happen from time to time, sometimes the server is just flooding clients with packets. Implementing own "reliable layer" is a titanic work which will not bring a big profit and probably you will be fixing bugs till the end of universe.
You are wrong UDP is the optimal choice for this game. I know it's seem unintuitive to choose UDP over TCP but it's not. To have a reliable / ordered connection you have to pay a price ( lower speed & lag spike). TCP and UDP both have packet lost & unordered packet the first one deal with it internally the other one don't bother at all.

-TCP hold hostage data when detect packet lost or packet arriving before it's timeline resulting a delay before the software are aware of data. Also to speed up TCP some will tempted to turn off Naggle but the benefit is marginal as you'll need to create your own buffer system for data as the ratio Sent:receive will no longer match 1:1.

-UDP on other hand keep the ratio Sent:receive as 1:1. As UDP don't bother at all about packet loss and unordered message. It's faster, because you can include in the datagram the #id of network with a sending buffer both side you can deal with packet lost and unordered packet easily by redundancy of data.
Packet lost : the game didn't care as eventualy a packet with all packet lost will be received.
Unordered packet : the game will drop any packet with lower id than the current network he are.

Rightly implemented you also remove completely the RTT ( round trip time ) of TCP for each packets it's take 200ms for a ping of 200ms.

. To simply send once & forget, with UDP you'll be able to remove the RTT as it's no longer required will result in each packet to be receive in 100ms with a ping of 200ms.

Post Reply

Return to “News”