Friday Facts #63 - The endless struggle

Regular reports on Factorio development.
User avatar
Drury
Filter Inserter
Filter Inserter
Posts: 782
Joined: Tue Mar 25, 2014 8:01 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by Drury »

DonRobo wrote:Reading the two articles you linked about how those two games (and Factorio) do multiplayer made me thing. Is there any reason you didn't go for a client+server model? As a developer who has never worked on a (proper or multiplayer) game before that sounds much more stable and straight forward, but I'm probably missing some obvious terrible problems. It would be nice reading about that decision in some future (or past?) Friday Facts post.
Peer-to-peer is better for mass data handling since you can use full power of all peers to handle the automated parts and then just send player interactions. Client-server solution would be too bandwidth-intensive and put too much strain on the server.

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 12888
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by ssilk »

I would like to have birthday-present-assembly in factorio. then I would need only to send the save for a present to a factorio owner.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

JackGruff
Fast Inserter
Fast Inserter
Posts: 134
Joined: Tue Oct 15, 2013 4:44 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by JackGruff »

I'm getting Mad Max vibes from that car.

Alfdaur
Long Handed Inserter
Long Handed Inserter
Posts: 72
Joined: Sat Mar 08, 2014 10:41 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by Alfdaur »

Nice ride! And happy birthday!

Factorio is a realy awesome game. So you did accomplisch something before your 30th birthday!

ysor123
Inserter
Inserter
Posts: 23
Joined: Mon Jun 02, 2014 8:31 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by ysor123 »

Is there a way to make a tree, having a CRC for each node. Changes on child nodes should result in CRC change of parent nodes. So you quick check from tree root and track down what should be resynced. One host machine should be the dominant and dictate state. Everything that should be synced should extend or implement a sync node. I do not know how your code is organized though. I can't tell if there is any smart way not to blow it up computing to many CRC values.

SnowDapples
Manual Inserter
Manual Inserter
Posts: 1
Joined: Tue Dec 09, 2014 5:59 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by SnowDapples »

Speaking of finding desyncs in maps, do you guys use foreign maps for those tests as well?
I have one that always desyncs when the second player joins, maybe it could prove useful.

kovarex
Factorio Staff
Factorio Staff
Posts: 8078
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by kovarex »

Such maps are now very helpful.

User avatar
cpy
Filter Inserter
Filter Inserter
Posts: 839
Joined: Thu Jul 31, 2014 5:34 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by cpy »

Drury wrote:
DonRobo wrote:Reading the two articles you linked about how those two games (and Factorio) do multiplayer made me thing. Is there any reason you didn't go for a client+server model? As a developer who has never worked on a (proper or multiplayer) game before that sounds much more stable and straight forward, but I'm probably missing some obvious terrible problems. It would be nice reading about that decision in some future (or past?) Friday Facts post.
Peer-to-peer is better for mass data handling since you can use full power of all peers to handle the automated parts and then just send player interactions. Client-server solution would be too bandwidth-intensive and put too much strain on the server.
That is most idiotic reason ever. I have 100/100mbit internet and i can't play with all players because one guy have slow upload and killing it for everyone, instead of server handling all the traffic we have to suffer everyone's connection speed. I host games for a reason so people with slower internet are in client mode and do not require public ip!

Also not limiting network speed by game itself would also help when downloading/uploading map. I feel like there should be option for P2P/client-server hosting. Without VPN/ UDP hole punching there won't be many internet games going on because almost nobody plays on LAN these days.

I just hope that this LAN multiplayer is P2P and promised internet high ping MP will be client-server style. Early alphas are pain sometimes. I guess it was easier to do than client-server? Factorio devs said it's lan only, so if anyone have 4-6 players on lan, does it run good?

kovarex
Factorio Staff
Factorio Staff
Posts: 8078
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by kovarex »

cpy wrote:
Drury wrote:
DonRobo wrote:Reading the two articles you linked about how those two games (and Factorio) do multiplayer made me thing. Is there any reason you didn't go for a client+server model? As a developer who has never worked on a (proper or multiplayer) game before that sounds much more stable and straight forward, but I'm probably missing some obvious terrible problems. It would be nice reading about that decision in some future (or past?) Friday Facts post.
Peer-to-peer is better for mass data handling since you can use full power of all peers to handle the automated parts and then just send player interactions. Client-server solution would be too bandwidth-intensive and put too much strain on the server.
That is most idiotic reason ever. I have 100/100mbit internet and i can't play with all players because one guy have slow upload and killing it for everyone, instead of server handling all the traffic we have to suffer everyone's connection speed. I host games for a reason so people with slower internet are in client mode and do not require public ip!

Also not limiting network speed by game itself would also help when downloading/uploading map. I feel like there should be option for P2P/client-server hosting. Without VPN/ UDP hole punching there won't be many internet games going on because almost nobody plays on LAN these days.

I just hope that this LAN multiplayer is P2P and promised internet high ping MP will be client-server style. Early alphas are pain sometimes. I guess it was easier to do than client-server? Factorio devs said it's lan only, so if anyone have 4-6 players on lan, does it run good?
How can server handling of traffic help it, when one of the client is slow. As long as the latency is same for everyone, they still have to wait for the data from the slow client to be transferred to the server.

User avatar
DaveMcW
Smart Inserter
Smart Inserter
Posts: 3699
Joined: Tue May 13, 2014 11:06 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by DaveMcW »

kovarex wrote:How can server handling of traffic help it, when one of the client is slow. As long as the latency is same for everyone, they still have to wait for the data from the slow client to be transferred to the server.
The server processes all client input at the tick it is received, then tells clients what the correct tick is. Clients must wait for the server's response before they do anything. (Or they can guess and correct themselves when the server responds.)

This guarantees each client has lag equal to twice their ping, but is not affected by other clients.
Last edited by DaveMcW on Wed Dec 10, 2014 10:27 am, edited 1 time in total.

User avatar
Phillip_Lynx
Filter Inserter
Filter Inserter
Posts: 541
Joined: Mon Jul 21, 2014 6:00 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by Phillip_Lynx »

Sorry for my incompetence :)
DaveMcW wrote:Clients must wait for the server's response before they do anything. (Or they can guess and correct themselves when the server responds.)
All clients for them self or all clients together?

In the first case I think the slowest will have maaany teleporting an suddenly apperaring players/factories until he catches up.
In the second case I see no difference between PtP.


But maybe I am totally wrong with my asumptions.

User avatar
cpy
Filter Inserter
Filter Inserter
Posts: 839
Joined: Thu Jul 31, 2014 5:34 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by cpy »

kovarex wrote:How can server handling of traffic help it, when one of the client is slow. As long as the latency is same for everyone, they still have to wait for the data from the slow client to be transferred to the server.
Because factorio is P2P, my guess is that every player upload actions to every other player. Now with 4 players that is 3x required upload for each person. Server-client situation keeps burden of uploading to everyone to server only and clients only need to send less traffic.
Next, because there is no such thing as internet neutrality some connections are just STUPIDLY shaped now with P2P and everyone sending everything to everyone, expose those BS shapers and increase lag by much. When serve is on good provider that have good route to every other guy it will lag much less than competing ISP that shape each others traffic. Since every traffic is server-client, their routers won't know that they play together because they are not routing traffic between them.

Not talking about overloaded routes! Some routes get overloaded by huge amount of services using same line, but some server in the middle of nowhere where routes are clear can route traffic faster than using overcrowded routes. With server-client we can try different host to see who has best case scenario host. Also NOBODY except host need to have firewall rules and public ip!

If someone have stupidly disproportional internet 90/1 then it would be as if host had 1mbit and that itself can be limiting, but it might be enough for client mode gaming. P2P bad! Client-server good! Transfering HUGE files? P2P good! Client server - bad (unless you have fast host)!

I have to play game with people who have 50/50 or better internet in order to have playable experience in P2P multiplayer. You said it correctly this is LAN multiplayer and not internet, so don't expect you can pull off P2P on the internet. Upload speed is route to hell and back on some internet connections.

First google result from stack exchange:

P2P

Pros:


- No need for a central server: this makes it much cheaper, and more viable for low-budget indie games.
- Scales very well(up to a certain point when the average client just cant handle the bandwidth).
- Very good for data distribution: Suits games where user-created content is dynamically synced.(also just look at torrents)
- More Stable: It can never happen that the server is having problems and no-one can play(implementation dependent).

Cons:

- Hard to implement: much harder to create a solid P2P architecture, than a server-client.
- Its very hard to prevent cheating in such a system, unless you designate an authoritative peer, but then again you lose much of the benefits of scaling well.
- Security is hard to achieve.
- A clients internet connection can influence the game for others too.
- Latency is usually much bigger(tough can be better when joining an internet game with multiple people from a lan network for example).

Server-Client:


Pros:

- Easy to implement: Its as straightforward as it gets.
- If implemented well, scales extremely well.(if the work can be distributed on multiple servers)
- Cheating can be avoided easily.(relative to P2P)
- Lower Latency: If the server has a solid connection the latency can be extremely low.
- A player's internet connection never affects the other's game.

Cons:


- Costs you money to run the servers: definitely not suitable for a free game.(unless you let the players set up a dedicated server, but that might be a security problem)
- If the server has problems, everyone has.

Architekt
Burner Inserter
Burner Inserter
Posts: 11
Joined: Thu Dec 04, 2014 2:21 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by Architekt »

cpy: Your idea about Client-Server architecture doesn't apply for Factorio. In game like Factorio every client has to synchronize with all others. That means that even with central server all clients will still have to wait for slowest one to synchronize. So the slow one will always ruin game for others. It may be bit better than P2P, but still not playable.

Drexir
Burner Inserter
Burner Inserter
Posts: 19
Joined: Tue Apr 29, 2014 5:04 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by Drexir »

Well late to the topic. Thanks kovarex for the excellent write-up. Amazes me that my first and only language is English yet it's your secondary language or third? Yet still write better English.

As far as the P2P thing goes. It's very critical for FPS or any high demand game where syncing is small in size but is critical to send and receive quickly. Which is why I was disappointed to see GTA IV was P2P. So if you wanted to play a game of team deathmatch or any game mode besides free roam it was hard to take seriously (as in did we just win due to luck of the draw or did we or 'I' perform better. Not to mention in that environment is going to be rampant with cheaters. Not even something as simple as checking the client's data for modification was performed. So anyone could go into the game files and suddenly you're smg has the ability to shoot rockets.

Also there was no voting so if someone wasn't playing by the rules you couldn't kick them out. You simply had to make all your games private. Which is why I and much smarter people than me made an admin system for the game. The host is the only one able to kick players. He is also capable of handing those rights to another person. If the host disconnects for whatever reason it initiates a vote so that the players can choose who has the new admin right. Simple system it can be abused though just like any other admin of a server in which you simply just don't join that persons "server". Actually to me that had more benefit of providing an anti-cheat measure system. Because in a dedicated server environment you're not always going to be guaranteed for someone with admin rights to be on the server. In the case of the host is well hosting so he has to be present.

As far as P2P vs server to client in syncing goes. Well that problem has never been solved. Many games try to get around it by using interpolation, lag compensation, basically all kinds of prediction systems because you will never be capable of syncing everyone at all times.

Imagine we have two players connected to a server.

Player 1: Connects.
Server: Tells Player 1 the state of the game (for this example) 10 times per second.
Player 1: Tells the Server what actions he has performed 10 times per second.

However there is a delay between Player 1 and the Server let's say he has a decent connection and the latency or delay is 30ms. So every 100ms the player receives a new state from the server and sends his actions (uploads) to the server. But you have to add on the 30ms delay. So...

Player 1: Tells the server I just performed some action and it took 130ms to let the server know.
Server: Does it calculations based on the new data and attempts to verify that what the player said is actually true.

Now here's where things get a little more interesting.

Player 2: Connects. Let's say just like in the real world his delay is longer to the server at 60ms.
Server: Tells Player 1 and Player 2 the state of the game.
Player 1: Receives it.
Player 2: Receives it 30ms after Player 1.

Now here's where it can get a bit messy with syncing.

Player 1: Performs some actions let's say shoots Player 2. It again takes 130ms to tell the server.
Server: Computes the new data and verifies it. And now that data is sent to Player 2.
Player 2: Receives it 160ms later.

So in total after Player 1 shot Player 2 it took Player 2 290ms to be aware of it.

However if you were to go in reverse order and Player 2 shoots Player 1 you still have a 290ms delay. For Player 1 to be notified of Player 2's action.
So in short a server-client model the player with the worse connection is in fact still causing every one else to have a worse experience than they otherwise would.

I haven't even talked about packets dropping or just simply getting stuck in traffic at times. If I drop a packet then we'll everyone frowns on me. Things get a little wacky. In fact sometimes people will exploit this by purposely saturating their upload bandwidth so longer delays or dropped packets are guaranteed. It's inevitable to happen which is why techniques like interpolation are used. But these techniques to get around these issues is never going to be rock solid.

User avatar
cpy
Filter Inserter
Filter Inserter
Posts: 839
Joined: Thu Jul 31, 2014 5:34 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by cpy »

But i have 100/100mbit net and if P2P decide that oh that guy with 1mbit upload should upload map to next guy, kicks in we're in for loooong wait. Also upload speed is limited by factorio itself not only your net. So what happens then?

LordFedora
Filter Inserter
Filter Inserter
Posts: 310
Joined: Fri Nov 07, 2014 3:46 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by LordFedora »

The single uploader is a problem of how factorio implemented p2p

It should be downloaded from every person connected at the same time as fast as their download speed can handle

User avatar
cube
Former Staff
Former Staff
Posts: 1111
Joined: Tue Mar 05, 2013 8:14 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by cube »

The single uploader is a bug. Plain and simple. In our local tests everyone is uploading at max speed they can, and once we fix this bug, it will work the same for internet game.s

User avatar
Devildog
Fast Inserter
Fast Inserter
Posts: 170
Joined: Tue Aug 19, 2014 8:19 pm
Contact:

Re: Friday Facts #63 The endless struggle

Post by Devildog »

kovarex wrote:The report from our struggle is here: https://www.factorio.com/blog/post/fff-63
Little late but, Happy Birthday kovarex! - Dec 6.
Image
Check Out Factorio Mod Madness 2.0 Here : http://bit.ly/1afTDl0
Mod Madness ModPack

kovarex
Factorio Staff
Factorio Staff
Posts: 8078
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #63 The endless struggle

Post by kovarex »

Devildog wrote:
kovarex wrote:The report from our struggle is here: https://www.factorio.com/blog/post/fff-63
Little late but, Happy Birthday kovarex! - Dec 6.
Thanks

Post Reply

Return to “News”