Friday Facts #76 - MP inside out

Regular reports on Factorio development.
immibis
Filter Inserter
Filter Inserter
Posts: 300
Joined: Sun Mar 24, 2013 2:25 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by immibis » Sun Mar 08, 2015 6:46 am

ofca wrote:Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?
Well yeah... why do you think it re-downloads the map if any one of those hundred thousand entities differs slightly?

MF-
Smart Inserter
Smart Inserter
Posts: 1235
Joined: Sun Feb 24, 2013 12:07 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by MF- » Sun Mar 08, 2015 9:48 am

Richwarf wrote:Have the "auto-save" become "auto-push", so all the clients still run on there own but one player pushes there game to all other clients. So this way everyone wins.
Also auto-save hapens in a reasonable time because it doesn't happen over the internet.
"auto-push" would therefore be the initial map download happening every now and then.

ssilk wrote:They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.
That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?

Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?

What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?
Everyone downloads the master player's version, hoping that not too much went wrong in their camps?

User avatar
ssilk
Global Moderator
Global Moderator
Posts: 10474
Joined: Tue Apr 16, 2013 10:35 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ssilk » Sun Mar 08, 2015 10:26 am

MF- wrote:
ssilk wrote:They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.
That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?
When multiplayer is stable of course. Which is in my eyes with the next version. If I would be a developer I would say, to replace this checking-code with the next version. Not now, cause that has the risk of bringing in new bugs.
Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?
As explained, it's a weighting of risks. I won't do that, cause the current version is not thought for that. Simple logic. The devs need to find bugs. If they add an option "turn down checksum traffic" now, they add the risk, that it is not only turned of by those, who need it, but also by those, which often play and so reduce the chance of finding new bugs easily.
What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?
The cause and the error will have a larger distance in time. It will be much harder to find the cause then, cause it can happen at any time after the last check.
Everyone downloads the master player's version, hoping that not too much went wrong in their camps?
That would be the worst case. :) they need the bugs first! Then you can continue playing.
Cool suggestion: Eatable MOUSE-pointers.
Have you used the Advanced Search today?
Need help, question? FAQ - Wiki - Forum help
I still like small signatures...

kovarex
Factorio Staff
Factorio Staff
Posts: 7387
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by kovarex » Sun Mar 08, 2015 3:49 pm

MF- wrote:
Richwarf wrote:Have the "auto-save" become "auto-push", so all the clients still run on there own but one player pushes there game to all other clients. So this way everyone wins.
Also auto-save hapens in a reasonable time because it doesn't happen over the internet.
"auto-push" would therefore be the initial map download happening every now and then.

ssilk wrote:They currently send as much checksum-data as possible over the line to see as soon as possible if there is some desync.
That calls for a question.
Since factorio shouldn't ever be finished,
when will be the right time to remove/reduce the checks?

Those people claiming that their internet cannot handle factorio
would really benefit from reduced checksum count/intervals, right?

What exactly should happen if the checksum interval is set to 20 seconds and a desync occurs, though?
Everyone downloads the master player's version, hoping that not too much went wrong in their camps?

The checksum is crc of some easy to calculate data, crc is 4 bytes, so you don't have to worry :)

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Mon Mar 09, 2015 4:16 am

immibis wrote:
ofca wrote:Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?
Well yeah... why do you think it re-downloads the map if any one of those hundred thousand entities differs slightly?
Please note that this answer doesn't even fly close to answering my post and I'm still interested.

chainedlupine
Inserter
Inserter
Posts: 48
Joined: Mon May 19, 2014 4:35 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by chainedlupine » Mon Mar 09, 2015 6:24 am

Good job on the NAT punchthru, BTW.

Seriously. Even with the LAN issues.

I've played AAA+ big-budget titles with peer-to-peer lockstep networking which couldn't even get that part right.

Tribble
Manual Inserter
Manual Inserter
Posts: 1
Joined: Mon Mar 09, 2015 12:34 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by Tribble » Mon Mar 09, 2015 12:35 pm

Definately loving the technical posts! Keep up the good work!

starholme
Fast Inserter
Fast Inserter
Posts: 198
Joined: Tue Oct 21, 2014 7:25 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by starholme » Mon Mar 09, 2015 2:57 pm

ofca wrote:Major changes from current architecture probably won't happen, nor would they be feasible. As to the hundred thousand entities, while one can't dispute their existence, I think there's only one troublesome entity in this game, and it's the more-or-less unpredictable player. Other entities are deterministic to a fault, and one may argue that them behaving differently on multiple ends are either 1) coding errors, 2) result of cheating, or 3) hardware/software errors outside the game, right?
Yeah, you are correct in saying that anything that breaks determinism(other than the player) is a bug, cheating, or hardware issue. The whole point of lockstep computing is that things are identical over a period of time. You can't solve the last two generally, so you still require things like the crc checks to determine when a client is out of sync. Once a client is out of sync, you must place them back in sync, rolling back any changes since the last time the crc passed. Currently factorio appears to use similar functionality as joining a multiplayer game: Download the entire save file from the host/other clients. That's probably the simplest approach, you really don't require much 'new' code.

I'm not a dev, so I can only guess, but it appears that they want to get the game to the point where you only ever get a desync when a player cheats, or there is a hardware glitch. At this point desync's would hopefully be so rare that it will NEVER make sense to spend developer hours making desync recovery faster. If it takes 100 dev hours to halve the desync recovery time, and your average player only see's a desync every couple hundred hours of gametime, those 100 hours are better spent elsewhere.

User avatar
darkrising
Inserter
Inserter
Posts: 25
Joined: Sun May 04, 2014 7:32 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by darkrising » Tue Mar 10, 2015 4:33 pm

I like the idea of p2p but what about the poor guy that is sitting on a 3G connection behind some crappy gateway which doesn't allow open ports? I'm assuming they would have negative effects on other parties playing the game. If a server could be setup on a box open to the internet with the same p2p principle surely that would make everything better for the dude on the 3G connection :)
Check out my gaming community, we hope to one day host Factorio servers! http://www.darkserver.co.uk
Also check out my mod: Alien Science https://forums.factorio.com/forum/vie ... =14&t=3373

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Wed Mar 11, 2015 3:07 am

Main problems in multiplayer that I see are:
- Desyncs
- Conflicts
- Responsiveness / Latency / Drops, general connectivity
- Governance / Administrative issues

Desyncs:
Solved in Factorio by redownloading the game state from [somewhere]. How the game determines who is desynced and who's not? How is this [somewhere] determined when there are, lets say, 20 players? Like starholme said, preventing random rare non-recurring non-deterministic desyncs will be a waste of developer time, since they are easily fixed after being detected. One direction that might be worth exploring, especially in light of game's ambition to grow infinitely which may cause problems with many megabytes (or more) data to be re-downloaded, may be some notion of snapshots/milestones, so not whole state has to be downloaded, but instead changes from last known good milestone? Obviously there's still the last resort of downloading the game state from [somewhere]. This milestone may be an autosave - there's the question of how much memory/computational time such an approach will take. Anyway, just an idea, in case somehow devs didn't think about it :)

Conflicts:
Two or more players are manipulating/interacting with the same object, be it a tile with conveyor belt and resources, or an enemy being hit by bullets. How are the conflicts resolved? Or are they even resolved, or the game is just allowed to go into desync state, which is then taken care of by desync resolving mechanism?

Latency:
Well, to be honest I don't see how this will be resolved while keeping the current architecture. Currently there's just no way for player with inferior hardware or terrible connection to play with others while not ruining fun for everyone. Everyone needs to receive this lagging player's action(s)/crc of game state before game may proceed to next step, which depends on 1) player's machine fully calculating current state of simulation, 2) player transmitting previously calculated information to every other player.

Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc.. Steam ID integration would work nicely here, to keep track of players' resources/inventory. There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can, and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.

What are your thoughts? Are there more aspects that I didn't think of?

User avatar
Nova
Filter Inserter
Filter Inserter
Posts: 921
Joined: Mon Mar 04, 2013 12:13 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by Nova » Wed Mar 11, 2015 4:18 am

Can't we just let this topic rest until we get the "real" multiplayer?
Greetings, Nova.
Factorio is one of the greatest games I ever played, with one of the best developers I ever heard of. Image

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Wed Mar 11, 2015 4:40 am

Will getting "real multiplayer" satisfy my curiosity? Or is that illegal in your state? ;)

User avatar
Piranha
Fast Inserter
Fast Inserter
Posts: 152
Joined: Fri Nov 21, 2014 8:15 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by Piranha » Wed Mar 11, 2015 4:45 am

ofca wrote:Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc..
One could probably create a mod to do this, if not.. post it to the ideas section and I strongly feel the devs would implement it for one of the next updates. It really shouldn't be to hard to set up the code for this.
ofca wrote:Steam ID integration would work nicely here, to keep track of players' resources/inventory.
This is obviously the easier answer, and would make the kick/ban thing simple. Soon steam soon you will be blessed with having Factorio.
ofca wrote:There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can
Check out the Videos section, there are currently 3 PvP series that have been played. None that were anything but friendly.
-Fishsandwich vs Xterminator
-UK vs USA
-Piranha vs Klonan
..and no DDoS attacks.. Although Fish and Nick may have been attempting that at one point ;)
ofca wrote:..and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.
I disagree, once 24/7 servers are up having a server that purposely ruins game play for specific people would be pointless as they would just leave and find another server.


My guess is 24/7 servers would be under 3 main category, Vanilla, Modded, PvP. Vanilla and Modded could just be ran 24/7 with admin privileges given to players so that no one person can destroy the world for others, with the possibility of pausing the game when no one is in the server, and maybe even private servers with a password or something similar. PvP would be need to be handled differently where two teams build bases and attack each other with one team the eventual winner and map resets. This is where my idea of having a capital structure would come in handy.

Anyways, whatever is in the future will be great. This game is awesome and I can see the potential for it to grow and grow and grow!

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Wed Mar 11, 2015 5:33 am

Uh, you totally misunderstood my post. I seem to be getting this a lot here, wonder if it's on my end :)

Lets go from bottom to top, because why not :P

Servers deciding whether a player is in sync or not wasn't meant as a way to ruin fun for players joining, but to prevent the clients from messing with others' game. Again, I'm not sure how the sync state and who's in sync or not is decided today.
DDoS attacks, hacking and stuff is the future once the game hits the wide world. Lets not compare current friendly PVP matches played by dedicated factorio fans to bored kids that will get their fun out of ruining game for others, especially streamers.
On the administrative part, I actually would like to agree with Nova - this is one part that may easily wait for final multiplayer shape.

kovarex
Factorio Staff
Factorio Staff
Posts: 7387
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by kovarex » Wed Mar 11, 2015 8:41 am

ofca wrote: Desyncs:
Solved in Factorio by redownloading the game state from [somewhere]. How the game determines who is desynced and who's not? How is this [somewhere] determined when there are, lets say, 20 players? Like starholme said, preventing random rare non-recurring non-deterministic desyncs will be a waste of developer time, since they are easily fixed after being detected. One direction that might be worth exploring, especially in light of game's ambition to grow infinitely which may cause problems with many megabytes (or more) data to be re-downloaded, may be some notion of snapshots/milestones, so not whole state has to be downloaded, but instead changes from last known good milestone? Obviously there's still the last resort of downloading the game state from [somewhere]. This milestone may be an autosave - there's the question of how much memory/computational time such an approach will take. Anyway, just an idea, in case somehow devs didn't think about it :)
Waste of time is creating snapshots. The solution is, to just avoid desyncs completely, which we kind of achieved already, at least for our game.
ofca wrote: Conflicts:
Two or more players are manipulating/interacting with the same object, be it a tile with conveyor belt and resources, or an enemy being hit by bullets. How are the conflicts resolved? Or are they even resolved, or the game is just allowed to go into desync state, which is then taken care of by desync resolving mechanism?
This problem is non-existent. One of the players is always the first one to interact with the object.
ofca wrote: Latency:
Well, to be honest I don't see how this will be resolved while keeping the current architecture. Currently there's just no way for player with inferior hardware or terrible connection to play with others while not ruining fun for everyone. Everyone needs to receive this lagging player's action(s)/crc of game state before game may proceed to next step, which depends on 1) player's machine fully calculating current state of simulation, 2) player transmitting previously calculated information to every other player.
This has nothing to do with latency, but rather the sustainable game speed. Well the solution is to optimise the game. People with bad hardware will just have to avoid playing huge maps anyway, we are not building a MMO here.
ofca wrote: Administration:
Hopefully current state is alpha-grade and this will be worked on later, but I would like to be able to decide whom I play with, kick/ban them etc.. Steam ID integration would work nicely here, to keep track of players' resources/inventory. There may be some kind of PVP mode, where people play on insanely large maps and clash into each other sooner or later, but I feel that p2p architecture will prove inadequate to situations, where competitive players go to resorts such as DDoSing each other, cheating and otherwise ruining fun for others in any way they can, and running 24/7 servers that clearly govern the game play and arbitrarily decide who's in sync and who's not may be the future.
Why would you DDos anyone? Why don't just disconnect from the game and pretent the other one disconnected? This problem is also non-existent. If you play competetively, you have some kind of account. People that are reported to be cheating a lot will just get banned, really easy.

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Wed Mar 11, 2015 9:12 am

Yes, it's obviously best to get rid of desyncs all-together, and I didn't see much of them even before 0.11.17 :)

I wouldn't DDoS anyone. And most of the multiplayer problems don't exist if you plan with one other person ;) - problems, actually all kinds of them, start to show when you're playing with multiple people over the Internet -- but it was stated numerous times that 0.11.x in no way represents the final quality of multiplayer, and it wasn't intended for Internet-based play; that's why I'm so interested what major changes are in the works for 0.12 that will make Factorio Internet-worthy.

From what I gather the effective latency (time it takes for tick to be processed -- or plainly, time between pressing w and your character moving) is slowest player's time to render next simulation frame+time needed to reach furthest player, right?

And one last thing that interests me the most: why did you choose the peer2peer approach instead of some twist on client-server? I don't advocate that it's a bad choice or anything, merely would like to know the reasons (and especially gains of one over the other) - some of the reasons may be extrapolated from latest FFF, but it feels sadly incomplete and leaving me hungry for more details :)

Thanks for your time :)

kovarex
Factorio Staff
Factorio Staff
Posts: 7387
Joined: Wed Feb 06, 2013 12:00 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by kovarex » Wed Mar 11, 2015 9:22 am

The main reason why we don't consider it to be fully internet ready is the latency. The problem is the delay between sending message and receiving it (ping time).

Solving this problem is the main part (apart making some better support for opposing teams) of the changes we plan for the 0.12.

The solution is to act like player started moving immediately, although the "real player" will start to move later. The mechanism needs to be simple, and there is never ever some kind of reversing, you really can't reverse in Factorio. The "ghost player" will just be reversed to the real player when he can't get to the position anymore, but this is mainly visual trick to make the game feel more responsive.

starholme
Fast Inserter
Fast Inserter
Posts: 198
Joined: Tue Oct 21, 2014 7:25 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by starholme » Wed Mar 11, 2015 5:16 pm

ofca wrote: And one last thing that interests me the most: why did you choose the peer2peer approach instead of some twist on client-server? I don't advocate that it's a bad choice or anything, merely would like to know the reasons (and especially gains of one over the other) - some of the reasons may be extrapolated from latest FFF, but it feels sadly incomplete and leaving me hungry for more details :)
I thought they went into pretty good detail. The biggest factor is that Factorio has the interesting property of ridiculous amounts of active things. The quantity of things that are happening essentially requires that all the work be done client side to reduce the network traffic. The slowest client computer controls the tick speed of the game.
In a client server model, you could run the tick speed at whatever the server could handle. But you have to send every action of every entity every tick(or at least every entity within a certain range of the client avatar). Requiring way more bandwidth.

So the client only sends player actions and crc's, and only receives player actions and crc's. Nothing else. Pretty much impossible to reduce the amount of traffic below this.

Currently, each client sends all their actions and crc's to every other client. So the latency is the round trip time between the two farthest clients. If you went to a 'lite' client-server model where the clients send all the actions and crc's to the server, then the server sends to all the other clients, your latency would be the round trip time between the farthest client to the server, PLUS the round trip time to the second farthest to the server. The server would also require enough bandwidth to forward all these messages.

Another overlooked benefit of p2p is that you should be able to keep playing if original hoster disconnects. When he boots back up(cause windows update restarted his computer without asking), he can just join again, using the address of any player still running.

Looking forward:
I wouldn't be surprised if eventually one of the clients in the p2p group becomes 'special'. There may be a special one already for all I know. This special client might handle the tasks that a server would in most games.
Things like being a proxy between clients if a client isn't reachable from the internet directly. Or deciding who can join the game, or who should be kicked from the game.
A 'dedicated' server wouldn't need to be special, really just another peer that doesn't have an avatar. Ideally, it could run without a GUI, and handle pausing/unpausing when there are no other clients.

User avatar
DaveMcW
Smart Inserter
Smart Inserter
Posts: 2806
Joined: Tue May 13, 2014 11:06 am
Contact:

Re: Friday Facts #76 - MP inside out

Post by DaveMcW » Wed Mar 11, 2015 6:13 pm

starholme wrote:If you went to a 'lite' client-server model where the clients send all the actions and crc's to the server, then the server sends to all the other clients, your latency would be the round trip time between the farthest client to the server, PLUS the round trip time to the second farthest to the server. The server would also require enough bandwidth to forward all these messages.
No. The latency for 'lite client-server' would be:

farthest client->server + server->farthest client
OR
farthest client->server + server->second farthest client

In other words, every client has latency equal to the average of their own connection to the server and the slowest client's connection to the server. The slowest computer would still set a limit on the speed of the simulation, due to the lockstep architecture.

This method scales up extremely well (assuming your server can handle everything). It is the only reasonable choice for massively multiplayer games.
Last edited by DaveMcW on Wed Mar 11, 2015 6:18 pm, edited 1 time in total.

ofca
Long Handed Inserter
Long Handed Inserter
Posts: 69
Joined: Sat Feb 21, 2015 1:49 pm
Contact:

Re: Friday Facts #76 - MP inside out

Post by ofca » Wed Mar 11, 2015 6:18 pm

Yes, I would be looking at some hybrid mode too, but when game hits steam and our thousands of positive reviews convince others to buy the game, we will see how well Factorio will scale into tens of players on the wild internet :)

Post Reply

Return to “News”