Friday Facts #46

[slpwnd]

The usual wall of text describing what we have been up to: http://www.factorio.com/blog/post/fff-46

[Lone_Player]

Look´s more like kovarex, take´s a close look at the network stuff. Surrounded with Green & Red Wires. Now where ist the chest....

[jeroon]

Friday Facts Multiplayer Tanks

Crashed paragliderso missing 3D artists

I like your definition of "Coop game", where you shoot your unarmed slave.. ehm.. partner in the face

[bobingabout]

The main question I have is how will the game perform when I try to play a multiplayer game with my friend.
I live in Britain, with a local ISP monopoly who have recently started providing Fibre to some areas, on a system that was already very strained and throttled to provide ADSL, now it's even worse.
My friend lives in Australia, literally the other side of the planet, and his local ISP monopoly is even worse. Aparantly, Sony tried to set up a server in Australia, and an admin wrote in the local chat "HOW CAN YOU LIVE WITH THIS SHIT!!!". that basically describes australian internet access. I think my friend has something like a 100kbps upload speed, thats kilobits, mine is supposed to be 1Mbps up and 8Mbps down, but last time I checked it was only 500kbps up, due to poor line quality.

we were having an unplayable multiplayer game of UnEpic together the other day, I'd swing my sword, and it would take about 2 seconds to actually swing, and then it went link dead.
so just to see how the link was, he pinged me. It wasn't a problem with the line, he said it was 280ms, thats a new record for his connection to mine. INSANELY FAST!!! 400ms is normal.



So... keeping scenarios like this in mind, how playable do you think multiplayer factorio would be between us?

[MF-]

Look´s more like kovarex, take´s a close look at the network stuff. Surrounded with Green & Red Wires. Now where ist the chest....

hehe, indeed.
I guess the high bush is a coverup for a substation?

[Glyph]

The main question I have is how will the game perform when I try to play a multiplayer game with my friend.
I live in Britain, with a local ISP monopoly who have recently started providing Fibre to some areas, on a system that was already very strained and throttled to provide ADSL, now it's even worse.
My friend lives in Australia, literally the other side of the planet, and his local ISP monopoly is even worse. Aparantly, Sony tried to set up a server in Australia, and an admin wrote in the local chat "HOW CAN YOU LIVE WITH THIS SHIT!!!". that basically describes australian internet access. I think my friend has something like a 100kbps upload speed, thats kilobits, mine is supposed to be 1Mbps up and 8Mbps down, but last time I checked it was only 500kbps up, due to poor line quality.

we were having an unplayable multiplayer game of UnEpic together the other day, I'd swing my sword, and it would take about 2 seconds to actually swing, and then it went link dead.
so just to see how the link was, he pinged me. It wasn't a problem with the line, he said it was 280ms, thats a new record for his connection to mine. INSANELY FAST!!! 400ms is normal.



So... keeping scenarios like this in mind, how playable do you think multiplayer factorio would be between us?

If they want semi-decent internet in Australia they're going to have to take to the streets at this point. Australia and Canada are two countries where I'd expect proper infrastructure but instead they have horrific cable monopolies worse than here in the U.S.

[mrrobot]

When I learned about this game it said "single-player" then I read about multiplayer in the FAQ and I was all excited. However after I learned about the decision for a peer-to-peer setup my hopes of being able to play it in multiplayer tend to go against zero.
I just can't see it working in this scenario. There's so much going on in the game that has to be kept in sync which reminds me of two games that failed to do it. One being Anno 16- or 17-hundred-something where they promised to implement MP but due to massive de-sync in game that promise was dropped. The other game being settlers 4, I believe. That was quite funny in a sad way. Sad because it was a LAN game and funny because it was de-synced so much that the game I played and the game my friend played were completely different. There was a forest in his dimension that I had chopped down to the last tree and our settlement flourished and thrived in my dimension and we were very close to kick the enemies rear and win the game when suddenly I heard my friend telling me it was game over because the enemy had defeated us.

The way I understand it is, with an example of a 4 player coop session, that there are 4 versions of the game running on 4 PCs. Each doing their own thing and syncing as much as possible everyone from and to everyone else. But you can't sync everything for everyone, thinking about a conveyor line that stretches for a few kilometers covered with ore - like several thousand bits of ore, mine fields, biters, hundreds of drones, etc. pp., each being an entity of its own - unless you've got a GB connection. That means the clients have to be running game logic by themselves which means duplicated effort. 1 world is simulated on 4 machines. Which will either result in a complete lag-fest or de-sync in the extend as described above and/or both. I guess there's a reason why peer-to-peer games that support more than 4 players are hard to find.
Edit: I mean MMORPGs use a client-server architecture for a reason.

/snip
something like a 100kbps upload speed, thats kilobits, mine is supposed to be 1Mbps up and 8Mbps down, but last time I checked it was only 500kbps up, due to poor line quality.

/snip (1)
he pinged me. It wasn't a problem with the line, he said it was 280ms, thats a new record for his connection to mine. INSANELY FAST!!! 400ms is normal.

(2)

So... keeping scenarios like this in mind, how playable do you think multiplayer factorio would be between us?

As for (2) I would expect quite unplayable. A friend of mine lives in a part of the country where internet connectivity is quite poor and our internet situation is similar to yours and your friend's. We can barely play awesomnauts with 3 players in game and once player no. 4 joins is a lag-fest at its finest.
The situation is quite similar for warframe - 3 players barely playable and no. 4 turns it into a lag-fest.
And about all the other peer-to-peer games we've had tried to play.

It doesn't matter that much where on the world you're located but what's in between point A and point B. I get a worse connection with my brother only 2 hours by car away, latency of 150-250 ms is usual, than to most places across the ocean all the way to the USA, latency of 50 ms is usual.

Speaking of latency (1). Ping is not very well suited to test the latency of your gaming performance. It's like comparing apples with oranges. The only meaningful measure would be within the app/game if it's not measuring its latency via ICMP but with normal app/game traffic on the same port(s). Usage of ping is unsuitable for 2 reasons, 1st one is that it's a different protocol than the app/game traffic, namely ICMP rather than UDP/TCP, the 2nd being that ISPs can prioritize this protocol/port to make their service appear fast(er) (e.g. 280 ms via ping vs. 400 ms in-game), or, due to abuse of ping (ping flood/DOS attacks) it can be completely blocked, i.e. packets are dropped and you won't get any reply but a time out. You can easily see that when you substitute the ping with a trace route (tracert) command. You will very likely see some time outs because those machines don't respond to ping.

I'm sorry, devs, for the lack of confidence in your decision but I just can't understand why you didn't decide for a client-server setup where the world simulation runs on one dedicated server with broadcasts of the state of the world and the clients making requests to the server about how they want to interact with the world. There's no need to wait for a reply to a can-I-walk-here request but instead update the real position is it wasn't possible, for instance. I know, easier said than done and all that(tm) and server-client has disadvantages of its own. It's just the way I would go about it. You've thought this through and didn't make this decision by tossing a coin so we'll see how it turns out in the end

Big annoyance at the moment, which prevents us from "playing more", is very frequent waiting for the other peer (which results in pausing the game for fraction of a second). In theory this should happen only occasionally due to the network lag, but now it happens pretty much all the time

Like the auto save lag since 0.10 which I blame on replays. 0.9 saves were much smaller, too, 6mb in 0.9 vs 124mb in 0.10 for a much bigger factory. I blame that on replays, which I would love to disable, and, hopefully, tons of debug data which will vanish in time.
On top of that auto save always strikes at an inconvenient time; while fending off hordes of biters, while removing a building, chopping down a tree, something like that.
I would appreciate one or all of the 3:
- A "Saving the game right now"-indicator; something like a small flashing diskette (I think those 3,5" disks were so cool which is probably because I'm getting old )
- An announcement before the game is saved, like a 10 seconds countdown or something to be able to prepare yourself for the multi-seconds lag
- Make it an auto save request that is executed at the most convenient time, i.e. after the "action" is complete; the building was removed/resource extracted/tree chopped down/not in combat anymore, etc. pp.

Apart form that I'm really, really loving the game - as in, my personal all time top 3 hall of fame kind of way.
Thanks for your hard work and keep it going I can't wait for the oil-tanks on rails which is my most anticipated item on your road-map, and fire, like out-of-control conflagration with fire fighter drones, and multiplayer goes without saying

Edit 2:

In theory this should /snip

Theory means that you know how it works but it doesn't.
Practice means that it works but you don't know why.
In Windows theory and practice are combined - nothing works and nobody knows why.

[slpwnd]

When I learned about this game it said "single-player" then I read about multiplayer in the FAQ and I was all excited. However after I learned about the decision for a peer-to-peer setup my hopes of being able to play it in multiplayer tend to go against zero.
I just can't see it working in this scenario. There's so much going on in the game that has to be kept in sync which reminds me of two games that failed to do it. One being Anno 16- or 17-hundred-something where they promised to implement MP but due to massive de-sync in game that promise was dropped. The other game being settlers 4, I believe. That was quite funny in a sad way. Sad because it was a LAN game and funny because it was de-synced so much that the game I played and the game my friend played were completely different. There was a forest in his dimension that I had chopped down to the last tree and our settlement flourished and thrived in my dimension and we were very close to kick the enemies rear and win the game when suddenly I heard my friend telling me it was game over because the enemy had defeated us.

The way I understand it is, with an example of a 4 player coop session, that there are 4 versions of the game running on 4 PCs. Each doing their own thing and syncing as much as possible everyone from and to everyone else. But you can't sync everything for everyone, thinking about a conveyor line that stretches for a few kilometers covered with ore - like several thousand bits of ore, mine fields, biters, hundreds of drones, etc. pp., each being an entity of its own - unless you've got a GB connection. That means the clients have to be running game logic by themselves which means duplicated effort. 1 world is simulated on 4 machines. Which will either result in a complete lag-fest or de-sync in the extend as described above and/or both. I guess there's a reason why peer-to-peer games that support more than 4 players are hard to find.
Edit: I mean MMORPGs use a client-server architecture for a reason.

Only the regular synchronization heartbeats (couple of bytes) and player inputs (only when player does something) will be transmitted over the network. The estimate is that this will add up to couple of kilobytes per second.

As for (2) I would expect quite unplayable. A friend of mine lives in a part of the country where internet connectivity is quite poor and our internet situation is similar to yours and your friend's. We can barely play awesomnauts with 3 players in game and once player no. 4 joins is a lag-fest at its finest.
The situation is quite similar for warframe - 3 players barely playable and no. 4 turns it into a lag-fest.
And about all the other peer-to-peer games we've had tried to play.

It doesn't matter that much where on the world you're located but what's in between point A and point B. I get a worse connection with my brother only 2 hours by car away, latency of 150-250 ms is usual, than to most places across the ocean all the way to the USA, latency of 50 ms is usual.

We are counting with latency of about 100ms, but this will be (later on) customizable. Though the latency will be the same for all the players in the game (so if one has slow connection the others will be slowed down as well).

I'm sorry, devs, for the lack of confidence in your decision but I just can't understand why you didn't decide for a client-server setup where the world simulation runs on one dedicated server with broadcasts of the state of the world and the clients making requests to the server about how they want to interact with the world. There's no need to wait for a reply to a can-I-walk-here request but instead update the real position is it wasn't possible, for instance. I know, easier said than done and all that(tm) and server-client has disadvantages of its own. It's just the way I would go about it. You've thought this through and didn't make this decision by tossing a coin so we'll see how it turns out in the end

Client server wouldn't work simply because the game state is sooo huge. Think of all the entities on the transport belts. Therefore we believe that the "lock step" multiplayer model is the only viable for us.

Also in the later versions there will be some latency hiding mechanisms (when you move with the player you don't need to wait for the confirmation from all the others).

[mrrobot]

Thanks for the insight. Now I've got a much clearer picture of your approach.

Only the regular synchronization heartbeats (couple of bytes) and player inputs (only when player does something) will be transmitted over the network. The estimate is that this will add up to couple of kilobytes per second.

This however confirms my fear about de-synced games, as in a Settlers kind-of-way.
You've got to make sure that every entity makes the same decision on every instance of the game, in the very same tick, and executes that decision in the same exact way.
For instance, player A and B meet at a biter nest. Both run through the nest in a shotgun frenzy fashion. The outcome I would expect is different, that the same biters choose different targets - the player they see first on each player's machine - and get killed by different players because they are running differently and occupy different tiles. Potentially the biter nest would be extinct on Player A's version of the game and player B might run for their life from biters that no longer exist on player A's machine. Those Schrödinger-biters could be pulled into a factory and cause havoc despite being dead - or - are they?
But then again I believe that for every problem there is a solution.
On a second thought... Factorio is not Settlers and de-sync has yet to be an issue, but if it were you could add a "repair"-button to be able to manually get the game in sync again, either by downloading or exchanging files in a different manner. Just a thought...

We are counting with latency of about 100ms, but this will be (later on) customizable. Though the latency will be the same for all the players in the game (so if one has slow connection the others will be slowed down as well).

However, network latency is something beyond the player's control. So if you update per tick, bad latency sounds like the game will be slow motion on latency worse than 100 ms. Fair enough as long as it's stable, i.e. I'd prefer a stable (if need be) artificially worse latency of ~180 ms than some jumping latency between 75 ms - 150 ms or rubber-banding or stuttering and the likes.

Client server wouldn't work simply because the game state is sooo huge. Think of all the entities on the transport belts. Therefore we believe that the "lock step" multiplayer model is the only viable for us.

That's what I'm saying. It's a huge simulation with potentially hundreds of thousands of entities. That's why I thought it would be more feasible to broadcast it like a radio station stream. But eventually it would be too much to handle. I can see your reasoning.

You certainly picked an "easy" game for a multiplayer candidate I'm very curious how this turns out and I wish you the best of success

[mrrobot]

Are you planning on support for logging off a game and later rejoin after the game moved on ? Or even new players join a game at an arbitrary time ?

[cube]

Are you planning on support for logging off a game and later rejoin after the game moved on ? Or even new players join a game at an arbitrary time ?

Right now we only have one player starting the game and others joining at any time.

[mrrobot]

Are you planning on support for logging off a game and later rejoin after the game moved on ? Or even new players join a game at an arbitrary time ?

Right now we only have one player starting the game and others joining at any time.

You rock! Awesome

[shadoh]

I hate peer-to-peer, trying to play Command & Conquer 3 on multiplayer is plagued with connection issues. I know 3 people who would have played that game if it weren't for issues such as that. We ended up having a few games over VPN (such as hamachi, OpenVPN, tunngle) but those programs are so hackish, wish it just worked properly.

Will it be possible to perhaps get around the connectivity issues by having the client-server model as an option?
Maybe have it automatically fall-over in to a client-server model if peer-to-peer fails to connect.

Here's my idea of how the fail-safe mode could work:
* client fails to establish connection to every other player (even if one connection fails)
* client connects to a primary player, who is the only one who really needs to do port-forwarding or UPnP stuff
* client directs all traffic to that primary, and the primary forwards his net messages to the other players.
* internally1 the other players would know about his fail-safe status, and not try to connect to him.
* the primary would then send other players status updates to the fail-safe player

1By internally I mean the players wouldn't know unless perhaps they checked the connection quality screen (or something similar), perhaps a message in the chat box upon connecting as well.

[Glyph]

I hate peer-to-peer, trying to play Command & Conquer 3 on multiplayer is plagued with connection issues. I know 3 people who would have played that game if it weren't for issues such as that. We ended up having a few games over VPN (such as hamachi, OpenVPN, tunngle) but those programs are so hackish, wish it just worked properly.

Will it be possible to perhaps get around the connectivity issues by having the client-server model as an option?
Maybe have it automatically fall-over in to a client-server model if peer-to-peer fails to connect.

Here's my idea of how the fail-safe mode could work:
* client fails to establish connection to every other player (even if one connection fails)
* client connects to a primary player, who is the only one who really needs to do port-forwarding or UPnP stuff
* client directs all traffic to that primary, and the primary forwards his net messages to the other players.
* internally1 the other players would know about his fail-safe status, and not try to connect to him.
* the primary would then send other players status updates to the fail-safe player

1By internally I mean the players wouldn't know unless perhaps they checked the connection quality screen (or something similar), perhaps a message in the chat box upon connecting as well.

I kind of wonder if by "peer to peer" they meant "client server" to begin with. It makes sense from a coding standpoint to have one host and a bunch of clients that connect to them. This is how, for example, Warcraft 3 worked.

[Nova]

I hate peer-to-peer, trying to play Command & Conquer 3 on multiplayer is plagued with connection issues. I know 3 people who would have played that game if it weren't for issues such as that. We ended up having a few games over VPN (such as hamachi, OpenVPN, tunngle) but those programs are so hackish, wish it just worked properly.

For me and every friend I know who plays Command & Conquer 3 (Tiberium Wars), nobody has any problem with hamachi or connection issues. C&C 3 is one of the games which work very well with Hamachi, unlike some others like Far Cry 2. I can't remember ever having problems with the game. (But maybe my brain is just a little forgetful... :> )
A small note: The addon, kane's wrath, indeed has problems with synchronity. That's because of the cheat protection. Restarting the computer (not only the game!) after every match has solved this problem for me and my friends.

[cube]

The game is really peer to peer. All players are almost equal (order of joining is occasionally used for tie breaks), you can connect to the game through any of the peers. This actually wasn't a goal in the design, but it kind of emerged.

If problems with this approach appear, we will add the packet forwarding as shadoh suggested. We already discussed this, but decided to skip it for now as it's not essential for basic MP.

[MF-]

Yeah, definitely a working MP first, see how it turns out.

thoughts about dedicated servers can come later.

[Glyph]

Personally I would much rather have one trusted game state. When MP servers become popular, P2P opens up all sorts of security issues, along with the standard sync issues it can cause. I know MP isn't nearly done, but that is where I'd really like to see it head.

[DaveMcW]

When MP servers become popular, P2P opens up all sorts of security issues, along with the standard sync issues it can cause.

I assume that if someone is cheating, the game will desync and everyone reloads from the host. You have to trust the host of course, but that's the same as trusting the server.

[Glyph]

I assume that if someone is cheating, the game will desync and everyone reloads from the host. You have to trust the host of course, but that's the same as trusting the server.

In true P2P there is no host, correct?

I am fine with "having" to trust a host. That is totally acceptable (and encouraged!) in sandbox games where each server/host is isolated from the others.