Friday Facts #51 - First MP Game

[bobingabout]

Okay, Every time I read what you're currently trying to do with your multiplayer net code, I bang my head on my desk.

You're struggling through methods that were abandoned a long time ago, because they don't work very well.
First the peer to peer system is a very bad method. Most people will have an Asyncronous net link, meaning they have a far faster down than up. Transmitting everything to 3 people in a 4 player game is... Silly to say the least, hence why it was abandoned a long time ago in favor of a client server system. To consider anything other than a client server system is silly.

For starters, take the occurance of 2 people trying to mine the same tree. don't forget about the 2 second lag. Both are successful at mining it on their end, because the other person isn't anywhere near it yet, and therefore both transmit that they did it. these 2 clients will then disagree, who is right? nobody knows, desync.
Take a server based system instead, the server gets to decide who is right, and will transmit back to one of them that mining it failed. if you take a terraria aproach, you don't receave your loot from mining a tree untill the server gives you the loot for mining that tree, so both players mine the tree, server says player 1 mined it, and gets the loot, player 2 who also mined it simply doesn't get any loot, the server updates all clients to say the tree is dead.



Several key points you need to keep in mind when doing net code.
Will it work on dial up with 1 second lag? if the answer is no, you're doing it wrong. (Downloading things that are large, such as maps being the only exception to this rule)
Add 100 players, will it still work? it doesn't matter if you intend to have only a 4 player game, such as most FPS games like borderlands do, will it work with 100 players, on dial up? if the answer is no, you're doing it wrong.

Look at games like Rift and EVE Online, hundreds, or thousands of players can be within the same area, because they're controlled and relayed through 1 sever, rather than transmitting all at once. And both will work on dialup.

Also looking for inspiration from Terraria, look at how monsters work, the server relays updated information about where they are and what they're doing periodically, between updates, the client makes it's best guess, if it guesses wrong, the monster is teleported to the correct location. This is known as Client Trust, it doesn't matter if a desync occurs, the error is corrected on the fly if it is wrong. Therefore all 4 players (in a 4 player game) do not have to be totally in sync for the game to function smoothly.

[DaveMcW]

For starters, take the occurance of 2 people trying to mine the same tree. don't forget about the 2 second lag. Both are successful at mining it on their end, because the other person isn't anywhere near it yet, and therefore both transmit that they did it. these 2 clients will then disagree, who is right? nobody knows, desync.

In client/server, each person sends a "mine this tree" message to the server (1 second). The server decides who wins and sends a message back (1 second). No desync, 2 second lag between input and result.

In peer-to-peer, each person sends a "mine this tree" message to the other person (1 second). Each client follows the same tiebreak method, and the same person wins on both clients. No desync, 1 second lag between input and result.

[kovarex]

Okay, Every time I read what you're currently trying to do with your multiplayer net code, I bang my head on my desk.

You're struggling through methods that were abandoned a long time ago, because they don't work very well.
First the peer to peer system is a very bad method. Most people will have an Asyncronous net link, meaning they have a far faster down than up. Transmitting everything to 3 people in a 4 player game is... Silly to say the least, hence why it was abandoned a long time ago in favor of a client server system. To consider anything other than a client server system is silly.

For starters, take the occurance of 2 people trying to mine the same tree. don't forget about the 2 second lag. Both are successful at mining it on their end, because the other person isn't anywhere near it yet, and therefore both transmit that they did it. these 2 clients will then disagree, who is right? nobody knows, desync.
Take a server based system instead, the server gets to decide who is right, and will transmit back to one of them that mining it failed. if you take a terraria aproach, you don't receave your loot from mining a tree untill the server gives you the loot for mining that tree, so both players mine the tree, server says player 1 mined it, and gets the loot, player 2 who also mined it simply doesn't get any loot, the server updates all clients to say the tree is dead.



Several key points you need to keep in mind when doing net code.
Will it work on dial up with 1 second lag? if the answer is no, you're doing it wrong. (Downloading things that are large, such as maps being the only exception to this rule)
Add 100 players, will it still work? it doesn't matter if you intend to have only a 4 player game, such as most FPS games like borderlands do, will it work with 100 players, on dial up? if the answer is no, you're doing it wrong.

Look at games like Rift and EVE Online, hundreds, or thousands of players can be within the same area, because they're controlled and relayed through 1 sever, rather than transmitting all at once. And both will work on dialup.

Also looking for inspiration from Terraria, look at how monsters work, the server relays updated information about where they are and what they're doing periodically, between updates, the client makes it's best guess, if it guesses wrong, the monster is teleported to the correct location. This is known as Client Trust, it doesn't matter if a desync occurs, the error is corrected on the fly if it is wrong. Therefore all 4 players (in a 4 player game) do not have to be totally in sync for the game to function smoothly.

You have very simplified view of the model when you think, that 2 players mining a tree would desync it, we would be very naive if we made it that way.
Sending resync from server to client in Factorio is just not possible, the world is large and desynch in one thing often means you have to redownload the whole map.

[bobingabout]

You have very simplified view of the model when you think, that 2 players mining a tree would desync it, we would be very naive if we made it that way.
Sending resync from server to client in Factorio is just not possible, the world is large and desynch in one thing often means you have to redownload the whole map.

2 players mining 1 tree was purposely simplistic, it could be any number of tiny things constantly happening that could cause a desync, I know, such as if a biter chooses to attack or not, occupying biters making a nest, or a worm... etc etc.

Also, redownloading the entire map from a single desync... Downloading a 100MB map on dial up speeds...
I'll just go make myself some dinner and come back in hours, maybe it will be half way done by then.

Pretty much the only person I game with on a daily basis literally lives on the oposite side of the planet. I live in the UK. on a 5 star rating system, I'd give UK internet a 2 star. He lives in Australia, from what he's told me, I'd give it 0 star. A quote from Sony who were trying to put a server for... I think it was planetside 2 in australia... cannot be repeated in a PG forum, but to paraphrase "WHAT THE FUDGE IS THIS? HOW CAN YOU LIVE WITH THIS CRAP?" It's why most australian servers are actually located in china.

If you have to re-send the entier map from 1 desync... you're doing it wrong. if you're detecting a desync at all, you know what the problem is, just decide which person has the correct version, and update that one item.

And I can't stress hard enough that every player sending out the same info 3 times, once to every other player is just wrong. Even if you don't want to use server software (Which I really recomend you do if you want your game to succeed, you'd end up with people creating "Always online" factorio servers, where people can drop in 1 at a time and build things), you should designate one player as the host, and have them act as server, so instead of everyone talking to everyone, everyone only talks to the host, and the host sends the info to everyone else. This means host gets the final say.

On the subject of sending the entire map to everyone who connects, and every time it desyncs... another advantage of a server-client, or host-client system is that you don't have to send the entire map to everyone, and have it constantly updated. Terraria for example does not, it has chunks, and only the chunk that you are standing on is up to date. As you move around the world, chunks that you enter are sent and updated. This drastically improves connection time, as well as reducing the possibility of desyncs.

Now, I know what you're thinking... this is a total rewrite of how the game works, not just adding net code. you're right, it is, and I can further see issues with buildings such as radars that keep you informed of what's going on in an area, even if you're not there.

The problem is... you've so far written the game as a single player game, and are trying to tack multiplayer on using shortcuts.

To quote my australian friend. "They're doing a minecraft, this is a bad thing."


I'm not saying what I'm sugesting will be easy, but at the very least you should consider the server-client or host-client's master-slave relationship between the 4 players, rather than having them all equil.

[kovarex]

The big difference between terraria/minecraft and Factorio is, that everything affects everything in the Factory.

One coal piece taken too late means less energy for the boiler -> less electric energy for the electric network -> ten thousands of machines working slightly slower. There are lot of different examples of the same thing.
To make it work as server->client the server would basically have to send all the chunks data the player sees every tick to enable the player to see what is happening, and that is FAR FAR bigger requirement on the network, than to send few bytes every tick to
other players saying what you want to do.

You have very simplified view of the model when you think, that 2 players mining a tree would desync it, we would be very naive if we made it that way.
Sending resync from server to client in Factorio is just not possible, the world is large and desynch in one thing often means you have to redownload the whole map.

2 players mining 1 tree was purposely simplistic, it could be any number of tiny things constantly happening that could cause a desync, I know, such as if a biter chooses to attack or not, occupying biters making a nest, or a worm... etc etc.

Well all of these examples need not to happen, when the game is really deterministic, everything always happen the same way.

[bobingabout]

Point taken... that would be some big desync issue.

Still, i didn't say "Server has to do everything, and relay it to everyone". Although I did imply that other games did that, if you read my closing paragraph, what my main sugestion was, rather than everyone talking to everyone, the server acts as the relay.

Something else you could consider is a client trust thing, if something desyncronises... don't make a big deal of it, instead, if you did get that power drain which caused things to slow down... instead of forcing a complete resync, only update information of things you directly interact with. server chest has 100 units of batteries, client only has 98, update to 100. fix issues as you interact with them, rather than demanding absolute syncronisation.
At the data heaviest scenario, resync just that one tile you're interacting with.

Just remember, I'm voicing an opinion, this is your game, not mine, at the end of the day, you have the final say, and I understand that, but also keep in mind that I'm not just some nobody, I do have some experience, so please listen to what I have to say before deciding not to use it. No I'm not sugesting you're doing otherwise.


Still, I have to ask. Is dedicated server software something in the pipeline?

[Rakshasa]

Just remember, I'm voicing an opinion, this is your game, not mine, at the end of the day, you have the final say, and I understand that, but also keep in mind that I'm not just some nobody, I do have some experience, so please listen to what I have to say before deciding not to use it. No I'm not sugesting you're doing otherwise.

Isn't this the developer formerly from the financial IT world?... so I assume if he has done the kind of programming I assume he has, then he should know what he's doing.

It is basically a deterministic finite state machine, and where the input of players change the state deterministically. EVE Online does something similar, even though the server is the final arbitrator of input instead of a deterministic function.

They also had plenty of issues over the years with desync, however those were bugs not laggy networks. As factorio is able to detect a desync immediately they should have be able to track down issues with ease, making desync a rare case.

Not to mention there would not really be a need to download the whole map if all clients autosave on the same ticks, allowing the client to only download a diff.

[cube]

Sending diffs after a desync is something in the do-it-later-if-we-need-it department

[-root]

Pretty much the only person I game with on a daily basis literally lives on the oposite side of the planet. I live in the UK. on a 5 star rating system, I'd give UK internet a 2 star. He lives in Australia, from what he's told me, I'd give it 0 star. A quote from Sony who were trying to put a server for... I think it was planetside 2 in australia... cannot be repeated in a PG forum, but to paraphrase "WHAT THE FUDGE IS THIS? HOW CAN YOU LIVE WITH THIS CRAP?" It's why most australian servers are actually located in china.

As the factorio australian in residence i confirm this to be true. Our internet speeds bite the big one

[cube]

As the factorio australian in residence i confirm this to be true. Our internet speeds bite the big one :D

Everything is poisonous and internet connection like this... how can you live there?!?

[MF-]

If you have to re-send the entier map from 1 desync... you're doing it wrong. if you're detecting a desync at all, you know what the problem is, just decide which person has the correct version, and update that one item.

Actually, you don't know WHAT desynced. All you know is, that a hash of the states differ.
To figure out what is different you'd need to transfer the entire state and compare

And I can't stress hard enough that every player sending out the same info 3 times, once to every other player is just wrong.

Will see how much bandwidth does sending player input cost.
I don't see any reason why that should be over 1kB per tick

The problem is... you've so far written the game as a single player game, and are trying to tack multiplayer on using shortcuts.

What? Determinism was always stressed as a multiplayer mechanism.
Small replays and ability to rewind a save to any position in time is just a bonus on top.
Sending only non-deterministic inputs (player) was always the plan.

To quote my australian friend. "They're doing a minecraft, this is a bad thing."[CITATION NEEDED]

How do you know what minecraft sends over the network?
Was minecraft ever using the peer-to-peer concept?

[micomico]

As the factorio australian in residence i confirm this to be true. Our internet speeds bite the big one

Everything is poisonous and internet connection like this... how can you live there?!?

I'm here betting it involves less than legal substances. Or a boat load of legal ones.

[MF-]

Followup:
Hmm.. perhaps complete redownload might not be necessary.

For a low-bandwidth scenario the area might be split into smaller sub-areas,
for whose the checksums would be sent and compared instead,
re-downloading only those that don't match.

The trouble is, that all sides have to divide the map in the same way of course.

[kovarex]

Followup:
Hmm.. perhaps complete redownload might not be necessary.

For a low-bandwidth scenario the area might be split into smaller sub-areas,
for whose the checksums would be sent and compared instead,
re-downloading only those that don't match.

The trouble is, that all sides have to divide the map in the same way of course.

Yes, you can use this recursivelly to really send just the needed parts, which can be just a small portion of the game sometimes.

[cpy]

Looking how fast this game runs (except saving part) i'm sure MP will run cool and smooth.

[Drury]

As the factorio australian in residence i confirm this to be true. Our internet speeds bite the big one

Everything is poisonous and internet connection like this... how can you live there?!?

Venomous* if you're referring to the wildlife

Well anyway not to spend the bulk of my time here correcting the devs' grammar mistakes, how about I make myself look smart in a different way

Okay, Every time I read what you're currently trying to do with your multiplayer net code, I bang my head on my desk.

You're struggling through methods that were abandoned a long time ago, because they don't work very well.
First the peer to peer system is a very bad method. Most people will have an Asyncronous net link, meaning they have a far faster down than up. Transmitting everything to 3 people in a 4 player game is... Silly to say the least, hence why it was abandoned a long time ago in favor of a client server system. To consider anything other than a client server system is silly.

RTS games use peer to peer systems because server-centric method gets overflowed (as it's already been said). The clients affect each other too much compared to, say, an FPS, where they just move and shoot a gun of a single dude. It works and there are no deterministic issues (such as in 1v1 tank duel, no problem determining who wins). In case of desyncs, it's either determined who is the one responsible by things like ping and artifacts (i.e. who gets to redownload whose map), or, as in case of my beloved Zero Hour, the players vote themselves. Point being, peer-to-peer is being done in Factorio-type games in the present day and it works, so hush.

[FishSandwich]

Both words are acceptable. The word poisonous actually refers to the word venomous, so in most cases we just say something is poisonous.

[silman]

Both words are acceptable. The word poisonous actually refers to the word venomous, so in most cases we just say something is poisonous.

Actually that's completely true, they pretty much the same thing (in terms of bad stuff that is meant to harm prey), but the distinction comes in the transmission of it into the body.

Poisons will affect you if they ingested, inhaled, absorbed (i.e., by skin), or otherwise get into your system somehow. Venoms must be injected directly into your blood stream in order to take effect (you can drink a glass of snake venom and it probably won't harm you).

Poisons and venoms can also be interchangeable depending on the animal. Tetrodotoxin is a poison in the pufferfish, but venomous in the blue-ringed octopus.

[shadoh]

I can confirm that Australia internet sucks (I lived in Launceston Tasmania for a while). From my experience even NZ internet is better, and I live in a rural township (true rural is 10x worse though).

In Aussie I got 6Mbs down on a good day. Here in Kaeo NZ, I get 22Mbps consistently (We have ADSL2+ and basically get maximum speed)

[Nova]

Weird people here... I got a maximum of 150 kilobytes per second until a few years ago, and I live in germany. ^^