Page 1 of 5

Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:09 pm
by slpwnd
Learn about technical aspects of new multiplayer map transfer mechanism: http://www.factorio.com/blog/post/fff-136

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:19 pm
by FacToRizeD
Would it be possible to use a peer-to-peer mechanism for sending the map?
This would allow multiple players to send the map data at once while ensuring the correctness of the data.

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:21 pm
by cube
FacToRizeD wrote:Would it be possible to use a peer-to-peer mechanism for sending the map?
This would allow multiple players to send the map data at once while ensuring the correctness of the data.
It should be working that way now if you enable the P2P option when starting the game. There might be bugs with that though.

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:26 pm
by Woehler
What, no fancy pictures showing the difference between the new and old transfer protocol D:

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:33 pm
by Phillip_Lynx
This post will be more technical than usual, so let us know if you found it interesting.
I love this type of FFF. Keep on making such posts :).

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:36 pm
by tyaigan
Image

But i loved the TL;DR

Too technical for me (coder and FFF lover.)

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:48 pm
by Yttrium
cube spent some time doing computer stuff
Very informative Friday Fact indeed! Keep it up Kappa

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 3:51 pm
by malecord
And so you started with CUBIC and ended with Westwood. You defeated history sir XD.

Anyway, this FFF made me smile. :D My master thesis work was about measuring a bunch of innovative cc algorithms when they have to live along each other. You brought me back the memories of the nights I spent metabolizing that RFC stuff you linked. Thanks God is over now. :lol:

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:20 pm
by Arch666Angel
All hail the cube!

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:21 pm
by Xterminator
While I didn't understand a ton of the technical stuff, it was still fun to read and gave me a better idea of how things work behind the scenes. :D

Very glad the transfer speed should be faster now though!

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:22 pm
by ratchetfreak
BTW wifi has its own retransmission setup on packet loss to avoid TCP being throttled way too much. So microwaves are less of a problem than your think (they still are but not as much).

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:24 pm
by Meritic
Well actually me and my friend dont see any difference. Download speed begins at about 1,5kB/sec yet rapidly drops to 0B/s. Actually it is worse then 12.30 because it never returns back as it did in 12.30. Currently we both got 12.31

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:25 pm
by Smarty
Arch666Angel wrote:All hail the cube!
but you got the color wrong it should be blue






Too technical Image

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:29 pm
by MatthewGP
I was curious if you were using a Network Emulator for your testing. Something like 'netem' on a linux system or one of many Network / WAN Emulators is invaluable when analyzing how latency, loss, and re-ordering affect your application.

Matthew

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:38 pm
by djnono
I actually love the technical fff :D :D
And as i happen to be a sysadmin/network eng., I actually understood all of that ;)

Though I must say the rfcs are one of the worst things ever, they are very informative and contains all the informations, but they sometimes are painful to read.
also there are a few tools that are invaluable (wireshark comes first to mind to debug something like that), and you can easily write a script that send a known data sample to the network as a loop, and one that can receive it, and check it agains a local stored copy of the original. (I had to do that once to find a transient error between several of our systems, I used a 10 gig file containtng pi decimals, i still have nightmare about it :( )

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 4:40 pm
by Ratzap
Somewhat light on the technical side but then I'm not really the target audience. It shows how long things take to get into common practice though doesn't it. RFC in 1999 and it's still working it's way into actual use. It's found it's way into some NFS and web server implementations though as options.

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 5:21 pm
by mtocrat
What I don't understand is why the maps need to be so big. Wouldn't a seed and a diff be both sufficient and relatively small?

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 5:28 pm
by RackAttack
mtocrat wrote:What I don't understand is why the maps need to be so big. Wouldn't a seed and a diff be both sufficient and relatively small?
I would imagine they don't want to use a seed because that will fail for any sort of custom map, and it will fail if their map generation algorithm ever changes.

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 5:43 pm
by ikiris
You really shouldn't have done this the way you did it. Either use TCP or something like QUIC, but honestly you should probably go with something distributed like bitswarm considering the potential size of the transfers, and the requirement for speed.

The TCP punching argument is actually pretty wrong. Based on this I get the feeling you're using IP literals, which is pretty broken on its own. See ipv6 and XLAT for examples as to why. Multiprotocol nat punching is no harder than single flow nat punching so I don't understand this argument at all, especially considering you can't ever guarentee you can even reuse a nat state with non specific tuples.

Lastly, for the love of god, please implement a basic UPNP requester, this isn't 1993. There should be 0 need to do this stuff manually with forwarding and firewalls. You're a network client / server app right now. Act like it.


(this rant from a slightly frustrated network/systems engineer)

Re: Friday Facts #136 - Map Transfers

Posted: Fri Apr 29, 2016 6:06 pm
by Narnach
Thanks, I enjoyed reading it. I think you managed to communicate a technical topic in a fairly accessible way. That said, I'm a programmer as well, so I might be overestimating the grokkability of this :-D

Funny how we always end up doing a 90% re-implementation of something else (TCP in this case), due to slowly discovering all the edge cases it actually solves.