Friday Facts #136 - Map Transfers

Regular reports on Factorio development.
slpwnd
Factorio Staff
Factorio Staff
Posts: 1835
Joined: Sun Feb 03, 2013 2:51 pm
Contact:

Friday Facts #136 - Map Transfers

Post by slpwnd »

Learn about technical aspects of new multiplayer map transfer mechanism: http://www.factorio.com/blog/post/fff-136

FacToRizeD
Burner Inserter
Burner Inserter
Posts: 5
Joined: Mon Apr 25, 2016 4:12 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by FacToRizeD »

Would it be possible to use a peer-to-peer mechanism for sending the map?
This would allow multiple players to send the map data at once while ensuring the correctness of the data.

User avatar
cube
Former Staff
Former Staff
Posts: 1111
Joined: Tue Mar 05, 2013 8:14 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by cube »

FacToRizeD wrote:Would it be possible to use a peer-to-peer mechanism for sending the map?
This would allow multiple players to send the map data at once while ensuring the correctness of the data.
It should be working that way now if you enable the P2P option when starting the game. There might be bugs with that though.

Woehler
Inserter
Inserter
Posts: 25
Joined: Wed Nov 11, 2015 8:47 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Woehler »

What, no fancy pictures showing the difference between the new and old transfer protocol D:

User avatar
Phillip_Lynx
Filter Inserter
Filter Inserter
Posts: 541
Joined: Mon Jul 21, 2014 6:00 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Phillip_Lynx »

This post will be more technical than usual, so let us know if you found it interesting.
I love this type of FFF. Keep on making such posts :).

tyaigan
Burner Inserter
Burner Inserter
Posts: 7
Joined: Sat Mar 05, 2016 12:18 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by tyaigan »

Image

But i loved the TL;DR

Too technical for me (coder and FFF lover.)

Yttrium
Long Handed Inserter
Long Handed Inserter
Posts: 81
Joined: Mon Jan 05, 2015 1:47 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Yttrium »

cube spent some time doing computer stuff
Very informative Friday Fact indeed! Keep it up Kappa

malecord
Fast Inserter
Fast Inserter
Posts: 131
Joined: Wed Mar 23, 2016 11:23 am
Contact:

Re: Friday Facts #136 - Map Transfers

Post by malecord »

And so you started with CUBIC and ended with Westwood. You defeated history sir XD.

Anyway, this FFF made me smile. :D My master thesis work was about measuring a bunch of innovative cc algorithms when they have to live along each other. You brought me back the memories of the nights I spent metabolizing that RFC stuff you linked. Thanks God is over now. :lol:

User avatar
Arch666Angel
Smart Inserter
Smart Inserter
Posts: 1636
Joined: Sun Oct 18, 2015 11:52 am
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Arch666Angel »

All hail the cube!

User avatar
Xterminator
Filter Inserter
Filter Inserter
Posts: 981
Joined: Sun Jun 15, 2014 4:49 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Xterminator »

While I didn't understand a ton of the technical stuff, it was still fun to read and gave me a better idea of how things work behind the scenes. :D

Very glad the transfer speed should be faster now though!
Image Image Image

ratchetfreak
Filter Inserter
Filter Inserter
Posts: 950
Joined: Sat May 23, 2015 12:10 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by ratchetfreak »

BTW wifi has its own retransmission setup on packet loss to avoid TCP being throttled way too much. So microwaves are less of a problem than your think (they still are but not as much).

Meritic
Burner Inserter
Burner Inserter
Posts: 12
Joined: Fri Apr 29, 2016 4:20 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Meritic »

Well actually me and my friend dont see any difference. Download speed begins at about 1,5kB/sec yet rapidly drops to 0B/s. Actually it is worse then 12.30 because it never returns back as it did in 12.30. Currently we both got 12.31

User avatar
Smarty
Global Moderator
Global Moderator
Posts: 816
Joined: Sat Oct 04, 2014 5:00 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Smarty »

Arch666Angel wrote:All hail the cube!
but you got the color wrong it should be blue






Too technical Image

MatthewGP
Inserter
Inserter
Posts: 26
Joined: Thu Mar 24, 2016 10:11 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by MatthewGP »

I was curious if you were using a Network Emulator for your testing. Something like 'netem' on a linux system or one of many Network / WAN Emulators is invaluable when analyzing how latency, loss, and re-ordering affect your application.

Matthew

djnono
Burner Inserter
Burner Inserter
Posts: 12
Joined: Fri Apr 29, 2016 4:24 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by djnono »

I actually love the technical fff :D :D
And as i happen to be a sysadmin/network eng., I actually understood all of that ;)

Though I must say the rfcs are one of the worst things ever, they are very informative and contains all the informations, but they sometimes are painful to read.
also there are a few tools that are invaluable (wireshark comes first to mind to debug something like that), and you can easily write a script that send a known data sample to the network as a loop, and one that can receive it, and check it agains a local stored copy of the original. (I had to do that once to find a transient error between several of our systems, I used a 10 gig file containtng pi decimals, i still have nightmare about it :( )
-- Amahashadow

Ratzap
Filter Inserter
Filter Inserter
Posts: 371
Joined: Sun Aug 16, 2015 11:15 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Ratzap »

Somewhat light on the technical side but then I'm not really the target audience. It shows how long things take to get into common practice though doesn't it. RFC in 1999 and it's still working it's way into actual use. It's found it's way into some NFS and web server implementations though as options.

mtocrat
Manual Inserter
Manual Inserter
Posts: 2
Joined: Fri Apr 29, 2016 5:17 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by mtocrat »

What I don't understand is why the maps need to be so big. Wouldn't a seed and a diff be both sufficient and relatively small?

RackAttack
Manual Inserter
Manual Inserter
Posts: 4
Joined: Fri Dec 12, 2014 6:26 am
Contact:

Re: Friday Facts #136 - Map Transfers

Post by RackAttack »

mtocrat wrote:What I don't understand is why the maps need to be so big. Wouldn't a seed and a diff be both sufficient and relatively small?
I would imagine they don't want to use a seed because that will fail for any sort of custom map, and it will fail if their map generation algorithm ever changes.

ikiris
Inserter
Inserter
Posts: 26
Joined: Sun Jul 19, 2015 5:57 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by ikiris »

You really shouldn't have done this the way you did it. Either use TCP or something like QUIC, but honestly you should probably go with something distributed like bitswarm considering the potential size of the transfers, and the requirement for speed.

The TCP punching argument is actually pretty wrong. Based on this I get the feeling you're using IP literals, which is pretty broken on its own. See ipv6 and XLAT for examples as to why. Multiprotocol nat punching is no harder than single flow nat punching so I don't understand this argument at all, especially considering you can't ever guarentee you can even reuse a nat state with non specific tuples.

Lastly, for the love of god, please implement a basic UPNP requester, this isn't 1993. There should be 0 need to do this stuff manually with forwarding and firewalls. You're a network client / server app right now. Act like it.


(this rant from a slightly frustrated network/systems engineer)

User avatar
Narnach
Inserter
Inserter
Posts: 32
Joined: Sun Mar 10, 2013 4:45 pm
Contact:

Re: Friday Facts #136 - Map Transfers

Post by Narnach »

Thanks, I enjoyed reading it. I think you managed to communicate a technical topic in a fairly accessible way. That said, I'm a programmer as well, so I might be overestimating the grokkability of this :-D

Funny how we always end up doing a 90% re-implementation of something else (TCP in this case), due to slowly discovering all the edge cases it actually solves.
Identity is cheap, reputation is priceless.

Let's Player over at https://youtube.com/narnach

Post Reply

Return to “News”