Friday Facts #136 - Map Transfers

[Rseding91]

What I don't understand is why the maps need to be so big. Wouldn't a seed and a diff be both sufficient and relatively small?

Try generating a 4000x4000 tile map once vs loading a 4000x4000 map and let me know how that goes. A rough estimate: generating would take 10~ minutes, loading takes 20 seconds.

[BlaatMekker]

Cool details, keep them coming .

[Proxy]

All hail the cube!

But Tesseracts are more Awesome, or maybe Pentachorons?
hmm, atleast Tesseracts can Transfer Liquids, Items and Infinite RF/t. xD

eitherway, this was to Technical for me. What i understood was that Downloading Maps in SMP for every Player is getting a Problem due to big/modded Maps (i guess). so they were thinking of new Systems to speed that Process up. (or let us Play a In-Game version of Galaga or something while it's Loading... )

[Urth]

I love the technical posts and thought process behind an approach. Please keep throwing the techies a bone every now and then!

[ketil]

It was fun to read, I love technical details, however, I feel that firewall, NAT-stuff are non-issues:

Unfortunately in Factorio nothing is ever easy . Using TCP would have three important consequences: It would force players to open TCP ports as well as UDP ports in firewalls and NAT servers, it would remove the possibility of using NAT punching now or later with MP matching server and it would cause problems with downloading from multiple players at the same time.

If you can open a port on UDP in a firewall then you can open a port on TCP as well.

NAT is ugly and deserve to die. The future is ipv6 with unique ip-adresses for each device. The best solution is that everyone get at least one ipv6 /64 subnet from their ISP, /56 or /48 if they need multiple subnets. If that is not possible then 6in4 and 6to4 exist, along with vpn services are workarounds, although at the expense of latency and bandwidth.

It is not an unreasonable requirement that all game servers (as in accepting new incoming unrelated connections) does not hide behind NAT.

[Popiel]

Speaking as a coder myself, I loved the detail. Thank you.

[studmuffin]

Being a Network Engineer, the technical explanation made me happy!

Which version is this going to be implemented in? Would I have to select the 12.32 experimental in steam to test it out?

[golfmiketango]

You really shouldn't have done this the way you did it. Either use TCP or something like QUIC, but honestly you should probably go with something distributed like bitswarm considering the potential size of the transfers, and the requirement for speed.

The TCP punching argument is actually pretty wrong. Based on this I get the feeling you're using IP literals, which is pretty broken on its own. See ipv6 and XLAT for examples as to why. Multiprotocol nat punching is no harder than single flow nat punching so I don't understand this argument at all, especially considering you can't ever guarentee you can even reuse a nat state with non specific tuples.

Lastly, for the love of god, please implement a basic UPNP requester, this isn't 1993. There should be 0 need to do this stuff manually with forwarding and firewalls. You're a network client / server app right now. Act like it.


(this rant from a slightly frustrated network/systems engineer)

Not sure what non-specific tuples are (sounds like some kind of skin condition to me) but I suspect ikiris is barking up the right tree. Rolling your own lossless out-of-order network transport on top of udp is just really damn hard to get right in a way that "just works" from the end-user's perspective. A lot of things that work great in the office are simply going to go tits up in the various slightly- (and occasionally, mostly-) broken networking environments factorio is going to find itself in. It's not an /impossible/ problem, and, indeed, it's a very interesting puzzle, so I understand the desire to solve it. But the number of weird edge-cases your code will be bumping up against is so great, that I'd recommend you try harder to "buy" rather than "build" this. Maybe take a look at ENET?

Also ikiris is probably right about nat-punching -- you're at the mercy of the person controlling the router; that guy/gal either is or is not capable of tweaking their router to allow incoming connections from the Internet. If you want to support a reliable conversation between two unroutable ingress points, you're going to need a middlebox, and if that's not acceptable, you're up a creek. UPNP/NAT-PMP works in a lot of environments. "Ask your network administrator to allow this port through the firewall" works in most of the rest. Beyond that you are going to need a new Internet.

[Rakshasa]

Do you do a checksum of the final map data?

<- BT client coder, helped a lot with obscure bugs.

[bobingabout]

I gotta ask. what happens on systems with high latency? IE, the packets are actually getting there, but your ack is like, 3 seconds? The issue in these cases is not that you're sending too much data, it's just taking a long time to get your ack back.

[Rseding91]

I gotta ask. what happens on systems with high latency? IE, the packets are actually getting there, but your ack is like, 3 seconds? The issue in these cases is not that you're sending too much data, it's just taking a long time to get your ack back.

3 second ping times == you're boned.

Otherwise adjust the latency slider on the game when you start up the MP session.

[Oxyd]

Do you do a checksum of the final map data?

Factorio does map checksums regularly, yes. Not only because of map transfers, but also because the game can simply desync whilst running.

[starholme]

I'd like to take this time to thank the Devs for wasting my afternoon at work. Got stuck in a wiki-walk through TCP congestion articles. Ended up reading about beekeeping.

[ikiris]

You really shouldn't have done this the way you did it. Either use TCP or something like QUIC, but honestly you should probably go with something distributed like bitswarm considering the potential size of the transfers, and the requirement for speed.

The TCP punching argument is actually pretty wrong. Based on this I get the feeling you're using IP literals, which is pretty broken on its own. See ipv6 and XLAT for examples as to why. Multiprotocol nat punching is no harder than single flow nat punching so I don't understand this argument at all, especially considering you can't ever guarentee you can even reuse a nat state with non specific tuples.

Lastly, for the love of god, please implement a basic UPNP requester, this isn't 1993. There should be 0 need to do this stuff manually with forwarding and firewalls. You're a network client / server app right now. Act like it.


(this rant from a slightly frustrated network/systems engineer)

Not sure what non-specific tuples are (sounds like some kind of skin condition to me) but I suspect ikiris is barking up the right tree. Rolling your own lossless out-of-order network transport on top of udp is just really damn hard to get right in a way that "just works" from the end-user's perspective. A lot of things that work great in the office are simply going to go tits up in the various slightly- (and occasionally, mostly-) broken networking environments factorio is going to find itself in. It's not an /impossible/ problem, and, indeed, it's a very interesting puzzle, so I understand the desire to solve it. But the number of weird edge-cases your code will be bumping up against is so great, that I'd recommend you try harder to "buy" rather than "build" this. Maybe take a look at ENET?

Also ikiris is probably right about nat-punching -- you're at the mercy of the person controlling the router; that guy/gal either is or is not capable of tweaking their router to allow incoming connections from the Internet. If you want to support a reliable conversation between two unroutable ingress points, you're going to need a middlebox, and if that's not acceptable, you're up a creek. UPNP/NAT-PMP works in a lot of environments. "Ask your network administrator to allow this port through the firewall" works in most of the rest. Beyond that you are going to need a new Internet.

tuple in this case being the state of SRC IP, SRC PORT, DST IP, DST PORT, Protocol (UDP/TCP/etc).

There are multiple flavors of NAT (Note here they're actually talking about port address translation, in addition to network address translation), and only a few of them work like they seem to think it does and allows simple punchthrough on a state.

All of this though is regardless of the fact that it is 2016 and they still expect users to have anything to do with port translations in their firewall instead of implementing a basic upnp library like every other user network server in existance. I'm honestly embarassed for their network stack if they are worrying about reimplmeneting the wheel of networking before doing simple and expected things like supporting upnp or simple NAT-T service discovery.

EDIT: changed network client to network server as it should be

[RackAttack]

You really shouldn't have done this the way you did it. Either use TCP or something like QUIC, but honestly you should probably go with something distributed like bitswarm considering the potential size of the transfers, and the requirement for speed.

The TCP punching argument is actually pretty wrong. Based on this I get the feeling you're using IP literals, which is pretty broken on its own. See ipv6 and XLAT for examples as to why. Multiprotocol nat punching is no harder than single flow nat punching so I don't understand this argument at all, especially considering you can't ever guarentee you can even reuse a nat state with non specific tuples.

Lastly, for the love of god, please implement a basic UPNP requester, this isn't 1993. There should be 0 need to do this stuff manually with forwarding and firewalls. You're a network client / server app right now. Act like it.


(this rant from a slightly frustrated network/systems engineer)

I'm not well versed enough in networking to know the details like ikiris seems to (though I have dabbled at various times as part of my job), but I have to agree with his overall sentiment here. It seems like you guys are trying to reinvent from scratch a solution to a fairly complex problem that has already been solved by others. I work at a company where we too have a tendency to write everything from scratch, and it's often a good idea in the long run because it results in more control and better understanding. So, I totally understand the urge to get your hands dirty and write a bunch of code, but when it comes to networking stuff, even we pull in resources from other companies who specialize in that area.

All that is to say, I'm glad to see that you're trying to fix the map download speed problem, and I hope your solution works. But if it has problems once it's deployed, I hope you'll go back and re-consider your base assumptions that are preventing you from utilizing widely used off the shelf solutions that are known to work well.

[sabriath]

I've written tutorials on the subject and made many programs for networking, so some suggestions from me:

1. As ikris points out, please request for mapping (I believe RFC 6887 - PCP mapping dynamics? It's been a few years, I might be rusty)
2. Using TCP does not require a return acknowledgement....it's automatic inside the protocol. You can setSendBufferSize (or setsockopt with SO_SNDBUF, depending on what tool you're using) in order to change the amount of data to buffer up through the protocol ahead of time, but a default amount is usually good enough. When you actually "send" the data, it returns a value which matches how much of the data is actually placed into this buffer for transfer. When this value is less than the amount you actually specified, that means the buffer is full and network has congestion. So, you truncate the data you just sent and wait a few ticks to try again. In order to not block the network onto any given player from the server, you can track the data-per-second rate of each player and keep them at an average level.
3. I would suggest using a separate TCP connection to send map details, and you can actually send the chunks on an as-needed basis. Any update information as the player receives it under the normal connection can be packed until the chunk is sent from the snapshot and applied afterward. This will allow players to immediately play within a small area at the start and the area will increase as chunks become available. It will help the server from having a bottle-neck of update data (as in #2, each connection can be divided by network traffic usage, so chunk data won't bog everything down).
4. In terms of saving locally, this could help even further by giving the player a map to start with, and then updating the chunks. Simply mark all chunks as "not updated" and request each chunk as needed while the player plays....there might be tearing, but at least they'll be playing immediately.
5. Make use of OOB for immediate data....like biter attacks, those are more important than details of whether the iron plate in the furnace is done.

[ikiris]

I understand the desire to avoid a full RTT and slow start, but my point is, you're doing it wrong if you need to for a simple transfer. I expect your real issue is simply running into windows not doing window scaling for uploads as a client. This is trivially worked around by the way. You can also bypass slow start.

But above all else, if you're trying to do something like this, you shouldn't be writing your own raw socket work, and use a library like civilized folk. If by some miracle you actually need to avoid a full RTT, just use QUIC. There is no reason you should be reinventing the world for simple stuff like this, which is very complicated, and you probably are missing pieces that others have figured out (as evidenced earlier - missing automated port mapping, improperly handling packet loss and sequencing issues, etc). I get you guys want to do your own stack for game comms in UDP, but simple file transfers aren't something that should be encompassed in that, not to mention how you deal with multiprotocol NAT/PAT etc.

[cube]

I gotta ask. what happens on systems with high latency? IE, the packets are actually getting there, but your ack is like, 3 seconds? The issue in these cases is not that you're sending too much data, it's just taking a long time to get your ack back.

With so high latency you won't be able to play, but the file transfer should still work.

[ssilk]

4. In terms of saving locally, this could help even further by giving the player a map to start with, and then updating the chunks. Simply mark all chunks as "not updated" and request each chunk as needed while the player plays....there might be tearing, but at least they'll be playing immediately.

It sounds like a solution but is not possible. The game needs to have the whole map, before it can start simulation, otherwise the determinism between two players computers isn't guaranteed. Simple example: you make a long belt far from the outlands into your factory. Then you need to load also that outpost, before simulation can start.
The FFF linked to other articles where that is explained.

Lately there was a cooler idea to transfer the last save in background (everyone can continue to play) and just before matching in transfer a diff of the map since then. That diff is much smaller of course. Another idea is based on the same idea (transfer last auto save), but then transfer the replay-save, so that the simulation needs just to fast forward until it is in sync.
Maybe a combination of both is another way?

Sorry, no links, cause written from iPhone

[DaveMcW]

It was fun to read, I love technical details, however, I feel that firewall, NAT-stuff are non-issues:

... NAT punching ...

If you can open a port on UDP in a firewall then you can open a port on TCP as well.

You should read about http://en.wikipedia.org/wiki/UDP_hole_punching

The same technique is sometimes extended to Transmission Control Protocol (TCP) connections, with less success due to the fact that TCP connection streams are controlled by the host OS, not the application, and sequence numbers are selected randomly; thus any NAT device that performs sequence number checking will not consider the packets to be associated with an existing connection and drop them.