[MP] Async map download

[gheift]

Hello,

I do not now, if it was suggested before. But would it be possible to implement the map download asynchrony?

I think of the following steps:

• player wants to join
• a snapshot/save of the game is created
• the new player downloads the map while the other can continue with the game
• when the new player has finished downloading, game stops
• another snapshot is created and the replay-data from snapshot one to this snapshot is sent to the new player
• new player applies replay-data
• game continues with new player

Regards,
Gerhard

[ssilk]

Yes, that was already suggested and we had a lot of discussions around it (most of it in General).
The outcome is, that it is possible, but there is a point, when it is not useful: When the factory becomes so big, that your simulation cannot keep up. So this feature highly depends on your computers capability and that is the biggest reason against it, cause the game cannot know, how fast you can keep up.

[starholme]

You know, I still don't see any reason that this couldn't work for the vast majority of situations.

New peer requests map -> host/peers(if p2p) creates save -> host/peers start transfer of save -> host/peers resume play -> host/peers store each action into a log(essentially anything that needs to be handled by all peers, like movement, clicks, etc.) -> host/peers finish transferring save -> host/peers begin transferring log -> new peer loads save -> new peer begins applying log as fast as possible(as many UPS as the machine can do) -> new peer catches up to existing peers -> new peer player 'joins' game.

It would fail in the situation that the new peer cannot run faster the the existing peers. So a large factory that only runs at 40 UPS on a fast computer, would not be easily joinable by a computer that runs less that 40 UPS.

In this case (or say, any case that the peer would need more than X seconds to catch up), the existing host/peers could pause while the new peer applies the log.

The only real reason not to do this is Dev time. Dev time to develop, troubleshoot, test, keep updated. And Dev time is probably the most limited resource at Wube.

[ssilk]

You know, I still don't see any reason that this couldn't work for the vast majority of situations.

You are right, but it is still a problem to make that feature work in every case (think to factories, that run with 10 UPS, should the players wait an hour?). So it is a feature, that needs a very, very high load of work to implement, but with an uncertain result.

Every product owner would reject such features, if there are others, that will bring more for the product. And as said, this is already discussed and a possibility, but I think (I cannot be sure!) it has a very, very low priority.

[Zeblote]

A start would be to actually use all the bandwidth, if I have 10mbit upload it should send the map at 10mbit not 1...

[ratchetfreak]

Your router/ISP may be throttling the upload. Unless I'm mistaken factorio uses UDP for communication and many ISPs don't really like it because a careless device can flood a network with packets because there is no built in congestion control.

[Zeblote]

Your router/ISP may be throttling the upload. Unless I'm mistaken factorio uses UDP for communication and many ISPs don't really like it because a careless device can flood a network with packets because there is no built in congestion control.

The problem is definitely with whatever custom protocol factorio builds on UDP, other games have no problem using the full bandwidth.

[ssilk]

We also had that discussion to improve the bandwith and and yes, that will bring much more and it becomes a bit off-topic now.

[gheift]

If faster than realtime processing is not possible, what about a delta compression like http://xdelta.org/?

[ssilk]

Hm. I'm skeptical. If you have a factory with 10,000 moving items (not so much) you have a delta of 10,000 each 1/60 second.
It takes some time to calculate that diff.

[gheift]

I did a test on a 30mb map: after 2 minutes it is only 500kb.

[ssilk]

Hm. That sounds indeed very interesting.

Before I install that myself, some questions:
How long did it take to calculate that?
How big is the difference for one, two, 10 seconds?
Did you used the zip or the unzipped contents of the save?