Ideas to increase multiplayer map transfer speed

[HL65536]

short version:

map transfer now is not very smart:
in case of a new player connecting or desync: whole map is transferred
solution:
-client already has parts of map information (earlier save or desynced partially wrong info)
-identify correct info and transfer only what is needed



long version with implementation ideas (do not read if you are not a programmer or at least interested in technical details):


my upload is 100KB/s, game says 50-60KB/s; with a 30MB map its almost 10min waiting PER PLAYER
change 1: use tcp to transfer map or fix existing data transfer algorithm to fully utilize upload speed

easy to implement changes:

step 1: identify existing map information:
if(desync) use existing partially wrong info
else if(client played on this server before) use latest save from playing on that server or let player choose best matching save
else generate map on client from map exchange string (further improvements could include the server checking which chunks are generated from this version of the map generator

step 2: check validity of information and transfer
this needs to have goups of data defined:
grouping one chunk together is not sufficient as the chunks are usually different between saves
one group must meet the following criteria:
the probability needs to be very high that the data does either exactly match or is completely different
good groups would be certain parts of chunks eg. all terrain info, all resource info or all building info of a chunk
algorithm ("while" means steps happen in parallel):
for each group in a chunk calculate a weak checksum/hash, client sends these to server while server sends info of which chunks are already generated to client
client generates chunks that are missing while server sends all groups that do not match to the client while client sends better hashes to server
check newly generated chunks for differences with the same procedure

as data is now divided far smaller and redundancy is reduced, zip is not a good option anymore, some other methods should be used eg. as chunkIDs are already defined by group membership, coordinates could be reduced to a few bits per entity
if zip gets any more compression out of it, there is too much redundancy

harder to implement but a better long term solution would be to use chunk loading system similar to minecraft

[Zeblote]

The file size could also be massively reduced by not saving any tiles/entities that can be re-generated from the map seed.

[ssilk]

Hm.

I think all of this has already been discussed and there should be a link collection with the current results.

[Zeblote]

Hm.

I think all of this has already been discussed and there should be a link collection with the current results.

I really don't think the last point has ever been discussed. It's always dismissed as "it's faster to load the map than re-generate it" but that doesn't apply when it takes someone 10 minutes to upload it.

[HL65536]

with 0.14 i really hoped for massive improvements but it seems as they are only moderate:
upload speed is now fully utilized so im down to 5min upload per player which is better but still far from optimal.
This is still preventing lots of players that want multiplayer from playing with a larger amount of people or with larger worlds.
Dont stop improving!

[ssilk]

I think that statement is better placed in General, cause a suggestion to not stop improving is - hehe - honorable, but useless.

And if you thoughtfully read their FFF's: they didn't increased the map-transfer-speed! That is something, that can be done with the next version. v0.14 changed, how the player connect to the game and how it catches up to be in sync with the current game.

[HL65536]

Today I found out about this mod and could not believe that it actually worked:
https://mods.factorio.com/mods/orzelek/undecorator

Does this mean in normal worlds the map transfer consists of 50% useless information?
If this is true, what else could be done to reduce map size drastically?
It seems like there was no optimization done at all for map size.

I would suggest to shift from ups optimizations to map size optimizations for a while so big maps are even an option in multiplayer.
I heard Rseding91 likes optimization jobs - here is one!

[Rseding91]

Today I found out about this mod and could not believe that it actually worked:
https://mods.factorio.com/mods/orzelek/undecorator

Does this mean in normal worlds the map transfer consists of 50% useless information?
If this is true, what else could be done to reduce map size drastically?
It seems like there was no optimization done at all for map size.

I would suggest to shift from ups optimizations to map size optimizations for a while so big maps are even an option in multiplayer.
I heard Rseding91 likes optimization jobs - here is one!

In 0.13 decoratives generated at 4x the normal rate compared to 0.12 which caused them to have that huge impact on save size. In the recent 0.14 it was reduced back to 0.12 levels so new maps in 0.14 won't have that massive save file bloat.
For 0.15 decorative entities were changed to not be entities and now there's almost no save file size impact (a few KB on small maps maybe).

You may think decorative entities are "useless information" but that's what virtually all of the map size is. Tile data, trees, biters, decoratives, and a very very small part is your actual factory. Sure we could just dump everything except your factory but then the map looks like garbage and you might as well just be playing on a white background. At some point there's a tradeoff between save size and gameplay.

[<NO_NAME>]

short version:

map transfer now is not very smart:
in case of a new player connecting or desync: whole map is transferred
solution:
-client already has parts of map information (earlier save or desynced partially wrong info)
-identify correct info and transfer only what is needed

I don't really know how desync works but i suppose that it often affects only a single chunk or at most a few of them. That would mean that a desynced player have already almost all correct chunks. In that case, simple comparision of control sums would reduce upload time drastically.
Now, I am waiting to someone who will explain me, why this is not so easy

[Rseding91]

short version:

map transfer now is not very smart:
in case of a new player connecting or desync: whole map is transferred
solution:
-client already has parts of map information (earlier save or desynced partially wrong info)
-identify correct info and transfer only what is needed

I don't really know how desync works but i suppose that it often affects only a single chunk or at most a few of them. That would mean that a desynced player have already almost all correct chunks. In that case, simple comparision of control sums would reduce upload time drastically.
Now, I am waiting to someone who will explain me, why this is not so easy

Yeah that's not how desyncs work at all. The game has no idea what's different between the two games - it only knows the CRC checks ended up with different results which means some part of it is different.

[<NO_NAME>]

short version:

map transfer now is not very smart:
in case of a new player connecting or desync: whole map is transferred
solution:
-client already has parts of map information (earlier save or desynced partially wrong info)
-identify correct info and transfer only what is needed

I don't really know how desync works but i suppose that it often affects only a single chunk or at most a few of them. That would mean that a desynced player have already almost all correct chunks. In that case, simple comparision of control sums would reduce upload time drastically.
Now, I am waiting to someone who will explain me, why this is not so easy

Yeah that's not how desyncs work at all. The game has no idea what's different between the two games - it only knows the CRC checks ended up with different results which means some part of it is different.

OK, so at the moment of desync the game has no idea what is desynced. I've expected that much. But why can't it find out?
Let's consider the following algorithm:
1) Desync is detected.
2) Both server and desynced client calculate checksums for each chunk.
3) Server sends all its checksums to client.
4) Client compares its own checksums and the ones received from server to find out which chunks are corrupted.
5) Client requests server to resend only corrupted chunks.
Is there a reason that makes it impossible?

[Rseding91]

Yeah that's not how desyncs work at all. The game has no idea what's different between the two games - it only knows the CRC checks ended up with different results which means some part of it is different.

OK, so at the moment of desync the game has no idea what is desynced. I've expected that much. But why can't it find out?
Let's consider the following algorithm:
1) Desync is detected.
2) Both server and desynced client calculate checksums for each chunk.
3) Server sends all its checksums to client.
4) Client compares its own checksums and the ones received from server to find out which chunks are corrupted.
5) Client requests server to resend only corrupted chunks.
Is there a reason that makes it impossible?

The game is not saved in chunks. It's serialized from the beginning to the end of the data that needs to be saved. So, without the previous bit of data the next makes no sense in most cases.