Multiplayer Map Save Idea

[Yezarul]

I know this won't effect many people but it is a quality of life for the better for everyone.

Currently when you connect to a multiplayer map it downloads the entire save *.zip file and then "catchs" you up to whoever is hosting.
This can be bothersome when you have a slow connection, especially if you have horrible upload.

My smallest save is 7mb with my normal games being 15-20mb. I have one game at 50mb.
My upload speed is 576 kbs. To upload that 50mb file to one person takes over 20 minutes, forget trying to upload it to multiple people, effectively stopping me from ever hosting the games that I WANT to play with others after having gotten far with them.

My download is 3.5 so downloading the 50mb file from someone still takes a few minutes (but not near as long as uploading to them).

So here's the idea/suggestion:
a checksum style for multiplayer games.

Multiplayer games (ie read online) would have to be saved (think autosave). Whether it remembers the savefile name of the one hosting or the multiplayer ID.
Save file would have say 90/95% of the game like normal but the checksum would kind of work like the current "catch up" is.

So
A) You have a multiplayer game you and a buddy played called Bob's Mods Emporium.
B) You two play for 24 hours on it and both save.
C) Buddy host but since he saved 5 seconds later than you there is a slight save descrepancy, the game would load Bob's Mods Emporium and download whatever rest is missing (catch up)

A) You have a multiplayer game you and a buddy played called Bob's Mods Emporium.
B) You two play for 24 hours on it and both save.
C) You host, but your buddy lost his save file (new computer or whatever) he has to download the full map.


I believe this should be possible (but I don't know the technicalities of the programming to know for sure).
This would allow for more people to host/share with friends (I wouldn't mind uploading the game to a buddy or two one time and then being able to host anytime after that, or sending the save file during off-time so they could move it).
It would allow for less taxation of servers hosting speeds/stability.
Re-connecting on DC wouldn't force you to wait near as long.

[ssilk]

I dunno exactly what to answer. This is a valid request and makes sense and is more or less possible.
But I tend to say, that it is much easier to have a faster internet-connection instead of having 1-2 developers spend 3-6 month (plus added complexity and bugs).

[Rseding91]

This wouldn't work. The save file differences after 5 seconds would mean you download the entire file anyway.

[Yezarul]

I dunno exactly what to answer. This is a valid request and makes sense and is more or less possible.
But I tend to say, that it is much easier to have a faster internet-connection instead of having 1-2 developers spend 3-6 month (plus added complexity and bugs).

I agree with you. Faster internet is usually the easier suggestion, but myself? I am on an island and literally the fastest I can get is 3.5 mb down and .5 mb up.

This wouldn't work. The save file differences after 5 seconds would mean you download the entire file anyway.

Gotcha.
Suppose it would be more effort than it's worth to re-do the save file to be able to make this happen.
Thanks for the response.

Edit:
What about making the save file smaller?

[BenSeidel]

What about just making the large, unchanging part of the save file "saveable".
I remember reading somewhere that Rseding said that the largest part of the save file (or was it memory?) was the generated terrain. If you could download the terrain once (or generate it locally) then just apply the things that could change, like the entities, and be done.

The terrain generation could occur in the opposite order of the terrain download, so when they meet in the middle you are done. Yes there are issues, such as when the terrain changes due to scripts or landfill, but a relatively simple hash/checksum should see that no chunk is not updated when changed.

The system could be as simple as
Connect to server, get a map "id" (generation seed).
in parallel:
{
Check saves for identical seed.
Open save, load terrain data.
transmit a chunk generation bitmap and checksum.
}and
{
start download of "factory" data (items, bots, biters, players etc.)
}
server receives terrain bitmap and saves & compresses new chunk data (if it hasn't done already, but in such a way that you could "skip" marked chunks).


Anyway, it would probably be too difficult to implement now (when compared to the benefits for the small number of players that it would affect), but it is theoretically possible. For those 1%ers, it would make the world of difference. I could even see it being useful when playing on a lan with a 50+mb file, for that 10 seconds it takes to transfer the file that is

[Rseding91]

What about just making the large, unchanging part of the save file "saveable".
I remember reading somewhere that Rseding said that the largest part of the save file (or was it memory?) was the generated terrain. If you could download the terrain once (or generate it locally) then just apply the things that could change, like the entities, and be done.

The terrain generation could occur in the opposite order of the terrain download, so when they meet in the middle you are done. Yes there are issues, such as when the terrain changes due to scripts or landfill, but a relatively simple hash/checksum should see that no chunk is not updated when changed.

The system could be as simple as
Connect to server, get a map "id" (generation seed).
in parallel:
{
Check saves for identical seed.
Open save, load terrain data.
transmit a chunk generation bitmap and checksum.
}and
{
start download of "factory" data (items, bots, biters, players etc.)
}
server receives terrain bitmap and saves & compresses new chunk data (if it hasn't done already, but in such a way that you could "skip" marked chunks).


Anyway, it would probably be too difficult to implement now (when compared to the benefits for the small number of players that it would affect), but it is theoretically possible. For those 1%ers, it would make the world of difference. I could even see it being useful when playing on a lan with a 50+mb file, for that 10 seconds it takes to transfer the file that is

Generating terrain is slower than map download. It's one of the slower things in the game.

[ssilk]

Optimize that!


I had games, where the map generator was to slow to generate land in front of me while driving with the train-layer.

[Rseding91]

Optimize that!


I had games, where the map generator was to slow to generate land in front of me while driving with the train-layer.

We already did https://www.factorio.com/blog/post/fff-112

[BenSeidel]

Generating terrain is slower than map download. It's one of the slower things in the game.

So why can I run the commands
/c game.forces.player.chart(game.player.surface, {lefttop = {x = -1024, y = -1024}, rightbottom = {x = 1024, y = 1024}})
/c game.speed=100
faster than I can download a 60mb file?

Just imagine how fast it would be if the terrain generation was multi-threaded?

[Rseding91]

Generating terrain is slower than map download. It's one of the slower things in the game.

So why can I run the commands
/c game.forces.player.chart(game.player.surface, {lefttop = {x = -1024, y = -1024}, rightbottom = {x = 1024, y = 1024}})
/c game.speed=100
faster than I can download a 60mb file?

Just imagine how fast it would be if the terrain generation was multi-threaded?

Because a 2000x2000 map is not 60 MB. a 2000x2000 map is roughly 8 MB including all of the entities.

Also chunk generation is threaded. It's not multi-threaded because tile generation still can't be done in multiple threads as the result of one chunks tiles changes the result of the next and entities can't be generated before all the neighboring chunks for the chunk to generate entities have been generated to corrected tiles.

[Nexela]

Also chunk generation is threaded. It's not multi-threaded because tile generation still can't be done in multiple threads as the result of one chunks tiles changes the result of the next and entities can't be generated before all the neighboring chunks for the chunk to generate entities have been generated to corrected tiles.

I must be tired.... I got halfway through this paragraph before I started thinking of woodchucks.