Catching up the Factorio

[Slondo]

This topic might have already been created, but I’m posting it anyway in the hope of making some impact.

What would the intention be?
Joining a server would be reduced to the level of any other game, like Minecraft, where you simply connect without having to wait several minutes—depending on the host's connection—for map downloads and "catching up".

I believe the game does this literally to run the map locally, minimizing data exchange between users after the initial login phase.
However, it is precisely this stage that makes the process of connecting to a server tedious; there ought to be a way to speed up this
pre-loading and "catching up" phase, or even eliminate it entirely and simply receive the necessary data from the host for the player's context.
After all, the entire process is already on the host, and the host is the one coordinating the match.

Making something as simple as joining a Minecraft server possible would greatly enhance the appeal of Factorio's multiplayer; currently, no matter how small the map size is, you have to go through both processes repeatedly—even if you only lose your connection for a few seconds.

I don't know how this would work under the hood, or if something like this is even feasible, but considering the game's long-term outlook—with no expansion in sight and Wube moving on to other projects—revisiting this system and making this aspect less tedious would be a huge QoL improvement for all players and the game itself.

[eugenekay]

Feature Exists: Server owners can configure the auto_pause_when_players_connect option in server-settings.json. The entire Savegame must still be Downloaded, but the Server will wait while a new Client does so. The downside is that this pauses the game for everybody else on the server.

Good Luck!

[Slondo]

The intention would be to literally eliminate the download.

No pauses, no catching up, nothing. Just connect and play.

[mmmPI]

In factorio the game doesn't send you the map, it send the inputs of the players that occured during a game as far as i understand, and then your computer re-run the history of the game, so you achieve a synchronized state with all the other player, you could then literally disconnect a map and continue playing on your own the copy you were having.

In minecraft afaik, when you connect to a server you receive data on your immediate proximity, so it's very fast, and if you move around , you are getting more and more information from the server.

In factorio , pausing the game when someone is joining makes it so no more inputs are added when a player is already trying to re-run a map to "catch-up', so the amount of work to re-run the history of the map doesn't increase at the same time as it is processed by the computer of the person joining.

The two system are totally different and meant to solve different problems imo, it's not possible to send to a player only the relevant information regarding its immediate proximity in Factorio, the game wouldn't work if only half of the factory is loaded, you really need to have the full factory for even your immediate proximity to make sense, if the belt lane is very long you can't just "load the far part later", the close part would break whereas in minecraft there is little to no problem if you don't have one part of the map loaded in your computer and simulated when you are in a totally different area.

I'm not sure how it works for the things with which players make circuit contraption in minecraft, though i suspect they would break if only one part is loaded so maybe every part connected to the same "wire" is loaded at the same time which in factorio equivalent would mean loading "the whole factory".

[Slondo]

I see your point, but it still doesn't make sense to send the entire map and try to synchronize the game state, given that the host is already processing the entire factory in the background. Clients shouldn't need to do that work; they should simply receive information and send updates whenever changes are made.

Perhaps the game does exactly that due to the sheer volume of items and to maintain simulation accuracy—avoiding visual glitches, I imagine. Still, it would be an interesting area to explore for the long term.

As for the Minecraft example you mentioned: Redstone circuits extending beyond unloaded chunks don't work unless a chunk loader is explicitly built or the player is nearby.

[mmmPI]

I see your point, but it still doesn't make sense to send the entire map and try to synchronize the game state, given that the host is already processing the entire factory in the background. Clients shouldn't need to do that work; they should simply receive information and send updates whenever changes are made.

Perhaps the game does exactly that due to the sheer volume of items and to maintain simulation accuracy—avoiding visual glitches, I imagine. Still, it would be an interesting area to explore for the long term.

It's very well possible that my understanding is incorrect, or that my wording makes it worse , i don't know the specifics of the savefile , only general ideas. I think the "simply receive information whenever changes are made" is the idea for players inputs, that's what the server transmit to the clients over the course of the game after the initial loading and catch up is done. If the server was to transmit the changes related to the factory / to the map , that would be too much bandwith, if you only have a "piece of the factory" loaded, maybe there are 100 belts entering and leaving this part,and the server would need to tell the client that only sees a part of the factory everything about the items on it, everytime they move that would need to be send accross the internet, with any delay causing the belts to have visible "lag" of the items, maybe 1000 robots every second, the worst being that just through a single wire of the circuit network, you can send a lot of data, like hundreds or thousands of signals which all changes their value every tick, that represent more information than can be reasonnably transmitted via internet.

Currently , only the "player" lags in multiplayer, never a part of a factory relative to another part, those are always "synchronized", the same thing is calculated on every client, this way the result don't have to be exchanged. If there is a bit of delay over the internet, it appears that a player can interact with places where they haven't yet had time to walk, or the opposite, they are teleported back where they were a little earlier, at a corrected position or everything seem to happen super fast for a little while after a freeze but there is a guarantee "no delay" between "some belts" and "some others". It may sound wasteful , but as far as i understand minimizing the amount of data that need to be continously exchanged is the goal there, "player one pressed W during 5 second" is much lighther than a list of "item XX and item YY moved on belt XXX" for every item , every belt every thing.

Again disclaimer, so i may be wrong it's just my understanding, which may be incorrect and unclear about the precise steps of a player joining and loading but to me it make sense x)

As for the Minecraft example you mentioned: Redstone circuits extending beyond unloaded chunks don't work unless a chunk loader is explicitly built or the player is nearby.

Here again i may be wrong, i didn't know about chunk loader in minecraft and i was a bit too young and inexperimented to understand and remember everything i read about factorio like 10 years ago, but i imagine a factorio map is like a redstone circuit in the way it function, everything is (or can be ) connected, at least with electricity, which prompted the different choice of architecture for the whole game. It would be like trying to make something that doesn't need a chunk loader and work even if the player is not nearby. "you load all the chunks at the beginning" then "the whole factory is nearby" would be my analogy. Even if it could be done without loading time for this particular part with some clever algo to determine which parts is connected to which to only "pre-load" those, like different redstone build one the same map with their group of distinct chunk loader, active as needed, that one moment when you accidentaly connect several thing would cause a spike of data required per client. It's not sustainable for a live multiplayer compared to "initial loading screen and then no more" even though the later is only a dream when you have frequent deconnexions.

I think unfortunatly those kind of discussions have more of an impact at the very early stages of the development of a video game, or even "before", for factorio it would be the core of the engine , the foundations that are very hard to modify without tearing down the whole building too.

[Alfonse215]

I see your point, but it still doesn't make sense to send the entire map and try to synchronize the game state, given that the host is already processing the entire factory in the background. Clients shouldn't need to do that work; they should simply receive information and send updates whenever changes are made.

This all presumes that "clients" are a thing in Factorio. They're not. Well, not in the same way that clients exist in Minecraft.

Minecraft is a client/server multiplayer game system. In such a system, the server is God. It has the map. It does all of the processing. Clients receive whatever information they need to in order to display the part of the map that the player can immediately engage with. And clients send back player inputs. But that's all a client does: provide a UI to interact with the server. Only the server runs the game; the clients are just interacting with it.

When a client logs into a server, the server talks to the client, sending it the map and the current state of the world that the client needs to know. Each client talks to the server, and the server talks to its clients. But crucially, clients do not talk to each other directly.

Factorio is not like that.

At an architectural level, multiplayer Factorio is a peer-to-peer style of multiplayer. In such an architecture, all peers are the same. Every peer has a copy of the map. When a player provides input, that input is sent to every peer which independently processes that input in lock-step with the other peers. If the player provides input that causes a chunk to generate, every peer must generate that chunk in their copy of the map.

This means that every peer is running the game in its entirety. Every peer is processing every movement of every inserter. They're processing every tick of every circuit network. Etc. They are all running the same Factorio simultaneously. All of this processing must be done in lock-step with every other peer. This is the essence of peer-to-peer multiplayer.

Yes, multiplayer Factorio has a "server". But that server is really just another peer with a few perks. The server gets to be the one who defines what "the game" is. They have a save that is the game. When people connect to the "server", they're just connecting to that peer and all the peers that one is connected to. The "server" is just a peer who has to be the first one to the party and the last one to leave. They define when the party starts and when it ends, but they're still just another guest.

The server in Factorio lacks the authoritative control that servers in a client/server game do.

The "catching up" that new "clients" have to do is just the result of this kind of multiplayer architecture.

[Slondo]

Yes, multiplayer Factorio has a "server". But that server is really just another peer with a few perks. The server gets to be the one who defines what "the game" is. They have a save that is the game. When people connect to the "server", they're just connecting to that peer and all the peers that one is connected to. The "server" is just a peer who has to be the first one to the party and the last one to leave. They define when the party starts and when it ends, but they're still just another guest.

The server in Factorio lacks the authoritative control that servers in a client/server game do.

The "catching up" that new "clients" have to do is just the result of this kind of multiplayer architecture.

This really clarifies why Factorio's multiplayer works the way it does.

Sometimes, it is frustrating trying to play with other people over LAN due to these download and catching up steps.

Maybe, who knows, in the future they could revisit this and make improvements without explicitly mandating the download. It is indeed a major change, but I believe it would greatly improve the overall dynamic; the biggest barrier in multiplayer is precisely the frustration and the various potential errors associated with that step.

Feature Exists: Server owners can configure the auto_pause_when_players_connect option in server-settings.json. The entire Savegame must still be Downloaded, but the Server will wait while a new Client does so. The downside is that this pauses the game for everybody else on the server.

They might have tools to try and speed up the catch-up process, as eugenekay mentioned, but that is something requiring configuration, something a casual player shouldn't have to deal with.

[Loewchen]

When a client logs into a server, the server talks to the client, sending it the map and the current state of the world that the client needs to know. Each client talks to the server, and the server talks to its clients. But crucially, clients do not talk to each other directly.

Factorio is not like that.
[...]
The server in Factorio lacks the authoritative control that servers in a client/server game do.

Factorio is not P2P, at least not since 0.14 (and even before it was not really P2P). How the game networks and how the game syncs are independent, synchronization is done via deterministic lockstep (the save plus inputs thing) but the inputs are all sent to the server/host and only received from the server/host and therefore does have full authoritative control.

[mmmPI]

Here are some (technical)references, about the lockstep : FFF-76 MP inside out

This is what i attempted to explain for the pieces i remembered, sorry if i butchered it

about the MP rewrite :

FFF-147 Multiplayer rewrite
FFF-149 Deep down in multiplayer

This i didn't even attempted x)