Multithreading by surface

[AivanF]

TL;DR

Multithreading splitting by surfaces

What?

As a software engineer myself, I understand why multithreading for game logic is not a part of Factorio for years: determinism model is good for multiplayer games with large files and lots of activities as it doesn't require constant sending of the whole state, but bad for parallelism as requires complicated algorithms to separate entities' calculations and many mutexes/checks to maintain consistency.

However, I think, separate surfaces are quite segregated in terms of calculations, and hence be relatively easily processed separately in different threads (similar to different zones in MMO RPGs).

Of course, this approach will require centralised access/handling of global variables and actions (like technological researches), but I suppose it's solvable and we still will get a performance boost. And this will break API compatibility, but I believe this is a fine price for a such breakthrough; especially if new major version do this anyway.

Moreover, in multiplayer games, it seems possible and cool to allow client game instances process only the surface where a player's character located on.

This idea can be extended into handling different surfaces (or groups of them) by different servers, allowing to have even greater factory worlds...

Why?

We have a couple of premises:
1. Some popular mods, where a single save file can contain lots of surfaces with large factories connected by relatively transfers of players and cargo.
2. In the coming brand new Factorio 2 Space Age similar mechanics will be a default part of the game.

And some desired sequences:
1. This will make large multi-surface single-player Factorio saves less laggy on most modern computers as they typically have 4-8 cores.
2. This will allow to have a dedicated powerful server with dozens of CPU cores to handle a large saves with many surfaces, meanwhile players with ordinary computers will be able to connect and play together, travelling and factorying across the same world. Kind of MMO!

PS. Some other players asked me to use my programming knowledge and think about a solution for the lack of multi-threading problem. Personally, I'd much more prefer to see better network performance of vehicles and fighting/shooting, as currently lags are make them unusable in multiplayer games... From a game design theory, I'm sure this would attract a broad audience providing opportunities for mental types of gamers to play together.

[Rseding91]

I recommend reading this viewtopic.php?p=563555#p563555

[boskid]

Moreover, in multiplayer games, it seems possible and cool to allow client game instances process only the surface where a player's character located on.

This message alone shows you have no idea about our architecture of multiplayer so you have no idea what you are talking about. Its not that we are simulating all surfaces because we want to keep CPUs warm. We simulate all surfaces because that is part of keeping all clients aware of entire game state so we do not need to transfer anything more than input actions after a player has already joined.

I have no will to describe all the shared states that are between surfaces at this point, but i will just list some of them: research manager, item number allocators, entity unit number allocators, lua events, item production/consumption statistics, shared state due to linked belts, shared state due to linked containers.

[AivanF]

We simulate all surfaces because that is part of keeping all clients aware of entire game state so we do not need to transfer anything more than input actions after a player has already joined.

Nah, I know. But look: clients don't need to be aware of surfaces they don't actually deal with. If one thinks the opposite, they look from point of view of the past implementation, not of the future one.

i will just list some of them: research manager, item number allocators, entity unit number allocators, lua events, item production/consumption statistics, shared state due to linked belts, shared state due to linked containers

Meanwhile, there are lots of high-loaded distributed data storing and processing systems and approaches (like a basic DB sharding), even realtime ones. Everything is solvable once it get desired! Lua states could belong to different processing groups, which can have non-overlaping ID counters for all entities you mentioned. The same about statistics, which can be calculated for different surfaces and summed for teams once a second. Linked entities are a bit more complicated, but usually they are created together with a new surface linked to an existing one (like Factorissimo2), so can be created in the same processing group, so that their surfaces are handled together. And the processing groups could have a special communication protocol, to share some information and perform common actions.

Yes, this is truly a lot, so I don't ask nor require, of course, to do what I tell, especially considering existing of some private business plans, or simply because it's probably more hard than profitable. I also understand that you may be really tired of hearing and explaining similar topic every year or maybe even every month. But it's hopefully a new, more realistic idea to consider, taking into account newcoming increasing usage of surfaces. Well, it's just a suggestion for a discussion. Not for imperative telling me, previously a big fan of Factorio, what I understand or not.

[boskid]

Due to amount of state variables we cannot simply "upload missing state" to a client when it decides to switch to a different surface. If a client has some pieces of state missing it simply cannot do anything, lua cannot run because it may be touching any surfaces at any point in time and having game frozen "because lua requested something that is not locally and we must get this or we cannot proceed with update due to determinism" is just non real. Because of amount of state variables we have an architecture where all players must know all state so we do not depend on state upload after a player is already connected. It is not something we are going to give up on as for us it makes MP possible at all. If you could guarantee all players have 10Gbit connection to a server in all cases and latency is guaranteed to be under 0.01ms then maybe we could do something but in a real world scenarios we have to deal with slow network connections, latency significantly larger than 2 tick and your proposal around not updating surfaces when client does not look there is just hopeless. Closing.

[mrudat]

TL/DR

To efficiently use multiple-core processors in Factorio 2.0, where we expect to have multiple large surfaces, it might be a performance win to split the save across a local cluster of Factorio servers managed via the GUI.

What?

Something like a built-in clusterio cluster, launched by the GUI, with each server holding one large surface or many small surfaces, each running in its own thread/process, and only occasionally interacting by moving batches of items or players between surfaces, possibly not running in lockstep if coupling between surfaces is sufficiently loose.

Why?

I believe a set of Factorio servers can achieve more total UPS than is currently possible with a somewhat parallelised Factorio.

[Silari]

Already suggested many times and can't be done since things can interact between surfaces - scripts can interact with anything anywhere at any time, and entities can have cross-surface connections.

Ref: viewtopic.php?f=71&t=111062

[Koub]

[Koub] Merged into the older thread with the same suggestion.

[boskid]

Please note this topic derailed from "multithreading by surface" to something about having partial game state on the clients which is not going to happen. Having full game state on the client is one of the fundamentals that allows us to have the game at all (with MP support) so any suggestions about not having full game state on a client is just going to be rejected by default. Do not reopen this topic.