Divorcing behavior from entities

[Drakomir]

Hello,

I really adore Factorio, and I love a modded Factorio even more (Bob, Angel, AAI, Deadlock <3). I wanted to start this topic for over year now, and after seeing the announcement trailer of Satisfactory I knew I just had to sit down and start writing.

When you place an entity (a prototype) in the world of Factorio, all its behavior is baked in. The entities that are currently available can been viewed here.
If you place inserter on the map, the way it behaves, how the UI reacts when the mouse hovers over it or click on it, is all baked in the entity's prototype.
When overriding/inheriting the prototype in your own mod for a custom variation of the inserter, you can adjust some properties like speed, artwork and stack-size, but not what you see in the UI when inspecting it with your mouse.

I think Factorio would become better moddable when the entity's behavior would be split up in components. To keep the example of the inserter, the components would be:

Code: Select all

- PhysicsComponent       ; Defines the physical dimensions of the entity.
- SelectionComponent     ; Defines the UI / interaction boundaries of the entity.
- InserterComponent      ; Defines the actual behavior of the entity.
- HealthComponent        ; Defines the an entity has health, how much and that it can be killed/destroyed.
- EnergyComponent        ; Defines whether this component consumes or produces (electric or chemical)-energy and how much of it. For example: the InserterComponent should aware / linked to one or more EnergyComponents added to the entity, so it can draw energy and operate in it's update cycle.
- LogicComponent         ; Defines logic/combinator capabilities of the entity.
- SpriteComponent        ; Defines the visual representation of the entity, if and what animations/shadows are displayed on what layer for example.
- UIComponent            ; Defines user-interface representation of the entity, what UI pops up when clicking on the entity and what UI is shown when hovering over the entity.
- AudioComponent         ; Defines the audio capabilities when the entity is operating.
- ContainerComponent     ; Defines the capability to store (and possibly filtered) items.
- ScriptComponent        ; Defines the capability to run a Lua-script using the entity as context, to support specific behavior or direct the other components.

When defining a new entity you also define which components it has and set the properties of that component. An entity is not constraint to have only one component of the same type, if you want to mod a hydra-inserter, add some additional Inserter-, Script- and Sprite-components
Components can be hierarchically structured when needed and changed at run-time (when a finished research adds new cabilities to existing entities for example).

This entity-component concept I am describing here is nothing new, it is widely adopted by all modern middleware game-engines. Ultimatily, by 'divorcing' the innate behavior of an entity into distinct components, the game's modding capabilities should be greatly expanded. It also allows for further performance enhancements and parallelization; the inserter-components's minimal data-structure can be memory-aligned and updated instead of dereferencing the entire entity in memory.
A mod-developer has little controle over how well an Vehicle-entity performs in great numbers, when the intent was only to create a 1x1-cell train system, but with this system the mod-developer only has to add and configure the components that it really needs to get the job done.
Obviously changing the entity-system is not a small endeavor, not something that can be squeezed in 0.17 release, but I am convinced that this modernizing change allows Factorio to grow to new heights and secure a place in the future, after all; it is the modding-support which yields such an amazing replayability.

Thanks for reading!

[Tim_412AO]

You've basically asked for the game engine to be rewritten from the ground up. While I applaud the thought put into software design, this cannot be done in the given engine. If you want to make your own game engine, go for it, but don't expect factorio to be able to do this.

[bobingabout]

They were trying to make some entity behaviour available on other entities, they made it so a furnace is basically just an assembling machine, but with an auto-selecting recipe. (with a few other minor name changes on variables, that is literally the only difference between the two), however, this kinda stopped there due to the difficulty of reconfiguring the engine. to do more would require too much of a re-write, which would only really benefit mod support.

There are a few other hidden (as in, not used in the base game) features of entities available to mods, such as being able to set recipe pollution modifiers, or the steam engine being able to be configured to "burn" fluids as fuel, instead of using their temperature. They're not undocumented. but these widespread universal shared features... well... very unlikely to happen.

[Rseding91]

This entity-component concept I am describing here is nothing new, it is widely adopted by all modern middleware game-engines.

I've yet to see any game out there use such a system and perform even moderately well.

From my 4 years experience working with Factorios C++ code base I'm convinced that such a system will never exist and perform the same or better than the static model that Factorio uses now.

The idea sounds nice but if you ever try to implement it you very quickly see how fast it falls apart performance wise due to the dynamic nature of every bit of logic every component has to do.

In summary: it's never going to happen in Factorio.

[kumpu]

Redoing all the entities is a bit much asked for but some kind of general purpose/highly customizable entity specifically for modding would open the door to so many possibilities. It would be so great and probably even give me joy in modding the game again.
And no one expects the same performance as vanilla from it.

[Rseding91]

Redoing all the entities is a bit much asked for but some kind of general purpose/highly customizable entity specifically for modding would open the door to so many possibilities. It would be so great and probably even give me joy in modding the game again.
And no one expects the same performance as vanilla from it.

I'm not against that concept but I've yet to see anyone write up how such an entity would work and haven't taken the time to try to write that up myself.

If someone did that it would go a long way towards me thinking about adding it to the game.

[eradicator]

I'm not against that concept but I've yet to see anyone write up how such an entity would work and haven't taken the time to try to write that up myself.

If someone did that it would go a long way towards me thinking about adding it to the game.

What technical level of writeup are we talking about here? Your perspective with detailed C performance analysis? Modders perspective with detailed "what would i do with it"?

[Rseding91]

I'm not against that concept but I've yet to see anyone write up how such an entity would work and haven't taken the time to try to write that up myself.

If someone did that it would go a long way towards me thinking about adding it to the game.

What technical level of writeup are we talking about here? Your perspective with detailed C performance analysis? Modders perspective with detailed "what would i do with it"?

That's what I don't know. All of that has to be figured out and then see if it can actually work to make such an entity where parts of it can just be shut off.

[kumpu]

I think from a modders perspective, Drakimor summed it up pretty well... We want modular control over the aspects the game already handles and fill in the remaining gaps with some lua.
An electricity consuming train wagon assembly machine that beeps when it's done? Sure thing.

Code: Select all

{
	type = "modding-entity",
	name = "foo-assembly-wagon",
	flags = {"foo"},

	modules =
	{
		--we want some wagon behaviour
		{
			type = "cargo-wagon",
			parent = "cargo-wagon", --inheritage for lazy people?
			passengers = 0,
			inventory_size = 0,
			...
			components = {"physics, graphics, placement"}, --which components this module defines
		},

		--and assembling machine behaviour
		{
			type = "assembling-machine",
			parent = "assembling-machine-2",

			components = {"UI, container, logic, energy"},

			lua_events =
			{
				on_production_finished = "function_that_beeps",
			}
		}
	}
}

So of course this is just a rough concept and I'd guess you would be quick to find out problems with it.
I can't give you more than my personal modding niche perspective as I'm not familiar with the codebase.

[davemcdave]

with how painlessly straightforward it was to get into modding through the wiki and gangsirs tutorial, where everthing just works, id hoped/imagined something like this would already be the case.
as i see it the suggestion asks for a prototype with the bare minimum of mandatory properties and as many as are feasble of the optional properties from other prototypes.
these optional properties probably default value null. but once defined they might come in bundles, ie when one has a value some others become mandatory

i cant say how this mixed entity would work without seeing how things currently work (which Rseding91 can see and replies "never going to happen").
but i can speculate:
on load the game makes a list of inserter types, if any modding-entitys have inserter properties defined, add them to that list, then in the update loop theres no "if this entity has inserter behaviour" check, that list of inserter types point to the list of mod-entity instances to be updated. similarly any mod-entity with crafting properties defined is added to the list of assembler types on load, then in crafting part of update
loop that says to update the crafting part of those entities. similarly for energy production and consumption, fluid and heat etc

energy_source
as a list with production_type = "input" or "output" and some priority for use in the machine itself. so a machine could prefer using its own solar and falls back on burner batteries at night. or a generator that first produces electricity then any unused is put out as heat.
solar power and the fluid use of steam engines would need to be framed as input energy sources. allowing to have generators turn any input type into any output type

next_direction
from tile prototype would be a powerful tool. pressing r while holding the item cycles the placable to the next entity. which isnt only usable for rotation, could cycle versions of the entity with different arrangements of fluid inputs, or for drills of different area with some compromise in speed.


like youve said, prolly isnt worth cost/benefit at this point. but is something to keep in mind for factorio2 or whatever game wube make next, to have some modder-entity baked in from the start with any new behaviours also being added to mod-entity

[Lubricus]

Some loose ideas that is probably not feasible.
The component model could fit better with data that is need together (components) stored directly in arrays.
Then you loop thru the arrays of data and the CPU cache should get an easier time... And the possibility to use SIMD instructions. Think more like vector operations on different properties of entities than loping thru entities and handle all the logic at the game object.

What I understand. Factorio now works with arrays of pointers to the data for entities. That makes the data for loops spread across the RAM and harder for the cache to predict what will be used next.

I have no idea how to implement something like that in an not to spaghetti object oriented code, and i want 0.17 now not in about 10 more years.

[Rseding91]

Some loose ideas that is probably not feasible.
The component model could fit better with data that is need together (components) stored directly in arrays.
Then you loop thru the arrays of data and the CPU cache should get an easier time... And the possibility to use SIMD instructions. Think more like vector operations on different properties of entities than loping thru entities and handle all the logic at the game object.

What I understand. Factorio now works with arrays of pointers to the data for entities. That makes the data for loops spread across the RAM and harder for the cache to predict what will be used next.

I have no idea how to implement something like that in an not to spaghetti object oriented code, and i want 0.17 now not in about 10 more years.

Show me a game that does what Factorio does as performant as Factorio is with anything you describe here and then I'll start to take these kinds of conversations seriously. Until then, all I've ever seen is hobby examples and poorly done slow games that claim to use any kind of entity component system.

[posila]

Show me a game that does what Factorio does as performant as Factorio is with anything you describe here and then I'll start to take these kinds of conversations seriously. Until then, all I've ever seen is hobby examples and poorly done slow games that claim to use any kind of entity component system.

That's a little bit harsh response, imho.
I am big believer in data-oriented design of game engines, and am convinced that there is lot of inefficiency caused by our OOP design. Even during rendering rewrite I saw how much difference small things can make - like doing an operation on range of structures/objects instead of doing the operation on one object at a time. Some of the optimizations that we are already doing (like replacing entities with specialized non-entity types, ...) are abandoning original purely OOP design and head toward something more data-oriented.

Anyway, any significant change to the entity system simply won't happen any time soon.

CppCon 2014: Mike Acton "Data-Oriented Design and C++"

EDIT: I am not talking about entiy component system specifically here, I have no idea if it would be good fit for Factorio or not, as it is hard to imagine current entities transfering to component system, because I am so used to how they work now.

[Rseding91]

Show me a game that does what Factorio does as performant as Factorio is with anything you describe here and then I'll start to take these kinds of conversations seriously. Until then, all I've ever seen is hobby examples and poorly done slow games that claim to use any kind of entity component system.

That's a little bit harsh response, imho.
I am big believer in data-oriented design of game engines, and am convinced that there is lot of inefficiency caused by our OOP design. Even during rendering rewrite I saw how much difference small things can make - like doing an operation on range of structures/objects instead of doing the operation on one object at a time. Some of the optimizations that we are already doing (like replacing entities with specialized non-entity types, ...) are abandoning original purely OOP design and head toward something more data-oriented.

Anyway, any significant change to the entity system simply won't happen any time soon.

CppCon 2014: Mike Acton "Data-Oriented Design and C++"

EDIT: I am not talking about entiy component system specifically here, I have no idea if it would be good fit for Factorio or not, as it is hard to imagine current entities transfering to component system, because I am so used to how they work now.

I should have worded my original message better. I was specifically talking about entity component systems. "Data driven" is not something I see as a concept but more of just "optimize your code". Less function calls, less virtual dispatch, smaller data sets and so on. That's all standard for making anything run faster.

Entity component systems are what I have a problem with and I believe that's what this original post is about: "Divorcing behavior from entities". AKA: making entity behavior components (an entity component system). I don't ever see that working as well as the current system(s) we have in place.

[bobingabout]

You do actually have a few component elements where appropriate. Most of the non-mobile power consuming entities just have an energy_source= tag, and you can specify which type they use. Boilers for example can run from heat, burner or electric. and 0.17 introduces fluid and void, both of which also work on the boiler, as well as the array being extended to some other entities, like the Radar.

There are plenty of other examples of such things that can be added in such a fashion, but the main issue there is, it would reduce performance for... from the base game perspective, no gain.
There is a bit of a trade-off, some things with a small theoretical performance hit have been implemented for mod support, but this will likely be more than just a small hit.

[Rseding91]

You do actually have a few component elements where appropriate. Most of the non-mobile power consuming entities just have an energy_source= tag, and you can specify which type they use. Boilers for example can run from heat, burner or electric. and 0.17 introduces fluid and void, both of which also work on the boiler, as well as the array being extended to some other entities, like the Radar.

There are plenty of other examples of such things that can be added in such a fashion, but the main issue there is, it would reduce performance for... from the base game perspective, no gain.
There is a bit of a trade-off, some things with a small theoretical performance hit have been implemented for mod support, but this will likely be more than just a small hit.

Those still don't divorce the behavior from the entity. Each entity that supports these dynamic energy sources has special logic in them to handle each type.

[H8UL]

I think the idea of hierarchical components that can be swapped at runtime, is indeed far too heavyweight. These sorts of component based approaches have lost a lot of popularity even in enterprise software (EJB, CORBA, SOAP, OSGI) -- a world notorious for over-engineering.

It's also nowhere near as fun to program these monsters.

Practically what gets you the important benefits is good old object composition and just a little inheritance. So wisely picking has-a relationships and is-a relationships.

You see the flexibility this gets you with projectile system. Projectiles have actions of type Trigger, which are (at least as far as modders are concerned) polymorphic. You can nest them. You can create entities with them. And since most things are entities including projectiles themselves, that gets you a lot of possibilities. This is object composition with good choice of inheritance.

So yeah, more of that approach will always be welcome. Only cases I can think of are quite specific. An example is more flexible achievements. I'd like more mods to add custom achievements. So being able to compose triggers/measurements/counters in some way would be really cool. But I can't think of a more fundamental example.

[Aidiakapi]

"Data driven" is not something I see as a concept but more of just "optimize your code". Less function calls, less virtual dispatch, smaller data sets and so on. That's all standard for making anything run faster.

I don't want to be presumptuous, but I think you're just running into the conflicting terminology surrounding ECS.
When there is talk of Entity Component Systems, you usually end up with two very different and very conflicting concepts. I'll try to describe both briefly, with a very simple example of having a "player" that just keeps moving in a direction.

1. The old-school Unity/UE4 approach to EC-S

The idea here is very similar to Object Oriented programming, except it heavily favors composition over inheritance. In pseudocode, it looks a bit like this:

Code: Select all

struct Entity // in Unity: GameObject, in UE4: Actor
{
    std::vector<Component> components;
    void update(float delta_time); // updates all components
    void render(RenderContext& ctx); // renders all components
};

struct Component // abstract, in Unity: MonoBehaviour, in UE4: ActorComponent
{
    virtual void update(float delta_time);
    virtual void render(RenderContext& ctx);
};

Components and some gameplay logic in a system like that might look like this:

Code: Select all

struct Transform : Component
{
    float x;
    float y;
    float angle;

    Matrix3x3 compute_matrix();
};

struct PlayerMovement : Component
{
    // How these are initialized varies heavily on a per-engine basis
    // for this example, I assume this is set via dependency injection
    // between the PlayerMovement constructor running, and the first
    // call to update.
    Transform* transform;
    Mesh* mesh;

    float player_speed = 3.f; // m/s
    float direction = 0; // radians

    void update(float delta_time) override
    {
        transform->x += std::cos(direction) * player_speed * delta_time;
        transform->y += std::sin(direction) * player_speed * delta_time;
    }

    void render(RenderContext& ctx) override
    {
        ctx.queue(mesh, transform->compute_matrix());
    }
};

Whilst such a system can be quite nice for rapid prototyping, it often has terrible performance implications. Every call is a kind of virtual dispatch, every component is often in a different place in memory causing a cache miss. Often times (though not necessarily, Unity doesn't do this) the Entity updates all the components, consequently, you're constantly updating different components, and constantly running different code, completely thrashing your instruction caches.

On top of this, because of the tight coupling between siblings, lifetime management of components can become quite tricky. Both Unity and UE4 use Garbage Collection (and all the problems that come along) to make this manageable.

2. Pure Entity Component System.

The previous technique is kind of an "Entity & Components" system, whereas this is "Entities, Components and Systems". It's a technique that builds the concepts of Data Oriented Design, into a combination of parts that happen to play very nicely together, resulting in a setup that's easier to maintain and has excellent performance. It comes with the downside of having a large up-front cost in terms of complexity, and learning curve. It is very different from the programming most people are used to.

The three primary parts are of course Entities, Components and Systems, which I'll describe, because there's really no associated code:

• Entity: Usually just an integer, a handle, an index into an array. A way to talk about one "logical unit" in your game.
• Component: Just some blittable data. Ints, floats, bools, arrays, matrices, vectors. Sometimes also pointers.
• System: A piece of code, which reads from certain components, and writes to others, and will run for every entity that matches some "selector" (or "group").

Writing the example in a system like this, would look something like:

Code: Select all

// Components
struct Position
{
    float x;
    float y;
};

struct Rotation
{
    float angle;
};

struct MoveData
{
    float speed; // m/s
    float direction; // radians
};

struct SharedMeshFilter
{
    Mesh* mesh;
};

// Systems
struct MoveSystem
{
    // This will function as our "selector", for determining which objects this system will run on
    struct Group
    {
        std::size_t length;
        ReadOnlyComponentArray<MoveData> move_data;
        ReadWriteComponentArray<Position> positions;
    };

    // How this gets passed in heavily depends on the particular implementation
    // but assume for now that it's dependency injection.
    float delta_time;

    void run(Group g)
    {
        for (std::size_t i = 0; i < g.length; ++i)
        {
            auto& md = g.move_data[i]; // MoveData const&
            auto& p  = g.positions[i]; // Position&

            p.x += std::cos(md.direction) * md.speed * delta_time;
            p.y += std::sin(md.direction) * md.speed * delta_time;
        }
    }
};

struct RenderSystem
{
    struct GroupWithoutRotation
    {
        std::size_t length;
        ReadOnlyComponentArray<Position> positions;
        ReadOnlyComponentArray<SharedMeshFilter> meshes;

        // This indicates that this group won't match any entities that have a Rotation component
        NoComponent<Rotation>;
    };

    struct GroupWithRotation
    {
        std::size_t length;
        ReadOnlyComponentArray<Position> positions;
        ReadOnlyComponentArray<Rotation> rotations;
        ReadOnlyComponentArray<SharedMeshFilter> meshes;
    };

    RenderContext* render_ctx;

    void run(GroupWithoutRotation g)
    {
        for (std::size_t i = 0; i < g.length; ++i)
        {
            auto& p = g.positions[i]; // Position const&
            auto m = g.meshes[i].mesh; // Mesh*

            // Calculate transform from position
            Matrix3x3 transform;

            render_ctx->queue(m, transform);
        }
    }
    void run(GroupWithRotation g)
    {
        for (std::size_t i = 0; i < g.length; ++i)
        {
            auto& p = g.positions[i]; // Position const&
            auto& r = g.rotations[i]; // Rotation const&
            auto m = g.meshes[i].mesh; // Mesh*

            // Calculate transform from position and rotation
            Matrix3x3 transform;

            render_ctx->queue(m, transform);
        }
    }
};

The main takeaways are:

• Your data is completely separate from any logic. It's just plain old data, no constructors, destructors, or anything fancy. Just allocate a block of memory and go.
• All your logic is contained in your systems, which specify what their inputs and outputs are.
• You use a group or selector to run your code on specific entities that have specific conditions, allowing you to do the minimal work, and often avoid branching inside loops.
• Making code multithreaded is usually a lot easier, because you already define the data flow between your systems. For example, if you had many players, you could update their positions all in parallel, and because you know that they're only going to read the move data, and write to positions, anything that doesn't depend on either of those can run in simultaneously.
• It actually becomes a lot easier to edit code, because all gameplay logic is separated, and if you want to change a system, you only ever need to change its own logic, and occasionally the system that runs before it, or after it.

The full approach is way too complicated to summarize in a somewhat compact manner, and there's many details I omit. ECS's build on the fundamentals of Data Oriented Design, and just take it a step further, making DOD something that is easily usable for gameplay code.

Some resources

• A talk by Mike Acton, about Unity's new ECS, which dives quite in-depth on a practical example.
• A blog post that looks at some challenges for data memory layout for implementing ECS's.
• Someone implemented an ECS with pure C++ metaprogramming. I wouldn't ever do that in production, because of compile times, but it shows quite a lot of interesting techniques, and goes into a lot of code that can help with understanding.
• Not that informative, but a very impressive demo, showcasing the potential performance.

[Rseding91]

I'm sorry but after reading that you've done nothing but further convince me that entity component systems are nothing but a bunch of smoke and mirrors destined to only appear in internet discussions and demo examples.

What you've described would be *terrible* for any game that has any amount of entity <> entity logic. AKA: your game actually does anything beyond render some cloud of particles.

Especially so in Factorio where virtually everything has some state that it's transforming each time it's updated and that state is effected by other states and in turn can modify those states effecting the next entity to update (power, if something is in a chest/assembling machine/furnace), if the circuit network signals are correct, if the logistic network signals are correct, and so on.) Everything touches everything else.

There's no "update the positions of all biters that are moving" loop in the game that so many ECS examples seem to love showing. There is a *per biter* section of logic that checks if the biter should keep running in the direction it was trying to go, if it can move in that direction or if it ran into something, if it should slow down so it keeps pace with the rest of the biters it's running with, if it should turn now because the path it's following turned, if it has failed to move too many times and should re-path and so on.

I still have not seen one game that uses an ECS do anything near what Factorio is doing. Until that happens every article, talk, or forum post read about ECSs does nothing but bury the idea deeper in the grave for me.

[Aidiakapi]

I'm sorry but after reading that you've done nothing but further convince me that entity component systems are nothing but a bunch of smoke and mirrors destined to only appear in internet discussions and demo examples.

Well, considering it's being used by most high performance AAA games, I don't think it's fair to call it either smoke and mirrors, or just demo examples.
Some studios that have spoken publicly (that I know of) about their usage of it include Naughty Dog, Blizzard and Insomniac Games.

What you've described would be *terrible* for any game that has any amount of entity <> entity logic. AKA: your game actually does anything beyond render some cloud of particles.

Yet that scenario is precisely what it excels at. Mike Acton's talk (first one I linked) specifically goes over the case for an n-body simulation, where every entity interacts with any other approximate entity. It's the worst case scenario for performance you can have.

Regardless, I'm not here to convince you, it's a tool, it's seen effective use for about a decade, it's becoming more prominent, and more important now that multithreaded CPUs are really kicking it up. I would suggest researching the topic a bit more before closing the book on it, but I think I've provided enough resources to look at at this point.