Modding API expansion for circuit network behavior.

[alercah]

TL;DR

Allow mods to exercise significantly more control over the circuit network, effectively allowing complete control over how entities and the circuit network interact.

What ?

Three major changes to the modding API and featureset of the game, which do not add a lot of overall complexity to the engine or core game, but expose a lot more of it to modders. Each change could be implemented independently, but I believe all three are needed to truly unlock maximum potential for modders. I've included a sample API spec (not necessarily complete) alongside each feature to enable more precise discussion, but I don't expect the final API would necessarily conform to this.

1. Mods would be allowed to define custom wire types, to allow adding more overlapping circuit networks to the game. E.g. blue wires, purple wires, etc. A mod could define a wire type for internal use between entities (especially hidden entities) it creates, to allow a mod to essentially create private circuit networks.
   
   Sample specification

   Allow wire prototypes in the data table:
   

   Code: Select all

   {
       type="wire",
       name="blue-wire", 
       color={0,0,1},
       network="circuit", # or "electric" for electric cables
   }
   

   
   This would initialize a LuaCircuitWirePrototype object into the game prototypes. The existing wire_id type, used to select wires, is replaced with a CircuitWirePrototypeSpecification, which could be either a prototype object or a string for its name.
   
   Item prototypes would have an additional wire_type field to explicitly select which kind of wire they relate to.
   
   Entity prototypes would have the draw_circuit_wires and draw_copper_wires fields replaced with a draw_wires field, which is a string -> string -> bool table: the first string is the network and the second is the wire name. "*" can be used as a wildcard; if it's used in the first field, then it points to a bool directly. The default would therefore be { "*" = true } (note that the rest of this proposal modifies this further; see the next section). Similar changes would be made to consolidate maximum_wire_distance and circuit_wire_max_distance into a single wire_max_distance field with type string -> string -> double.
   
   I don't specifically have reason to propose custom electric wires, but that could be an option too. For instance, heavy-duty cables might allow even longer spans between power poles, or possibly other traits could be added to cables like attenuation over distance or maximum throughput. These would be major features in their own right, however, so for now I propose that custom electric wires either be forbidden altogether (limit the number of electric wire types to 1) or all have the same properties (and thus only be useful for module-private wire types).
2. Mods can directly add, modify, or remove connection points on entity prototypes. This would allow the mod to link entities to the network that otherwise wouldn't be able to.
   
   Sample specification

   The entity prototype definition in `data` changes significantly to accommodate this.
   
   The draw_wires and wire_max_distance fields would be created per the previous section, even if custom circuit types are left out (in which case, rather than strings, they would use wire_type as the key rather than two strings. The draw_wires field, however, would not use boolean values, but values of a newly-defined type wire_connection_destination which has values none, internal, external, and all---internal connections refer to connections between the same entity, and external connections to connections to other entities.
   
   Additionally, a player_connectable_wires field is added which has the same string -> string -> wire_connection_destination type as draw_wires does. This specifies which kinds of connections can be manipulated by the player, rather than only by script.
   
   The existing fields in prototypes in the data table used for connection positioning (including circuit_wire_connection_points, circuit_wire_connection_point, left_wire_connection_point, right_wire_connection_point, input_wire_connection_point, output_wire_connection_point, and connection_points) are all replaced with a single connection_points field applicable to all entities (not just poles, combinators, etc.). This is a string -> table table, with the string being the name of the connection point, and the elements having the following fields:
   

   • wire_types: a field specifying the wire types that can be connected to this point, in the same format as draw_wires above. (Strictly, this field isn't necessary and player_connectable_wires plus careful coding will work, but this can prevent unwanted connections from arising due to interaction of mods or coding mistake.)
   • draw_wires, wire_max_distance, player_connectable_wires: overrides the corresponding value of the entity for this connection point only. Only wire types mentioned (or covered by wildcards) in these fields have their values overridden.
   • bounding_box: a bounding box for the connection, needed to let the player select between multiple. If not present, the entity's bounding box is used. If the player is drawing with a wire type which cannot connect to this point, then the bounding box is ignored (so two connections for different wire types can overlap safely).
   • position: Either an array or an array of arrays. Either way, the content is a table with keys "wire" and "shadow" to vector of four elements, specifying the wire positions. When there are multiple positions, the engine distribute the drawable wires for this connection point between the points during prototype loading, permanently assigning them to one or the other (this is not preserved across saves, however it is deterministic). This field is optional if there are no drawable wire types.

   Example for small electric pole, where the electric wires all go along the center and circuit wires are evenly distributed along the sides.

   Code: Select all

   connection_points =
   {
     {
       name = "electric",
       wire_types = { electric = "*" },
       position =
       {
         {
             wire = util.by_pixel_hr(0, -246.0),
             shadow = util.by_pixel_hr(245.0, -34.0)
          },
          # etc.
       },
     }
     {
       name = "circuit"
       wire_types = { "circuit" = "*"},
       position =
       {
         { # existing red wire connection
           {
             wire = util.by_pixel_hr(41.0, -183.0),
             shadow = util.by_pixel_hr(284.0, 28.0)
           }, 
           # etc.
         },
         { # existing green wire connection
           {
             wire = util.by_pixel_hr(-40.0, -240.0),
             shadow = util.by_pixel_hr(204.0, -31.0)
           },
           # etc.
         }
       }
     }
   }
   

   The wire type prototype, if the previous feature is included, has an additional field default-connectable. It's a boolean defaulting to true that, if false, excludes the wire types from wildcards in the wire_types field, so that it can only be connected to specially designated
   
   
   Appropriate LuaConnectionPointPrototype and LuaConnectionPoint classes would be added, and the existing wire_connection_id and circuit_connector_id types would be replaced with a ConnectionPointSpecification which could be one of them or the point's name.
   
   Hardcoded control behaviors (which is all there is unless the last feature is included) look up circuits from hardedcoded string names, available as defines. It's an error if an entity lacks the necessary connections for its control behavior.
3. Mods can specify the control behaviors of entities directly, and set some parameters. Existing control behaviors are made more granular so that this is more useful. Additionally, a new control behavior is added allowing a mod to control behavior directly.
   
   Sample specification

   The entity prototype in the data table is given a new field control_behaviours. It is an array of tables whose type field is a control_behavior.type applicable to this entity, as well as a description field which is displayed in the entity's control box where the user gets to control it. The rest of the table is parameters to the control behavior, including the connection points it's associated with.
   
   The built-in control behaviors are reorganized as follows. For each connection type parameter, a single connection point can be used or, for inputs, a magic constant is defined that connects it to the logistic network instead (existing behaviors that can read from the logistic network are split in two into one reading from the logistic network and one reading from the circuit network). If not specified, every behavior has a single connection parameter. Additionally, all behaviours that are run-time configurable have a operable parameter defaulting to true.
   

   1. accumulator: Renamed to read_charge. Applicable to any entity with an electric energy buffer. Reads the percentage that the buffer is full to a specified connection.
   2. constant_combinator: Unchanged except for the addition of a connection parameter.
   3. generic_on_off: Renamed to set_on_off and toggles based on the condition evaluated on its connection. When an entity has multiple of this behavior, all conditions must be true for the entity to activate.
   4. container, storage_tank: Renamed to read_contents and applicable to any entity with contents. Reads the contents of the entity in a specified inventory; defaulting to the entity's output inventory.
   5. logistic_container: Loses the container functionality to read_contents and renamed to set_logistic_request. Applicable only to containers capable of making requests.
   6. programmable_speaker: Loses the on/off functionality to set_on_off.
   7. rail_chain_signal: Renamed to read_rail_signal, applicable to both signal types.
   8. rain_signal: Loses the reading functionality to read_rail_signal and renamed to set_rail_signal.
   9. roboport: Split into read_logistic_contents and read_robot_stats.
   10. wall: Split into read_proximity_sensor and open_gate.
   11. arithmetic_combinator, decider_combinator: New left_input_connection, right_input_connection, and output_connection parameters are added.
   12. inserter: Loses on/off to set_on_off. Renamed to read_hand_contents.
   13. lamp: Loses on/off to set_on_off. Renamed to set_color.
   14. mining_drill: Loses of/off to set_on_off. Renamed to read_miner_resources.
   15. train_stop: Loses on/off to set_on_off. Split into read_train_contents, read_train_id, and send_to_train.
   16. transport_belt: Loses on/off to set_on_off. Renamed to read_belt_contents.

   A new behavior is added called user_mod which has a name parameter with the behavior name input_connections and output_connections parameters, which may be arrays or tables of connections as preferred by the callee, and a params table containing arbitrary content. The named behavior must be registered as per below.
   
   A new global is added called `control_behaviors`, which allows registering control callbacks similar to commands. Registration accepts two additional parameters only_when_output_needed and only_when_input_changed, which allow the callback to limit which ticks it's called on. Each callback is passed a table containing the entity in question, the current tick, and the input_connections, output_connections, and params specified in the behavior. Each tick, or when input changes and/or the behavior is connected to an output circuit which needs its information, the callback is called. The function must return a table ConnectionPointSpecification -> SignalID -> int which specifies the output signals from this behavior (which are summed with other behaviors at each connection point) on each connection.
   
   The circuit control GUI is updated to allow each individual behavior to be toggled and configured freely. Ideally there would be some provided way to specify a configuration GUI for custom behaviors without needing to modify individual item GUIs, but I don't know the GUI system well enough.

Possible issues ?

I don't know if it would be too much of a mess for backwards compatibility; I don't know how much it matters given that it would be a significant overhaul and the results would probably be majorly positive. Attempts to retain backwards compatibility, like with `circuit_connector_id`, could prove problematic. I've tried to write the sample spec in a way that respects performance costs, but I'm not the sort of player who builds factories for which UPS becomes a serious problem and I don't know the engine, so I don't know for sure.

Why ?

There have been frequent requests for more advanced behaviour be given to circuits, such as separating wire control or more machines connected and controlled. There are mods that give this advanced behaviour, but they frequently require using additional hidden entities with lots of scripting or constructing extra entities nearby.

The specific thing that sent me on my quest to trying to learn about modding and circuit networks was measuring flow through a pump, as a way to determine directly if the fluid supply to a train was too low (the workaround I'm currently using is measuring the last tank before the loading pumps; another one would be to compare values for the train on different ticks).

I think one of the major impediments to bringing this into the base game (forgive me if I'm wrong) is that there are so many parameters and so many options that it's hard to decide what to implement. This proposal is intended to open up this design space for modders, so that the best ideas can be coded, reviewed, and iterated in the community before inclusion in the core game.

Here's a list of things this enables or simplifies, which features are required for them, and how:

1. (#2) Separating wire control can be done, even to some extent for pre-existing control behaviours, by hiding their existing connection point plugged into them, and replacing it with another connection point in the same position. The GUI is modified by the mod to allow selecting individual wire outputs for individual controls, and then script adjusts connections between the visible connection point and the hidden one accordingly. #3 is required to be able to separate behaviors onto different networks where one entity has effectively multiple.
2. (#2, #3) New behaviors can be directly added to entities, even those that cannot currently be connected to the circuit network, by adding connections with appropriate behaviors. With #3 alone, existing behaviors can still be recombined without too much extra effort. Examples include toggling furnaces or assemblers or reading their contents, reading flow rate from pipes, etc.
3. (#2, #3) New combinators can be created directly, such as a time-delay combinator, a selector (use one input to choose one of multiple outputs to send a second input through to), or a filter combinator, without needing multiple entities or excessively complex scripting.
4. (#2, #3) Many mods like Shortwave and LTN can be made simpler by not requiring hidden entities, instead using multiple connection points on its stops.

Some design notes:

1. The weird multi-position distribution thing for connection points is to allow modules that add wire types not have to worry about the positions they connect to, and modules that add power poles not to have to worry about new wire types, without creating an awful result. By having the game distribute them during loading, it makes it so that if a module adds blue and purple wires, the game will select one wire to go with red wires and one to go with green wires. The requirement of determinism makes it unlikely that you get a royal mess where wires criscross over different kinds of poles. An alternative would be to simply say that there are only three wire connection slots and each wire type has to pick one, but this way lets modders add more if they like.
2. Built-in control behaviors are retained to avoid incurring performance cost and exposing proprietary source by moving them to the lua.
3. The mod control behavior is distinct, as opposed to relying purely on events, to make exporting behaviors easier, to parameterize them, to increase performance by avoiding the need to test event filters for entities that have no custom behavior, and to avoid giant fights over the behavior-toggling GUI.
4. Splitting existing built-in behaviors and allowing multiple per entity is done to make them more reusable and modular, and to create logical progression towards the creation of more built-in simple ones such as for reading parameters not currently exposed on the network. In particular, this allows modules to take advantage of existing C++ code to add existing behaviors to entities that do not have them. There would be many immediate improvements that could be considered
5. I tried to avoid setups that would require the base game to add hidden connection points to things. For instance, read_logistic_contents would be enough to implement a logistic input to another connection by adding a hidden connection point and using that.
6. Callbacks are used so that scripts cannot modify the values of signals directly or at weird times during processing, but only at the appropriate time in a tick.

[Rseding91]

It sounds like you don't understand how the game is written/works internally. Everything is compiled - you can't runtime change how something works without editing the source code and recompiling the game to add support for it.

A super simple example which I did for 0.17.0 a few months ago: adding whitelist/blacklist support to filter inserters:

The total functionality of this feature: when in "whitelist" the inserter will grab anything which matches the filters it has set. When in "blacklist" it only grabs things which don't match the filters it has set.

• Create a new enum (whitelist/blacklist) in the Inserter header file with values 0 and 1
• Add the Whitelist property to the inserter class header file
• Add a read function which returns the inserter whitelist mode to the inserter class header file
• Add a write function which takes the new whitelist mode to the inserter class header file
• Add a static parse-filter-mode function to the inserter class header file which takes a name and returns the whitelist/blacklist enum
• Add a static parse-mode-to-string function to the inserter class header file which takes the whitelist enum and converts it to the string name
• Add the map saving/loading of the whitelist value in the inserter cpp file with proper map version checking
• Increment the map version for the new value the inserter is conditionally loading
• Add the blueprint-string save/load of the whitelist value in the inserter cpp file
• Add the write whitelist mode function to the cpp file and make sure if someone tries to set the mode to the current value it does nothing. Otherwise if the inserter doesn't support filters do nothing. Otherwise set the new mode and if the inserter is setup and isn't a ghost activate the inserter
• Add the parse-filter-mode function to the inserter cpp file which converts a string into the whitelist enum or throws an error if the string isn't one of the whitelist or blacklist keywords
• Add the filter-mode-to-string function to the inserter cpp file which converts the enum into the string name or aborts with the invalid enum value
• Change the inserter function which checks if a given item matches the set filters to respect the current whitelist/blacklist enum value
• Change the inserter copy-paste-settings function to also copy the filter mode value if the inserter uses filters
• Change the inserter fast-replace function to also copy the filter mode value if the inserter uses filters
• Add the InputAction handling for setting whitelist/blacklist on a given inserter to the GameActionHandler class
• Add the GuiAction handling for setting whitelist/blacklist on a given inserter
• Add the set-whitelist-blacklist handling logic to the CommonActionHandler class checking if the current opened entity is an inserter
• Set the last user on the inserter when the whitelist/blacklist is changed by a player
• Add a switch GUI element to the inserter GUI header file
• Add the override function for switch-value-changed to the inserter GUI header file
• Add the "whitelist" and "blacklist" locale entries
• Populate the new switch GUI element in the inserter GUIs constructor with the 2 new locale entries
• If the inserter supports filters add the switch GUI element to the GUI, set the switch state to the same as the inserters state, and add the event listener.
• In the inserter GUI cpp file add the switch-value-changed function and if the source is the blacklist/whitelist switch GUI elemenet send the GUI action that the switch state was changed
• Add the lua API read and write functions to the LuaEntity header file
• Add the lua API read and write function bindings to the LuaEntity cpp file
• Add the lua API read function in the LuaEntity cpp file with a description about what it can be used on, what it does, and the types it returns. If the LuaEntity isn't an inserter throw an error saying so. Otherwise check if the inserter supports filters and if not return nil otherwise convert the whitelist enum to a string and return that.
• Add the lua API write function in the LuaEntity cpp file. If the LuaEntity isn't an inserter throw an error saying so. Otherwise read the input string from Lua, convert it into the whitelist/blacklist enum, and call the set-filter-mode function on the inserter.

All of that just for a switch in the inserter entity type. In total that commit changed 23 different files, added 161 lines of code and deleted 14.

To put it bluntly: what you propose simply will never happen. It would mean throwing out the entirety of the code around entities and starting over with the most generic system possible. Such a system would be so bloated with abstractions that nobody would be able to understand what it's actually doing and the performance would be so poor that you probably couldn't ever leave red science without it slowing down.

[alercah]

I can accept that it's too much work to implement---it definitely would require some significant refactoring to make it work, based on my understanding of the existing architecture. But I'm surprised by your suggestion that it would involve completely rewriting the entity system and adding a lot of abstraction---are control behaviors not already a separate object (in C++)? I figured they were because of get_or_create_control_behavior().

That said, I've never seen the source, so you know lots I don't know and I'm not going to tell you that your codebase works a certain way. Only that I thought it maybe worked a different way.

(Incidentally, there are highly abstract systems like ECS that actually can perform quite well---so it's not a given that it would necessarily be slower. But changing fundamental architectures would be such a big project you might as well do nothing else for a few months!)