Data linking between stages/layers

[ZwerOxotnik]

I want to find/see who and how uses data linking between stages/layers in Factorio. (Currently, I'm not interested in LuaRemote)
I know a few cases of it and I don't seem to notice other uses...

I might describe the topic more, however, I can't do it right now.

[Pi-C]

As far as I know, LuaRemote exists only in the control stage (once you're in a newly created or loaded game). But it seems you're interested in tracking changes during the data stage -- is that correct? I don't think it's possible to get information on what mod has changed a prototype. The only thing I can think of is tracking whether a prototype has been changed at all: Once you're done with creating/modifying a prototype, make a copy of it in a global table. At the start of the next stage, compare your copy with the current state. Basically, something like in the following example …

In data.lua:

Code: Select all

require('util')
my_mod_data = {}

…

data.raw.recipe.rail.enabled = true
my_mod_data.rail =  util.table.deepcopy(data.raw.recipe.rail)

…

In data-updates.lua:

Code: Select all

if util.table.compare(data.raw.recipes.rail, my_mod_data.rail) then
	-- Both tables are the same
else
	-- Prototype has been changed between stages
end

[ZwerOxotnik]

As far as I know, LuaRemote exists only in the control stage (once you're in a newly created or loaded game).

It's possible to make some things before loaded game: gist

But it seems you're interested in tracking changes during the data stage -- is that correct?

No, not really. Imagine you have [constant] data which you need/may use on all/several stages/layers. Okay, it'll be just a lua file which return a some table/data. However, how you're going to manipulate the data and organize it? I mean, you can just parse a table to create almost identical settings with different names and do something like validation with the same data on data/control stage, OR the data can describe a behavior. (I can't come up with an obvious and simple example in Factorio now, and I'm still thinking about it... although this gist may clarify something)
(well, someone even creates a whole UI tree using 1 table)

[eradicator]

No, not really. Imagine you have [constant] data which you need/may use on all/several stages/layers. Okay, it'll be just a lua file which return a some table/data. However, how you're going to manipulate the data and organize it?

Sounds contradictory. If the data is constant you don't need to manipulate it. If you need to transport data from one startup phase to another just store it in a global table.

In any other scenario you could try to encode it and hide it somewhere where it survives the stage transition, but ugly hacks like that are likely to be removed if one of the devs gets too annoyed about it. If the engine does not support a usecase then the best solution is usually "just don't do it" (though there are exceptions like pre-init cross-mod communication).

[ZwerOxotnik]

Um, I seem to have confused people.

"Another" example, a mod has settings.lua, settings-final-fixes.lua, data.lua, control.lua and... cross_data.lua

And I'll just put in the cross_data.lua a table with keys as names like this:

Code: Select all

return {
	setting_name1 = {is_important = true}, -- and other data
	-- etc
	setting_name2 = {is_important = false}
}

or `tile_name = {is_important = false, image = ""}`

Each stage will require the cross_data.lua for their own purpose under different circumstances.

• settings.lua will create only some settings because of environment and those "parameters".
• settings-final-fixes.lua will do something similar.
• Almost same story with data.lua to create prototypes.
• control.lua have to filter and check existence of the settings in the game to store it locally, using the info in events.

[Pi-C]

I tried to do the same in Bio Industries. The problem is that the game provides different data in the various stages:

• Settings stage: mods
• Data stage: mods, data, settings
• Control stage:
  • Initialization: remote, script
  • script.on_init: remote, script, settings, game, global
  • migrations: script, settings, game, global (not sure about remote/rendering)
  • script.on_load: script, settings
  • running game: remote, script, settings, game, global, rendering

In order to have access to all settings during all stages I made the ugly compromise to only use setting_type = "startup". The mod is meant to be modular, so complete branches (techs, entities, items, recipes) can be turned off by a startup setting. During the data stage, I check the respective settings and only create those things if the setting is enabled. Usually, that would be enough, but I have some compound entities where hidden entities need to be added when the base entity has been placed. So, I need to share data between the data and the control stage. They are stored in a file that's required from both stages.

Code: Select all

local ret = {}

ret.compound_entities = {
  ["bi-bio-farm"] = {
    tab = "bi_bio_farm_table",
    base = {
      name = "bi-bio-farm",
      type = ret.HE_map.assembler,
    },
    hidden = {
      connector = { … },
      pole = { … },
      panel = { … },
      lamp = { … },
    }
  },
…
}
…
return ret

The table contains all data needed to create the entities during the data stage, or for creating subtables in global during the control stage. But it's a complete list of all prototypes that may exist -- and some of those may not have been created. So I've made a function that will clean up the list and return just the data of existing prototypes:

Code: Select all

local common = {}
  common.rebuild_compound_entity_list = function()
    local ret = {}
    local base_prototype, base_name, base_type, h_type

    for c_name, c_data in pairs(common.compound_entities) do
      -- Is the base entity in the game? (Data and control stage!)
      base_name = c_data.base and c_data.base.name
      base_type = c_data.base and c_data.base.type

      base_prototype = game and game.entity_prototypes[base_name] or
                        data and base_type and base_name and
                        data.raw[base_type] and data.raw[base_type][base_name]
      if base_prototype then
        -- Make a copy of the compound-entity data
        ret[c_name] = util.table.deepcopy(c_data)

        -- Check hidden entities
        for h_key, h_data in pairs(ret[c_name].hidden) do
          -- Remove hidden entity if it doesn't exist
          if game and not game.entity_prototypes[h_data.name] then
            ret[c_name].hidden[h_key] = nil
          end
        end

      -- Clean table (Control stage only!)
      elseif game then
        local tab = c_data.tab
        if c_data.tab then global[c_data.tab] = nil end
        …
      end
   …
    return ret
  end

return common

Perhaps something like this would work in your case as well?

[ZwerOxotnik]

Perhaps something like this would work in your case as well?

Yes

It may seem that I'm beating around the bush, however, what if a mod use such a way globally rather than for concrete cases?
And, perhaps, something can be implemented to organize such concept better...


Also, about settings. If a mod need other specific condition in settings, then there are 2 solutions/workarounds to have other interactions with the settings.

1. Control the settings at control stage to constrain/make dependencies between each other. (Example: nothing can't change setting_2 as long as value of setting_1 = 5)
2. Create a global table for settings and make a system, GUI/interface (with new events) to maintain, manipulate the new system. It's a complex way. (example: official PvP scenario) And, I'm not fully sure, but probably there's some universal solution in this way for Factorio.

And, the settings stage is first stage, what if someone want to get/check some data from the data stage ((after) second stage) to change/create settings?

After the settings stage of loading has finished, the Lua state is discarded. Changes and functions defined during the settings stage will not carry over to any other stages.

At the end of this stage all setting prototypes are constructed and the startup settings are read from the mods directory "mod-settings.dat" file.

I don't remember/know how mods see settings at the data stage etc

[eradicator]

Um, I seem to have confused people.

I'm mostly confused about what you actually need "modding help" with. You haven't stated any concrete question and you seem to already understand the topic quite well. Unless you just want to look at other mods who require the same file in more than one stage? In that case you can just download a bunch of mods and see if they use "names.lua" or "const.lua" or similar, which are common names for the concept. (I use that myself i.e. here)

And, the settings stage is first stage, what if someone want to get/check some data from the data stage ((after) second stage) to change/create settings?

That makes no sense. You can not read data from the future. Data-stage dynamically generates stuff based on settings, thus using data-stage output to influence settings-stage would create a recursive dependency that can't be resolved.

[Pi-C]

I don't remember/know how mods see settings at the data stage etc

That's explained in the settings tutorial.

And, the settings stage is first stage, what if someone want to get/check some data from the data stage ((after) second stage) to change/create settings?

That makes no sense. You can not read data from the future. Data-stage dynamically generates stuff based on settings, thus using data-stage output to influence settings-stage would create a recursive dependency that can't be resolved.

You could read data from the past, though. My idea was, you change a startup setting, then the game applies the data and restarts, so then these settings must be available for use. Alas, the settings table doesn't exist yet during the settings stage -- it's only created there, so you can't change a setting now based on the changes you made during the previous run.

[ZwerOxotnik]

... you seem to already understand the topic quite well. ...

Nope, I'm still wandering around, and I feel like I'm not familiar with similar situations fully and other variants of solutions.
It's monotonous to guess where I can find examples and only a few people make such things.

Probably, it's tough to understand what else I want to see in this topic because I don't specify/narrow questions, description and/or it's very similar or refer to common things, or I just confused everyone.

That makes no sense. You can not read data from the future. Data-stage dynamically generates stuff based on settings, thus using data-stage output to influence settings-stage would create a recursive dependency that can't be resolved.

Um... It might exist as another stage, isn't?

And I kinda lost among paradigms, approaches etc... Maybe, if I'll picture it, then it'll be easier for me to continue and describe the current topic... I'll, probably, just try decorators etc in lua

[eradicator]

It's monotonous to guess where I can find examples and only a few people make such things.

That's the way it works though. I certainly don't remember every mod I looked at, let alone specifics like mod structure. So the only thing I can link to is my own stuff. And besides most modders (myself included) are amateurs who might not even know themselfs whatever it is you want to know. There's probably better code bases to learn lua from than factorio mods.

Probably, it's tough to understand what else I want to see in this topic because I don't specify/narrow questions

Yes. If you ask vague questions you will get vague answers.

Um... It might exist as another stage, isn't?

I don't understand that question. There are three stages (settings/data/control) and three phases ( /-updates/-final-fixes). These are hardcoded and can not be changed by modders. Also while that distinction of "stage" and "phase" is my own habit and other people might have different names for those I've never seen anyone call them "layers".

And I kinda lost among paradigms, approaches etc...

Every approach has pros and cons. And often you'll only know which one would've been best after you've made the mistake of using a different once. And "the best approach" also depends on what kind of mod you're making. Modding is supposed to be fun, so learning on-the-go is imho just fine.

Also your posts would be easier to read if you stopped using those annoying spoiler tags.

[ZwerOxotnik]

So, probably, nothing will be here for a long time.

Although, I got an idea and found a target for experiments. There's a game called as "Node Wizardry" and I like customization a lot. If I'll be still inspired and productive, perhaps, I'll make a similar and simpler realization of the concept without nodes...
And I'm going to make mod integration without interfaces (although I'll use some workaround(s) to implement it, and I'll set a mod interface for other cases anyway)