Remote interface for resource config registration.

Replaces resource spawning system, so that the distances between resources are much bigger. Railway is needed then.

Moderators: orzelek, Dark

Post Reply
User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Remote interface for resource config registration.

Post by eradicator »

So. I'll just make a new thread here and not derail things ;).
Original Quote
"Described a bit": Well, i need an exact spec (see below)
"Works nicely": Uher...there's no programmatic definition of "nice", so i'll skip checking that.
"Registered on first tick": I'll have to look into the timing of remote interface. Should work.
"Should not change during game": Trivial. Just write lock.
"Disable normal gen in data": Trivial. Just provide a function.
"Modification to existing": Easy. Provide a method to give a mod an existing table and get back a new one.

If you have any actual interest in this i'm sure we can figure something out (and i can benefit from it in the future ;).
If you've already given up on it internally...please tell so i don't have to waste effort :p.

So...i wrote a very small sample verifier. The main question is: What do you think about the "specification" part? Is it easy enough for you to fill in the missing values (for now all number ranges are 0 to 10000)? Do you feel comfortable writing a complete specification if i write the verification function?

Code: Select all

local specification = {
  type = {
    type = 'string',
    optional = false,
    allowed_values = {['resource-liquid']=true,['resource-ore']=true},
    },
  --
  allotment = {
    type = 'integer',
    optional = false,
    min_value = 0,
    max_value = 10000,
    },
  --
  spawns_per_region = {
    type = 'table',
    optional = false,
    values = {
      min = {
        type = 'integer',
        min_value = 0,
        max_value = 10000,
        optional = false,
        },
      max = {
        type = 'integer',
        min_value = 0,
        max_value = 10000,
        compare = {min = 'greater_or_equal'},
        },
      }
    },
  --
  richness = {
    type = 'integer',
    optional = false,
    min_value = 0,
    max_value = 10000,
    },
  --
  size = {
    type = 'table',
    optional = false,
    values = {
      min = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        },
      max = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        compare = {min = 'greater_or_equal'},
        },
      }
    },
  --
  min_amount = {
    type = 'integer',
    optional = false,
    min_value = 0,
    max_value = 10000,
    },
  --
  size = {
    type = 'table',
    values = {
      min = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        },
      max = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        compare = {min = 'greater_or_equal'},
        },
      }
    },
  --
  starting = {
    type = 'table',
    values = {
      richness = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        },
      size = {
        type = 'integer',
        optional = false,
        min_value = 0,
        max_value = 10000,
        },
      probability = {
        type = 'decimal',
        optional = false,
        min_value = 0,
        max_value = 1,
        },
      }
    },

  }
  
--[[
local spec_spec = { --change with great care! this specifies the specification format.
  string = {
    allowed_values = {'table','string'},
    optional = 'boolean',
    },
  integer = {
    min_value = 'number',
    max_value = 'number',
    },
  }
  ]]

local err(name,key,message)
  error(
    'Error occured during RSO config import:\n'..
    'Problematic node: '..name..
    'Problematic key: '..key..
    'Problem: '..message)    
  end
  

local function validate_config(config)
  for name,node in pairs(config) do
    for key,spec in pairs(specification) do
      --config entry is not a table (completely broken)
      if type(node) ~= 'table' then
        error('This is not a table: '..serpent.line(node))
        end
      local value = node[key]
      --optional key is missing
      if not (spec.optional or value) then
        err(name,key,'Mandatory key is missing')
        end
      --checking strings
      if spec.type == 'string' then
        if type(value) ~= 'string' then
          err('Value must be a string')
          end
        if not spec.allowed_values[value] then
          err(name,key,'Value must be one of :'..serpent.line(spec.allowed_values))
          end
        end
      --checking integer
      if spec.type == 'integer' then
        if not (
          (type(value) == 'number')) and
          (tonumber(value) == math.floor(tonumber(value)))
          ) then
          err(name,key,'Value must be an integer')
          end
        if (value < spec.min_value) or (value > spec.max_value) then
          err(name,key,'Value is out of range: '..spec.min_value..' to '..spec.max_value)
          end
       --check table
       --TODO
          
      end
    end

Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

One of reasons I did not try this is the fact I'm not a fan of lua.. :D
It seems you are well versed in it :)

Another big reason is trying to explain what certain values do... and that will be a bit tricky for few. Might get me to rebalance the actual calcs a bit to be more consistent.


Specification looks really good. There are also entities in config but I don't think adding them here is sensible. Actual definition for them is more complex since it defines base components. They are only used for bobs enemies now I think.

I can extend the spec to cover all the needed stuff without problems once the basic stuff is working. It would also need bool type for one rarely used field. I also see now that there is min_amount and minimum_amount depending on res type - that might need fixing on definitions side first to use only one name.

Validation of multiple resource spawns might need to happen later on to actually check if ores exist - I would take care of that since parts of this are already done when processing the config file. And they need running game to check the prototypes.


I'm not sure about one thing with remote interfaces - how they work in regards to loading order of mods. I'm not sure how that would affect mods trying to register to RSO.

Thanks for looking into this - it's something potentially useful that I did not have the will to do alone... lua is strange ;)

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

orzelek wrote:I'm not sure about one thing with remote interfaces - how they work in regards to loading order of mods. I'm not sure how that would affect mods trying to register to RSO.
I looked at my favourite page again:
Data Lifecycle wrote:4. control.lua initialization

During this stage each mod's control.lua is loaded and executed in their own Lua instance that will be owned by that mod for the remainder of the play session. Each mod has its own Lua instance and own global table to store data. Because this is run every time a save file is created or loaded you don't need to restart the game to see changes made to the control.lua file. Simply restarting or reloading a save will re-run this stage.

During this stage access to the global table is not available. The script table and the remote table are however available. Note, although the global table has not been setup if a mod does populate the table with some data it will be overwritten by any loaded data.

At the end of this stage (if loading a save file) mod data saved in the map file is loaded and the global table for each mod is restored.
5. control.lua init/loaded

Using the mod order each mod is setup:

When creating a new game, script.on_init() will be called on each mod that has a control.lua file.
When loading a save game and the mod existed in that save game script.on_load() is called.
When loading a save game and the mod did not exist in that save game script.on_init() is called.

During the script.on_load() event handler access to the game table is not available. This handler is meant for only 3 things:

1. Re-setup meta-tables. Meta-tables are not persisted through save-load.
2. Re-setup conditional event handlers (subscribing to an event only when some condition is true to save processing time).
3. Create local references to data stored in the global table

Attempting to change the contents of the global table during the script.on_load() event handler is not allowed. Doing so can lead to desyncs if the mod is used in multiplayer and will generate an error if the game detects it has been changed in any way.

During the script.on_init() event handler access to the game table is available and any and all changes deemed necessary by the mod are free to be performed without risk of breaking anything.
So far as my current knowledge and thought process goes...
  • Each mods control.lua is executed in loading order (respecting dependency). RSO sets up the remote.interface
  • Each mod's on_init is called. Each mod calls remote interface "add_config_to_rso". RSO verifies each call.
  • At the end RSO's own on_init is called. The game starts
(The main thing to consider here is that remote.interface can be defined outside of an event, but it can not be called outside of an event... if i remember correctly. *sigh* So much stuff to look at.)

So...depending on what you do in on_init/on_load (i don't know about internals like that) is, that the dependency changes. RSO might need to load after mods, so RSO would need to depend on the mod, instead of the other way around. This would mean that any given ModA can't use both dynamic registration and static registration as static requires RSO to load before the mod? This can be engineered around ..i think. It all depends on what you use on_init/load for. But even if the remote.interface ends up only working for 50% of the mods due to what they want to do, i think it would still be a win. (And if we can make the devs add a reverse dependency you don't even need to depend on anything yourself :p)

As far as i understand the config is currently dynamically generated on each load right? Or is it stored in global with some kind of access protection?
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

Order here would imply that any remote registration will take place first so then order of calls doesn't matter.

I would need to move building of initial config from RSO init to first chunk gen since I can't guarantee that all mods adding configs from their init would be before RSO without tons of dependencies.

Currently config is built and stored in global table. It's rebuilt on any mod changes but in practice it could be build from start every time.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

orzelek wrote: I would need to move building of initial config from RSO init to first chunk gen since I can't guarantee that all mods adding configs from their init would be before RSO without tons of dependencies.
If it can be moved to "first chunk" that would be perfect as it ensure that all mods have done their registration buisiness. And also get rid of any nessecity for dependencies whatsoever :D. (Though it's probably still a good idea for other mods to add it to indicate they're compatible).

I'll try to complete the proof-of-concept parser till next monday.
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

eradicator wrote:
orzelek wrote: I would need to move building of initial config from RSO init to first chunk gen since I can't guarantee that all mods adding configs from their init would be before RSO without tons of dependencies.
If it can be moved to "first chunk" that would be perfect as it ensure that all mods have done their registration buisiness. And also get rid of any nessecity for dependencies whatsoever :D. (Though it's probably still a good idea for other mods to add it to indicate they're compatible).

I'll try to complete the proof-of-concept parser till next monday.
Dependencies I have are mostly for disabling stuff. They are not needed for any of the control parts.
I tried to drop bob's one once... and ores were spawning back again :D Not even 100% sure why.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

Haven't forgotten about it btw. Working on the validator right now ;).
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

eradicator wrote:Haven't forgotten about it btw. Working on the validator right now ;).
Heh np.
No hurry on it since I don't know when I get to work on it.

I was planning to do major changes when 0.17 comes out. That would include the remote interface and maybe experimental ore balancing algorithm.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

orzelek wrote:No hurry on it since I don't know when I get to work on it.
And i don't know when i get to work next. So i'd rather do it now. :p
Files don't rot so you can always use it later ;).

First prototype is 95% done. I've attached it. The only thing that's missing so far is checks for the sub_spawns table, i hope i can get that done tomorrow. Also i don't really know which values are optional/mandatory. I.e. Must a resource have richness? Or min_amount? etcpp. That's something that'll need some work. Best would be if you could give me a list, explanation etcpp of all implications (i.e. "if it has richness it must also have size", "if it's an enemy base it must have force", etc or something). No code needed. Just plain words would do. Also i have no clue what clear_range is supposed to be, so that's not specified yet either.

That said, the prototype already found a bug in one of the config files: dp77ores.lua specifies:

Code: Select all

starting={richness=8000, size=15, probability=0,5}
for some of the ores. Note how it says "probability = 0" followed by a lone five. Instead of 0.5. ;)

Oh, and ofc the spec is very rough and just assumes everything can be a huge number, as long as it is a number. But that's hopefully easily fixed by your knowledge ;).

I've also included an example implementation of the remote interface, so you can throw stuff at it via console.
Attachments
rso-config-validator_0.0.1.zip
(125.18 KiB) Downloaded 106 times
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

I don't think you need to process enemy parts. There are only 2 sets of those currently and I doubt it will be more.

I see you are already validating setups that are defined now - that was my plan too. The one with error is a nice catch :D

I'll try to write up a description of whats needed and when - I would pass on enemies I think. They might need an update soon actually since I might need to upgrade this settings to allow for multiple enemy spawner types with some randomnes there.

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

Rodger. Adding value range verification for the that one missing table would probably only take 15mins or so with the system already in place. So if you decide to not overhaul enemys, or you want verification before that it's not a problem ;).

And in the meantime i'm pondering how i can implement the optional/mandatory rules. I've managed to disgress into natural-language rule parsing (no, i don't really want to do that ...). Here's a few made up syntax examples with random conditions for your enjoyment:

Code: Select all

  Rule({condition={'starting'},requirement={{'starting','min'},{'starting','max'}},junction='xor',msg='not both!'})

  Rule(config.minimum_amount) : must_not_have(config.min_amount),
  
  Rule(config.min_amount) : precludes(config.minimum_amount),
  Rule(config.min_amount) : implies(config.minimum_amount),
  
  Rule 'If config.type is  resource_ore it must have richness.',
  Rule 'If config      has richness     it must have size.',
  
  Rule('If','type','is','resource_ore','it', 'can't have', 'force')
  Rule('If','type','is','resource_ore','then','starting', 'mustn't have', 'richness')
  Rule('If','config.type','is','resource_ore','starting', 'can't have', 'config.force')
  Rule('If','config','has','size_per_region_factor','it', 'must have','absolute_probability')
  
  Rule('If','config','has','richness','then','it','must have','allotment')
  Rule( if('config','has','richness'), then('it','must have','allotment') )
  
  Rule('If config has starting.min then it must have starting.max or starting.richness and allotment')
  
Possible, but not particularly practical. I'll be waiting for instructions. ;p
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

You sir are going into scary areas :P

I think that tables you had in previous example work. They would need to be extended to add other tables like those for multi resources or size. I think that limitation of unique key name in table is not a problem here.

And I did realize that accidentially richness is multi use now with different variants depending on resource type. I don't think thats good for validation. Might rename richness for fluids to make sure there is no duplicate name with different meaning (fluid_richness).

I'm pretty flexible with changes as long they can be easily applied to existing configs. And after recent development in some mods and todays FFF (we might have kind of creep.. ?) I think that entity is good in validator. And I'll need to extend it to add spawner table with allotments most likely. And additional setting conditional.. maybe. I'm on a fence with the setting - inside RSO it would be needed with internal definition but it would be not needed if mod would prepare config based on it's settings and pass proper variant to RSO.

Tbh I'd rather change some problematic stuff in configs if it will simplify validation. Just make debugging easier :P

User avatar
eradicator
Smart Inserter
Smart Inserter
Posts: 5206
Joined: Tue Jul 12, 2016 9:03 am
Contact:

Re: Remote interface for resource config registration.

Post by eradicator »

orzelek wrote:Tbh I'd rather change some problematic stuff in configs if it will simplify validation. Just make debugging easier :P
You worry about how the configs are best for use with RSO, i worry about how to validate them ;).

So far there hasn't been anything difficult. It's just a lot of typing to do. Richness=100, and Richness={min=1,max=2} variants are already validated, though ofc knowing nothing about why two variants exist there's no logic differentiate between them, they're both simply "valid". I assume you're saying that {min,max} variant is for fluids only?.

What i am pretty sure about after writing the prototype is that value-range validation ("is richness too large?") and requirement validation ("does it have richness at all"?) need to be different steps, because the rules don't combine well, and are also different per type for the second step.

As for the Rule() approach that's mainly an idea i had to make the system (hopefully?) easier for not-me-ppl (i.e. you) to use. If i had to write it just for myself i'd just write a giant block of if/then/else. But there's so many ways to solve that. Main thing i need to decide which way is best is having the rules written down in the first place. Or at least partial rules so i know what they look like. I.e. i don't know if min must always be accompanied by max, or if there are defaults, etcpp.

But if we're aiming at a 0.17 launch date we still have time to figure things out ;).
Author of: Belt Planner, Hand Crank Generator, Screenshot Maker, /sudo and more.
Mod support languages: 日本語, Deutsch, English
My code in the post above is dedicated to the public domain under CC0.

orzelek
Smart Inserter
Smart Inserter
Posts: 3911
Joined: Fri Apr 03, 2015 10:20 am
Contact:

Re: Remote interface for resource config registration.

Post by orzelek »

Tbh I'm looking at configs as someone who mostly knows what configs are doing.
But certain things like reuse of same keyword for different purposes might be misleading later on for others trying to make the config.

I will try to prepare some short document that will describe various config types and whats in them. Not sure when.. it sounds like documentation ;) And some values (like exact meaning of richness for example) will be really difficult to describe doue to convoluted stuff thats done with them before they change into actual ore patch.

And the idea with table for optionals that holds all the valid and optional elements for this table with true/false to state if they are optional is something that looks easy to modify. Anything thats not in the table and found is element it describes would be considered invalid. So it work from my side due to ease of use :)

Post Reply

Return to “Resource Spawner Overhaul”