[0.17.x] demo-resource-autoplace.lua sucks

[bobingabout]

As the title says, demo-resource-autoplace.lua sucks, and here's why.

TL;DR? See the bottom.

The idea, maths and final result of this file and it's functions seems pretty positive on the surface.
In the base game (no mods), it works pretty well, allowing each patch of ore to spawn with an almost guarantee, however, when you start modding, things start to fall apart.

on initial release, I didn't understand the script at all, I had no idea what it was doing. So to try and understand, I create a serpent block log of data.raw.resource["iron-ore"].autoplace, to give the results. I expected something similar to the old system, easy to understand tables, peaks, tags with numbers etc... you know, a readable LUA table.
What I found instead was a complete convoluted mess, it looked like it tried to store the maths as tables, it made no sense to me, I have never seen anything like it in the game at all.

So, I looked at just, you know, using it. This wasn't easy, the functions were not globals, to be able to call them I had to include the file from another mod, base mod. Upon doing this, it seemed that all locals (the key one being next_resource_index) were RESET.
So I edited the base game, changed the local for next_resource_index to a global, and also the functions themselves. This worked, I could just call the functions to add my new ores, so I submitted a PR and had it added to 0.17.5.

This however is not the end of the story, you see, there's 2 other variables, locals in file called starting_resource_count and regular_resource_count, set to 4 and 6, the total number of resources in the base game. These, as far as I understand are important for the "Can't place an ore on top of another" mechanic.
The result of these values is that when you add another ore via modding, and generate a map in the game, there's a chance that your new ore can be assigned to the same location as one of the previous ones, as it is only trying to set 4 unique starting area ores, and 6 map wide. With adding just 1 ore, it didn't happen often, so I did not notice.

Cue every modder having access to it, and plenty adding their own ores, plus me converting 15 of my 16 resources to use the new system. Due to me attempting to have a very low count for gems, it would not spawn at all in the new system, so I had to resort to continuing using the old system for it.
In any case, there are now 7 starting area ores and 21 ores total on the map using this new system. Most of the time, due to no overlap not being guaranteed, you only actually see 6 or less of the 7, not to mention seeing patches where one earlier ordered small size ore appears in the middle of a larger later ordered ore.

Anyway, that highlights a fatal flaw, those numbers, starting_resource_count and regular_resource_count, set to 4 and 6, can't be accessed by modders. And if you assume you changed them to globals, the base game ores have already had their autoplace data generated before the modder has access to them.


As a conclusion, the dependence on those 2 variables kind of breaks the entire process. The current solution to a mod like mine wanting to add lots of ores is to steal the file, modify those 2 values to account for all resources, delete all previously generated autoplace data, and start over. Even if they were globals it would require changing the numbers and regenerating all data.

A possible solution would be to create setup data in the data phase(can be stored on autoplace to be overwritten later), but then in a later phase such as data-updates, run a script that first counts how many total ores there are, and how many are starting ores, update the numbers, and then run the data generation in one go.
However, there's a problem.
What if someone creates an ore in the data-updates phase, after all these calculations are done? you'd have to start over again.
How about we calculate it in data-final-fixes instead? In theory, nobody should be adding anything new there, but... modders will be modders, and you can't guarantee that they don't.
You'd have to do it later.
By this point, you're back to the old base game method. setup data, (in the old method, this is peaks and noise layer names and the like) is stored in autoplace for the actual data to be calculated only when you generate a map, or at the very least, in the game's code. This is the ideal solution. It pretty much needs to be AFTER the last possible place where modders can add new ores, which is after all LUA data.



So, Key flaws.

1. It is dependant on the user manually calling get_next_resource_index() for every resource, and passing the value back to the ore generator. In theory, you don't need this, it could simply be an internal value that increments each time resource_autoplace_settings is called. next_resource_index itself is a global, so can be changed through direct access if you need to change it, such as resetting. (or maybe not, Arch666Angel actually makes good use of using the same number twice, such as placing a small patch of infinite coal in the middle of a coal patch)
2. The non-overlap rule is dependant on starting_resource_count and regular_resource_count, which in the current implementation are effectively constants. These need to effectively both be infinite in the current implementation, or calculated on the fly after all resources are known. (which means move it from LUA to C++)
3. It doesn't even work when you're trying to create low yield patches of ores. unless I'm doing it wrong, but I was after a patch of ore with a total size of less than a thousand. I ended up with a single huge patch of ore on the preview that appears only if you set frequency to maximum.

So the request is to address the 3 flaws here.

[orzelek]

I had the idea few times and I still see it could work. I think that RSO should be changed to be in C++ and integrated with the game. It has no problems like this
It would have few other that I'm not sure how would work - one of them being not fully calculatable inside only one chunk.

[bobingabout]

I had the idea few times and I still see it could work. I think that RSO should be changed to be in C++ and integrated with the game. It has no problems like this
It would have few other that I'm not sure how would work - one of them being not fully calculatable inside only one chunk.

Well, you did a great modding solution. I hope RSO continues into final release mod sets.

[B_head]

I created a mod to make it easy to add resources.
https://mods.factorio.com/mod/resource_autoplace_kaizen

[Arch666Angel]

After we already had two lengthy discussion on discord: Fully agree

[leighzer]

Yes, I unfortunately agree as well. I did the exact same and did a serpent through data.raw and was unable to gleam any useful information. I do agree with Bob's analysis of the flaws.

I'm sure these will be addressed in time given the map gen is still new. I like the direction things are moving and it is nice to be able to fall back to generating things the old way.

[darkfrei]

+1 for simple for beginning modders resource autoplacement, that must have as less as possible unreadable parameters. Other stuff must be optional and/or on the game side.

[Therax]

I'm in the process of making some tools that post-process all the resource prototypes, counting them up, determining which ones are starting area resources, and then modifying the noise expressions to set skip_span and skip_offset appropriately. It's quite painful and certainly very fragile if and when autoplace parameters ever change.

Consider this a +1 on changing how mods are expected to add new autoplaced resources, and an offer to share said code with other mods doing this sort of thing.

[Airat9000]

demo-resources.lua not works in code in entity

resource(
{
name = "coal",
order = "b",
map_color = {r=0, g=0, b=0},
mining_time = 1
},
{
base_density = 8,
regular_rq_factor_multiplier = 6.0,
additional_richness = 31000000,
starting_rq_factor_multiplier = 1.1
}
),

example edit code bob tin ore
normal place in entity

2019-04-03_14-16-22.png (709.41 KiB) Viewed 4458 times

[TOGoS]

So, Key flaws.

1. It is dependant on the user manually calling get_next_resource_index() for every resource, and passing the value back to the ore generator. In theory, you don't need this, it could simply be an internal value that increments each time resource_autoplace_settings is called. next_resource_index itself is a global, so can be changed through direct access if you need to change it, such as resetting. (or maybe not, Arch666Angel actually makes good use of using the same number twice, such as placing a small patch of infinite coal in the middle of a coal patch)
2. The non-overlap rule is dependant on starting_resource_count and regular_resource_count, which in the current implementation are effectively constants. These need to effectively both be infinite in the current implementation, or calculated on the fly after all resources are known. (which means move it from LUA to C++)
3. It doesn't even work when you're trying to create low yield patches of ores. unless I'm doing it wrong, but I was after a patch of ore with a total size of less than a thousand. I ended up with a single huge patch of ore on the preview that appears only if you set frequency to maximum.

So the request is to address the 3 flaws here.

I have a proposed change that would address #1 and #2, which is that rather than the hard-coded 4 and 6 that are used for starting and regular resource patch type counts, noise variable called "resource-patch-type-count" and "starting-resource-patch-type-count" are used which are incremented by get_next_resource_index() (actually "starting-resource-patch-type-count" is only different if get_next_resource_index() is never called -- once it is, those counts are kept synchronized so that passing a single resource index explicitly makes sense). Mods can still use get_next_resource_index() and explicitly pass it into resource_autoplace_settings(...), but if one is not provided then it will be generated automatically by calling get_next_resource_index() internally.

[bobingabout]

I have a proposed change that would address #1 and #2, which is that rather than the hard-coded 4 and 6 that are used for starting and regular resource patch type counts, noise variable called "resource-patch-type-count" and "starting-resource-patch-type-count" are used which are incremented by get_next_resource_index() (actually "starting-resource-patch-type-count" is only different if get_next_resource_index() is never called -- once it is, those counts are kept synchronized so that passing a single resource index explicitly makes sense). Mods can still use get_next_resource_index() and explicitly pass it into resource_autoplace_settings(...), but if one is not provided then it will be generated automatically by calling get_next_resource_index() internally.

I'm interested to see the results.
Also, #3, I actually was doing it wrong, I managed to solve that issue.

[eradicator]

Autoplace is one of the places i'm still "scared" of. Not being a "math person" doesn't help. If even @Bob and @Angel can't get it to work it looks like i was right to keep my hands off it. :/

All i ever wanted was something simple like:

Code: Select all

{
amount_per_square_kilometer={min=1e6,max=1e7,average=2e6},
tiles_per_patch = {min=100, max=2000, average=300},
shape='banana',
}

And for the occasions where i need "one tile patches" i guess i'll be using on_chunk_generated forever.

[bobingabout]

Autoplace is one of the places i'm still "scared" of. Not being a "math person" doesn't help. If even @Bob and @Angel can't get it to work it looks like i was right to keep my hands off it. :/

All i ever wanted was something simple like:

Code: Select all

{
amount_per_square_kilometer={min=1e6,max=1e7,average=2e6},
tiles_per_patch = {min=100, max=2000, average=300},
shape='banana',
}

And for the occasions where i need "one tile patches" i guess i'll be using on_chunk_generated forever.

for the record, I did actually take the time to copy the file and create an edited version that fixes the issue, but unless everyone uses my method, it doesn't work as intended, especially since everyone seems to be using the same solution... their own unique copy. So if you have 2 mods that do this, the ores from the 2 mods will break each other.

I'm currently working with TOGoS to come up with a base game solution that works for modders, so in theory everyone can use it instead of their own conflicting copies.
And by working with, I mean TOGoS is doing all the work, we're just bouncing ideas off each other and testing with my ores mod. (as it provides lots of mod added ores)
He's made progress, but there's still a few issues to iron out. I'm not going to go into any more detail than that right now.

[darkfrei]

Why they don't use the name of resource as seed? If a is 1 and b is 2, the seed for the resource placing will be unique for every resource.

[bobingabout]

Why they don't use the name of resource as seed? If a is 1 and b is 2, the seed for the resource placing will be unique for every resource.

it's more complex than that.

Basically, with the current generator, before you even start you need to know how many total resources they are because it will generate patches, then every total_resources + resource_index patch, you'll get one for that resource. if total_resources isn't incremented, then you will end up with ore E always overlapping with ore A. if you do increment the resources count, then you'll still get overlaps because the first 4 think there's only 4 total, but the 5th thinks there's 5, and the 6th thinks there's 6 etc... so resource A will place it's stuff at locations 1, 5, 9 and 13, resource B will place it's stuff at locations 2, 6, 10 and 14, C at 3, 7, 11 and 15, D at 4, 8, 12 and 16... E at 5, 10, 15 and 20, F at 6, 12, 18 and 24 etc... as you can see, by the time you get to E, 5, 10, and 15 are already used by A, B and C, and in theory 20 will already have been used by D if I counted that far, and F, 6 and 12 were already used by B and D, but if I kept counting 18 would also have been used by B and 24 by D.

So part of the changes to the system so far is to not define the total counts in the autoplace data of the resource itself, but make that external of it (still in data, just defined somewhere other than on each resource), so when each resource is defined doesn't matter as long as they all have unique indexes and the highest index doesn't go past the total number of indexes used (as in, they're sequential).

of course this only works correctly if get_next_resource_index() is used, as with these changes, that returns a unique index number AND increments the counter. (actually, with the changes, not specifying an index will call get_next_resource_index() internally now, as I suggested in my first post)
then you'll have the issue of "what if I want resource X to use the same location as resource Y?" as per angel's infinite ores where he places a small infinite patch in the middle of a resource of the same type.
currently, the only way to get that would be to know it when you get it (IE, call get_next_resource_index(), store the number, then use it on both resources), but we're working on it (IE, I suggested, what if you just pass "iron-ore" as the resource_index variable of "infinite-iron-ore"?)

If you have any suggestions, you're welcome to post, but this is a very complex piece of code that only TOGoS truly understands, and only a few others understand the basics about it. I mean, I know what it's doing, but I don't think I have the skill to create a solution worthy of being incorporated into the base game.
Bob's mods and Angels do make use of a customised copy of the file that I worked on, and went over with Angel, but then that's only a solution if everyone uses it. but it's not really to the standard that the base game would want. (EG, my solution to not knowing the total ores when you add them was to call a different function that adds the data to a table and generates the actual autoplace data later, in the data-updates stage. TOGoS's solution was to remove the count, and store that somewhere else, so there's no need to store the request and generate the data later)

[Therax]

So part of the changes to the system so far is to not define the total counts in the autoplace data of the resource itself, but make that external of it (still in data, just defined somewhere other than on each resource), so when each resource is defined doesn't matter as long as they all have unique indexes and the highest index doesn't go past the total number of indexes used (as in, they're sequential).

There's still my solution, which is to do a final fixup pass to change all the offsets based on the set of resources defined after all mods have loaded. If such a thing happened in base/data-updates.lua or base/data-final-fixes.lua, that would probably be sufficient.

You can find my code that does this here: https://github.com/mspielberg/Xander-Mo ... #L188-L244

I would personally be okay with telling mod authors that it's okay to define new resources in data.lua, but defining them in data-updates.lua or data-final-fixes.lua will go wrong. That's already true for the parallel case of mods creating fluids that need to have barreling recipes generated in base/data-updates.lua, so it probably wouldn't even require many, if any, mod changes.

[bobingabout]

I would personally be okay with telling mod authors that it's okay to define new resources in data.lua, but defining them in data-updates.lua or data-final-fixes.lua will go wrong.

my standard opinion on this is... everything that needs to exist should exist by the end of the data phase. if you NEED something to exist before creating something else, use a dependency.
so all items, entities, technologies and recipes as a minimum should exist at the end of data phase.
then in data updates, you're free to go around changing icons and ingredients/results on recipes/resources etc.

[eradicator]

I would personally be okay with telling mod authors that it's okay to define new resources in data.lua, but defining them in data-updates.lua or data-final-fixes.lua will go wrong.

my standard opinion on this is... everything that needs to exist should exist by the end of the data phase. if you NEED something to exist before creating something else, use a dependency.
so all items, entities, technologies and recipes as a minimum should exist at the end of data phase.
then in data updates, you're free to go around changing icons and ingredients/results on recipes/resources etc.

How would that work in cases where the new resources are derived off the old ones? I.e. as a rough example i need a deravative resource for *every* resource. Let's call it "contaminated xyz-ore". As i need every other ore to exist, data-updates is the earliest place i can create these resources in. Currently contaminated ore isn't being auto-placed for game-mechanic reasons, but i'd prefer if it wasn't completely impossible to auto-place it if i later find that the current mechanic doesn't work.

[darkfrei]

The data-final-fixes is exactly what you need: resulting calculation of all of previous added resource prototypes.

By the data-updates comes all script-created resource prototypes. Here the mod order must be defined by mod dependencies.

The data has all vanilla-like prototypes, where nothing was added by the script, but data:extend only.

[bobingabout]

I would personally be okay with telling mod authors that it's okay to define new resources in data.lua, but defining them in data-updates.lua or data-final-fixes.lua will go wrong.

my standard opinion on this is... everything that needs to exist should exist by the end of the data phase. if you NEED something to exist before creating something else, use a dependency.
so all items, entities, technologies and recipes as a minimum should exist at the end of data phase.
then in data updates, you're free to go around changing icons and ingredients/results on recipes/resources etc.

How would that work in cases where the new resources are derived off the old ones? I.e. as a rough example i need a deravative resource for *every* resource. Let's call it "contaminated xyz-ore". As i need every other ore to exist, data-updates is the earliest place i can create these resources in. Currently contaminated ore isn't being auto-placed for game-mechanic reasons, but i'd prefer if it wasn't completely impossible to auto-place it if i later find that the current mechanic doesn't work.

I did say "Standard opinion", there are exceptions to every rule. And while my own personal edits to resource-autoplace included in my mod need to have all resources be defined before data-updates, hopefully the solution for the base game doesn't have the same requirements. (Which is why TOGoS is working on a change so you don't need to know how many ores there are going to be when you start adding them)

From what I've seen, the change works, there's just a few issues with the starting area.