Page 1 of 1

Issues checking for ore in on_chunk, for later acting mods

Posted: Fri Jul 13, 2018 11:46 pm
by Avacado
I wrote Whistle Stop Factories, which spawns giant factories in the wild using on_chunk_generated. One of the features I'd like my mod to have is to not spawn on ores (my factories can't be moved, so they'd potentially prevent access to the ores unless someone were to want to destroy the factory). I already check for ores before spawning my factories, but some mods add ores on the on_chunk event after my mod goes, so are effectively putting ores UNDER my factories. The two partial solutions I've come up with both have problems:
  • First, it appears event order happens in the same mod load order, so I can add optional dependencies for many of the better know ore adding mods, but this is only a partial solution since any mods missing from my list of dependencies can still spawn ores under my factory
  • My second approach was to have the on_chunk_generated append each chunk to a queue and then have a on_nth_tick work through that queue so that each chunk is processed on a slight delay. The problem was, that even if I set it to, say every 5 ticks, the entities would spawn but wouldn't appear on the minimap until I got closer and saw them with my own vision. I assume that the minimap was just a snapshot of what was there when the chunk spawned or something, at least when using the .chart command, which is maybe why the structures didn't show up on the minimap until I explored them a second time
Any thoughts on a better approach to making sure I don't spawn on ores and still have my structures show up right away on the minimap?

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 12:14 am
by orzelek
You started similar topic as this one here:
viewtopic.php?f=79&t=61520

And as for your question - you can't do anything to fully prevent the issue. For example RSO can spawn ore patches outside current chunk and they can land under your factory buildings this way since game allows ore under buildings.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 8:11 am
by eradicator
I consider adding dependencies the right approach. Yes, some new mods can add new stuff. But the new-release-frequency of mods that spawn ores in on_chunk_generated should be pretty low. I'd assume most do it either via RSO (which is solved by dependency) or via normal autoplace, (which should occur before the event?).

on_nth_tick won't work as you can't guarantee it runs on a later tick. It's a fixed interval that might run on the same tick if you're "unlucky". You'd have to use a dynamic on_tick handler that can guarantee execution n ticks after the event.

As for charting i'd try calling chart() on the chunk you want updated, possible using unchart_chunk() before if nessecary.
orzelek wrote:For example RSO can spawn ore patches outside current chunk
You mentioned this in the other thread too and i don't quite get it. You still have to generate those "outside" chunks first don't you?. In which case they should raise a normal generation event which is handled according to dependency order?

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 11:22 am
by orzelek
eradicator wrote:I consider adding dependencies the right approach. Yes, some new mods can add new stuff. But the new-release-frequency of mods that spawn ores in on_chunk_generated should be pretty low. I'd assume most do it either via RSO (which is solved by dependency) or via normal autoplace, (which should occur before the event?).

on_nth_tick won't work as you can't guarantee it runs on a later tick. It's a fixed interval that might run on the same tick if you're "unlucky". You'd have to use a dynamic on_tick handler that can guarantee execution n ticks after the event.

As for charting i'd try calling chart() on the chunk you want updated, possible using unchart_chunk() before if nessecary.
orzelek wrote:For example RSO can spawn ore patches outside current chunk
You mentioned this in the other thread too and i don't quite get it. You still have to generate those "outside" chunks first don't you?. In which case they should raise a normal generation event which is handled according to dependency order?
As far as I know it's a bit of a cheat. And it fails when ore patches get really big so they get cut out.
I'm not 100% sure how engine trets those but either game can check for collisioin and spawn on ungenerated chunk or it does some kind of partial generation when collision checks or entity spawns are requested outside of exisitng chunk.

I've been thinking about proper solution and it's possible from few versions since we can order chunk generation. It would need actual proper ore generation queue with chunk validation. And then it will be happening after actual chunk generation event - potentially few ticks after if game would need to generate the chunks. It would solve issues with cut out ores when they get big and allow for massive ore patches with very low frequency.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 11:38 am
by eradicator
Huh. That sounds hacky indeed. Though it would imply that RSO adds ores even before on_chunk_generated is raised, and thus the execution order for mods with dependency wouldn't change. Unless the engine does something weird there.

Is chunk generation really that new? I thought it's been there for some time. Also you might not even need a queue. You can use the new LuaSurface.force_generate_chunk_requests() to generate all chunks you need "instantly". Though a queue would be better for performance ofc. Nope. You'd end up generating the ores after on_chunk_generated is raised for every other mod (and your own a few times). Queue is required.

@OP:
If you unchart_chunk() the surrounding chunks after generating your factories then you get a chance during on_chunk_charted to check again for ores and remove them before the player sees them. Might look a bit weird when a radar "scans" a chunk and suddenly 3 others go dark :p.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 11:52 am
by orzelek
Triggering on chunk in my own mod is not a problem since it keeps track of those and what has been spawned.
But queue is needed due to fact that chunk generation is happening in background queue. And this method would cause RSO to spawn ores few ticks after all the chunk generated events for whole field were generated.

I didn't see the force chunk generate method.. I might need to look into it. It would mean that I don't need a queue just re-entry protection in chunk generation handler during actual generation. I was a bit affraid of performance hits if I'd need to generate 20 chunks in a go and then spawn ore patch there.

Playing around with chart/unchart is tricky. Main problem is that charting of map is stored per force but chunk generation is forceless. So you need to make some assumption about how to figure out force to use in those. It's not a problem in single player but could get tricky during multiplayer. RSO has option to chart ores when spawning but it does it for all the forces then.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 12:17 pm
by eradicator
Not having tested any of this and having little experience with chunk gen, i think it works like this:
  • Engine decides to generate a chunk from the chunk generation queue.
  • on_chunk_generated is raised
  • mods do their stuff in order.
If you use just generate_chunk() you're simply adding to the existing queue and nothing changes.

But! If you call force_generate_chunks() during an on_chunk_generated event you have a problem.
  • Engine decides to generate a chunk from the chunk generation queue.
  • on_chunk_generated is raised
  • mods do their stuff in order.
  • on your mods turn you call force_generate_chunks()
  • on_chunk_generated is raised for ALL chunks in the queue.
  • all mods including your own have to process each chunk
  • execution goes back to the original handler that called force_generate_chunks()
So realistically, you can only use force_generate_chunks() at the very end of a handler, because if you do anything after that you do it after on_chunk_generated has been processed by all other mods.

What i was imagining was more like:

Code: Select all

global.cache = {}
script.on_event(defines.events.on_chunk_generated,function(e)
  local chunk = get_chunk_identifier(e.surface,e.area)
  if not global.cache[chunk] then
    calculate_ore_field_to_queue(chunk) --can generate an arbirary number of chunks
    end
  fill_chunk(chunk)
  end)
This way you generate all ore fields to cache first, and you don't need to bother with manually generating chunks at all, because the cache will simply already exist for a chunk that contains an "adjacent" ore field. The cache structure would be something like:

Code: Select all

cache = {
  ['chunk_identifier_1'] = {{pos=orename},{pos=orename},{pos=orename},{pos=orename}}
 }
This also avoids any performance spikes during generation, because chunks are generated in the exact same order and speed they would be without the mod.

I vaguely recall reading that RSO uses "regions" internally, and always assumed that they worked like this. But apparently they don't?

Edit: Hm. Ok, wait. Didn't you say you're also checking the surrounding chunks to get a "better location" for an ore field. That would probably be what's messing up the order of on_chunk_generated even for dependant mods. Though i'm not sure how you can even do that without using force_chunk_generate().

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 2:57 pm
by Avacado
I consider adding dependencies the right approach. Yes, some new mods can add new stuff. But the new-release-frequency of mods that spawn ores in on_chunk_generated should be pretty low. I'd assume most do it either via RSO (which is solved by dependency) or via normal autoplace, (which should occur before the event?).
I think you're right that that does sound like my best approach.
on_nth_tick won't work as you can't guarantee it runs on a later tick. It's a fixed interval that might run on the same tick if you're "unlucky". You'd have to use a dynamic on_tick handler that can guarantee execution n ticks after the event.
This doesn't matter as long as in any given tick all of the on_chunk_generated is either done entirely before the on_nth_tick event or entirely afterwords. If on_chunk_generated does everything first, then I'll still get what I wanted, because it acts after ever single other on_chunk_generated action, even if it runs during the same tick. If on_chunk happens after, then on_nth_tick won't process that chunk because it won't have been added to the queue yet (since I add it to the queue in my on_chunk event, which is all I would do in the on_chunk event). That being said, apart with the other problems this introduces, I agree with what you said in the other thread that this is probably the wrong approach.

Orzelek, collision_masks were mentioned a lot in the other post, but I'm not sure if I fully get the implications of adding those:
  • I assume the main feature of collision_mask='resource-layer' is it would prevent the player from placing on ore
  • But would it prevent a script from placing factories on ore?
  • And would it prevent ore from being placed under my factories?
  • So then both create_entity and can_place_entity would fail?
I thought in earlier tests of my mod I got the factories to actually spawn on water, which makes me think that this is a pretty useless feature for my script if no blocking is actually done for anyone but the player, unless RSO explicitly avoids violating the collision, which is what you were debating about in the other post, right? So it doesn't really do anything except prevent in-game placement by player/bots and maybe serve as a flag that other mods can choose to explicitly respect?

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sat Jul 14, 2018 3:21 pm
by eradicator
Avacado wrote:
  • So then both create_entity and can_place_entity would fail?
Create_entity almost never fails, and does not check collision, it'll happily spawn anything anywhere. Want 50 assemblers on the same water tile? No problem. The only case i know of where it fails is when you try to spawn two loaders at the exact same position (i bet there are more, but you get the idea). If you want to check for collision you have to call can_place_entity before. If you want collision with ore only then count_entities_filtered{area=assembler.bounding_box,type='resource'} should be equivalent.

@nth_tick: sadly there's no documentation on event order. https://lua-api.factorio.com/latest/def ... nes.events isn't sorted alphabetically so it *might* be the order. Or it might not. I posted a request for explanation. Maybe we're lucky and it's not "the order changes too often, don't rely on it.".

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sun Jul 15, 2018 2:58 am
by Avacado
eradicator wrote:@nth_tick: sadly there's no documentation on event order. https://lua-api.factorio.com/latest/def ... nes.events isn't sorted alphabetically so it *might* be the order. Or it might not. I posted a request for explanation. Maybe we're lucky and it's not "the order changes too often, don't rely on it.".
My point was the order doesn't matter though.
  • nth_tick runs with nothing in queue
  • On_chunk event starts
  • ... Some Mods on_chunk before
  • My mod adds chunk to queue
  • ... Some Mods on_chunk after
  • On_chunk Ends
  • nth_tick runs with the chunk in the queue
It doesn't matter if nth_tick is before or if nth_tick is after, it just would affect whether the nth_tick that processes it is later in the same tick or in the next tick. Either way by the time the nth_tick is running, all of the on_chunk processing will be done for the particular chunk that was added to the queue.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Sun Jul 15, 2018 11:27 am
by orzelek
To figure out how on_chunk works we'd need to play around with log's and generation of big areas in game.

And usage of on_nth_tick would still require check if all chunks were actually generated. Thats why I was thinking about resource generation queue separately.

RSO works on regions - this means that when first chunk in region is encountered all the ore fields inside region are prepared and placed into table. It doesn't generated those fields at this stage so they only exist in chunk that was determined to be their center. After that generation of actual chunks triggers full calculation of ore field along with collision checks. Then collision avoidance makes a decision should field be planted as is or potentially moved a bit to make it more complete.

In queued system this step from on chunk gen would get calculated, used to estimate the area of the field and then queue all chunks in there that are not generated for game to generate. Then it would wait and periodically check if all the chunks are ready and when they are up it would do the collision check and place the ore. With this variant you would have no knowledge from outside RSO when the ore patch is placed - it could be any number of ticks past it's triggering by on chunk generated.

I have no idea how game behaves with chunk generated events if you would trigger forceful generation of chunks from handler. I think that way RSO spawns ores now might be actually causing this anyway - it can place ore in chunks that are outside of current one. I did not debug how it really behaves - it seems to work well enough. I think current method starts to fail once you create fields that are 5+ chunks in size - it will start cutting into ore patches visibly then.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Mon Jul 16, 2018 4:36 pm
by eradicator
@Avacado: Ah yea, I was stupid and didn't see the on_nth tick order doen't matter. (Unless there was a need to do stuff in a defind tick)

@orzelek:
I still don't understand how you can do
orzelek wrote:Then collision avoidance makes a decision should field be planted as is or potentially moved a bit to make it more complete.
this without having all chunks generated. Which would mean you're doing it after on_chunk_generated from the perspective of every other mod. This requires further testing how collosion checking of not-yet-existing chunks interacts with on_chunk_generated.
orzelek wrote:In queued system this step from on chunk gen would get calculated, used to estimate the area of the field and then queue all chunks in there that are not generated for game to generate. Then it would wait and periodically check if all the chunks are ready and when they are up it would do the collision check and place the ore.
I think the queue system should be able to do this with partial fields placed in chunks as soon as they are generated, same as vanilla. So much to test...

Slightly deviating from other possible solutions it might be a good idea to additionally implement a custom event on_rso_chunk_processing_completed that is guaranteed to be raised after RSO is done with whatever it does in that particular implementation. This could then be used by mods that particularly care about doing things after RSO and would not be affected by any possible delays etcpp.

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Mon Jul 16, 2018 4:46 pm
by orzelek
The fact that collision checks work in chunk that might not be generated seems to be a hidden game feature. At least that they work for at least few chunks away from generated areas. It works for now - if it would not it would kind of force me to switch to queued system.

And in queued system doing partial fields would not work - since collision avoidance needs to see whole area before making a decision. Then after the decision is made it can spawn whole field at once. It could be practical to make collision checks per chunk to not dump all of it in one tick but that would require additional data collection for final decision making and would need separate recalcs after failed decision is made and whole area gets moved by chunk in one direction.

I was thinking about having a callback that ore field has been generated - not sure if I can make custom event or not.

Any queued solution will be vastly more complex then what's there now and being forced to do this could end badly ;)

Re: Issues checking for ore in on_chunk, for later acting mods

Posted: Mon Jul 16, 2018 6:25 pm
by eradicator
(*sigh* browser deleted 15 minutes of post writing...)

Did a quick hack and you're right. can_place_entity works up to 3 chunks outside the generated area and doesn't raise on_chunk_generated when doing so.

Code: Select all

/c 

--[[pregenerate chunks to stop  auto-generation]]
game.player.surface.request_to_generate_chunks({0,0},20)
game.player.surface.force_generate_chunk_requests()

script.on_event(defines.events.on_chunk_generated,function(e)
  print('New Chunk: '..serpent.line(e.area))
  end)

for i=1,100 do
  print('Can place here at <'..i..'>?: '
  ..tostring( game.player.surface.can_place_entity{
    name='oil-refinery',
    position={x=32*i,y=0},
    force=game.player.force,
    build_check_type=defines.build_check_type.manual,
    })
  
  ..((not game.player.surface.is_chunk_generated({x=i,y=0})) and ' (Chunk does not exist.)' or '')
  )
  
  end
orzelek wrote:I was thinking about having a callback that ore field has been generated - not sure if I can make custom event or not.
Yes. "Trivial". It has a minor quirk that the calling mod must register the event in on_init/etc because remote.call can not be done outside of events.

In RSO control.lua body:

Code: Select all

local rso_event_ids = {
  on_rso_chunk_generated = script.generate_event_name()
  }
local rso_interface = {
  get_rso_event_ids = function() return rso_event_ids end
  }
remote.add_interface('rso_interface',rso_interface)
In RSO chunk generation:

Code: Select all

script.raise_event(rso_event_ids.on_rso_chunk_generated,
  {area = {{x=0,y=0},{x=32,y=32}}, --chunk position
   surface = LuaSurface,
   -- ore = --whatever other things the event should carry
   
   -- [[factorio will automatically add three more items:]]
   -- modname = 'rso'
   -- name = rso_event_ids.on_rso_chunk_generated,
   -- tick = game.tick
  })
In dependant mods control.lua on_init:

Code: Select all

local function register_to_rso()
  local rso_event_ids = remote.call('rso_interface','get_rso_event_ids')
  script.on_event(rso_event_ids.on_rso_chunk_generated,function(e)
    --dostuff
    end)
  )
script.on_init(register_to_rso)
script.on_load(register_to_rso)
script.on_configuration_changed(register_to_rso)