"Evaluation" functions for surface.find_entities_filtered

[Reika]

By that I mean something you can pass in and have it evaluated to determine whether the entity is applicable.

Analogies to this can be found in SpaceEngineers or Minecraft, which are C# and Java respectively:

SpaceEngineers:

Code: Select all

GridTerminalSystem.GetBlocksOfType<IMyTextPanel>(displays, b => b.CubeGrid == Me.CubeGrid && isBatteryDisplay(b.CustomName));

Where "displays" is populated only with blocks for whom CubeGrid is the appropriate value and which isBatteryDisplay returns true

Minecraft:

Code: Select all

world.selectEntitiesWithinAABB(Class p_82733_1_, AxisAlignedBB p_82733_2_, IEntitySelector p_82733_3_)
-------------
List list = this.worldObj.selectEntitiesWithinAABB(EntityChicken.class, this.boundingBox.expand(5.0D, 3.0D, 5.0D), IEntitySelector.field_152785_b);
-------------
    IEntitySelector field_152785_b = new IEntitySelector()
    {
        private static final String __OBFID = "CL_00001542";
        /**
         * Return whether the specified entity is applicable to this filter.
         */
        public boolean isEntityApplicable(Entity p_82704_1_)
        {
            return p_82704_1_.isEntityAlive() && p_82704_1_.riddenByEntity == null && p_82704_1_.ridingEntity == null;
        }
    };

Where list is populated only with entities which return true for isEntityAlive() and which have a null riding and ridden entity value.

Sample use case in factorio:

Code: Select all

local function hasHealth(entity)
    return entity.health and entity.health > 0
end

local entities = surface.find_entities_filtered(area = box, force = game.forces.enemy, checks = hasHealth)

[Therax]

It's of course already possible to do this yourself:

Code: Select all

function find_entities_filtered_with_predicate(surface, args)
  local predicate = args.predicate
  local entities = surface.find_entities_filtered(args)
  local out = {}
  for _, entity in ipairs(entities) do
    if predicate(entity) then
      out[#out+1] = entity
    end
  end
  return out
end

If the idea is that doing this inside the API would give better performance, that would probably be slower. Changing context from C++ to Lua and back again, such as for invoking Lua callbacks, is often considerably slower than doing a Lua function call from inside Lua. The same issue does not apply in SpaceEngineers and Minecraft, where both mod code and the core engine are in the same language and runtime environment.

[Reika]

It's of course already possible to do this yourself:

Code: Select all

function find_entities_filtered_with_predicate(surface, args)
  local predicate = args.predicate
  local entities = surface.find_entities_filtered(args)
  local out = {}
  for _, entity in ipairs(entities) do
    if predicate(entity) then
      out[#out+1] = entity
    end
  end
  return out
end

If the idea is that doing this inside the API would give better performance, that would probably be slower. Changing context from C++ to Lua and back again, such as for invoking Lua callbacks, is often considerably slower than doing a Lua function call from inside Lua. The same issue does not apply in SpaceEngineers and Minecraft, where both mod code and the core engine are in the same language and runtime environment.

No, I ask for this feature based on experience. If you have many thousands of entities in a region that you do not care about - such as particle effects, corpses, etc - a generalized call to find_entities is extremely slow and causes a MASSIVE memory leak as the returned list grows to insane size.

I have multiple threads on this topic.

viewtopic.php?f=49&t=52253&p=305788#p305505

viewtopic.php?f=49&t=52298#p305823

[Rseding91]

Having find_entities_filtered accept a lua function as the filter is actually more expensive than doing find_entities_filtered and checking the results locally in Lua because it would have to create the Lua objects twice: once for the predicate check and once for the return result.

When you say "memory leak" what do you mean? A memory leak is memory that the program has allocated that it no longer has any reference to. If you run your code and exit back to the main menu does it still retain what ever memory was allocated while your mod was running?

[Therax]

No, I ask for this feature based on experience. If you have many thousands of entities in a region that you do not care about - such as particle effects, corpses, etc - a generalized call to find_entities is extremely slow and causes a MASSIVE memory leak as the returned list grows to insane size.

As noted in both the threads you linked, you'll have much better luck (and memory utilization) by calling find_entities_filtered type-by-type for each type you potentially want to examine.

For the particular case of your orbital strike weapon, you could get much better performance by collecting the list of entities in the affected area once, before firing the first shot, caching the list (in global if the firing sequence extends over multiple ticks), and reusing it for follow on shots. This would both reduce the number of calls to find_entities_filtered, and mean you don't ever have to scan over the particles created in the area by your first shot.

[Reika]

When you say "memory leak" what do you mean? A memory leak is memory that the program has allocated that it no longer has any reference to. If you run your code and exit back to the main menu does it still retain what ever memory was allocated while your mod was running?

I mean that the RAM used after a firing sequence is complete is drastically larger than it was before, such that a few dozen successive shots often crashes the game due to out of RAM errors (at least on my machine where Factorio has a total of 6GB or so to work with). Logging to the main menu releases this memory - usually - but going back into the save does NOT use it up again, meaning it is not tied to some part of the save itself (eg the global table).

I should mention that with some rewrites I did around the time 0.16 dropped, this problem is not as severe as it once was, but the RAM spike is still noticeable.

For the particular case of your orbital strike weapon, you could get much better performance by collecting the list of entities in the affected area once, before firing the first shot, caching the list (in global if the firing sequence extends over multiple ticks), and reusing it for follow on shots. This would both reduce the number of calls to find_entities_filtered, and mean you don't ever have to scan over the particles created in the area by your first shot.

No, this does not work, because the act of destroying spawners creates new biters - not in vanilla, but many mods, including NatEvo and NauvisDay do this - and because new biters can and often will run in from the surrounding area (killing spawners seems to send a 'call for help').

[Rseding91]

Can you give me a basic example mod that shows the high memory usage? I seem to remember something like that in the past so if you have a way to reproduce it I can look into fixing it.

It sounds like the Lua garbage collection isn't being aggressive enough when a large amount of new stuff is thrown into the Lua state and then quickly discarded.

[Reika]

Can you give me a basic example mod that shows the high memory usage? I seem to remember something like that in the past so if you have a way to reproduce it I can look into fixing it.

It sounds like the Lua garbage collection isn't being aggressive enough when a large amount of new stuff is thrown into the Lua state and then quickly discarded.

Some of the older downloads of EndgameCombat would be the first thing to try; the last build released for 0.15 was probably the one used in that initial thread.
https://mods.factorio.com/download/Endg ... 1024d72621

As stated, newer versions still have it, though less severely.

Also, I should mention that I have somewhat rewritten orbital-strikes.lua, which has not been released yet. However, the updated code is on my GitHub: https://github.com/ReikaKalseki/Endgame ... trikes.lua

[ownlyme]

find_entities_filtered should return nil if the table is empty

[Klonan]

Can you give me a basic example mod that shows the high memory usage? I seem to remember something like that in the past so if you have a way to reproduce it I can look into fixing it.

It sounds like the Lua garbage collection isn't being aggressive enough when a large amount of new stuff is thrown into the Lua state and then quickly discarded.

Some of the older downloads of EndgameCombat would be the first thing to try; the last build released for 0.15 was probably the one used in that initial thread.
https://mods.factorio.com/download/Endg ... 1024d72621

As stated, newer versions still have it, though less severely.

Also, I should mention that I have somewhat rewritten orbital-strikes.lua, which has not been released yet. However, the updated code is on my GitHub: https://github.com/ReikaKalseki/Endgame ... trikes.lua

The 'memory leak' is probably not related to the Lua side.

When you orbital strike the location, you kill a huge area of spawners, biters, worms etc.
To look pretty, when these die they create a 'blood-explosion', which creates particles each tick
And very soon there exists thousands if not 10s of thousands of particle entities in such a small area
Each particle needs to assign a non-insignificant amount of memory, and multiplied by 10,000, and you have your sudden RAM usage spike

My suggestion: Try looping through the blood explosions in data-updates and reducing the particle count by 10x, see if it makes a difference

[Reika]

Can you give me a basic example mod that shows the high memory usage? I seem to remember something like that in the past so if you have a way to reproduce it I can look into fixing it.

It sounds like the Lua garbage collection isn't being aggressive enough when a large amount of new stuff is thrown into the Lua state and then quickly discarded.

Some of the older downloads of EndgameCombat would be the first thing to try; the last build released for 0.15 was probably the one used in that initial thread.
https://mods.factorio.com/download/Endg ... 1024d72621

As stated, newer versions still have it, though less severely.

Also, I should mention that I have somewhat rewritten orbital-strikes.lua, which has not been released yet. However, the updated code is on my GitHub: https://github.com/ReikaKalseki/Endgame ... trikes.lua

The 'memory leak' is probably not related to the Lua side.

When you orbital strike the location, you kill a huge area of spawners, biters, worms etc.
To look pretty, when these die they create a 'blood-explosion', which creates particles each tick
And very soon there exists thousands if not 10s of thousands of particle entities in such a small area
Each particle needs to assign a non-insignificant amount of memory, and multiplied by 10,000, and you have your sudden RAM usage spike

My suggestion: Try looping through the blood explosions in data-updates and reducing the particle count by 10x, see if it makes a difference

I will try this, but A) why would that cause the RAM to not be recovered until logging to the main menu, as opposed to shortly after the particle entities are culled, and B) what can I do for release code, since reducing particles in all cases is clearly undesirable?

[mrvn]

Ram can only be freed at the end. So when you allocate 1000 blood drops and one iron plate and then free 1000 blood drops nothing can be freed to the system. Only when you also free the iron plate any memory can be returned. But the memory can be reused by the game, e.g. for the next iron plate.

So you might just see memory fragmentation effects.

[Nexela]

find_entities_filtered should return nil if the table is empty

Why? All that would do is add more lua code bloat

Code: Select all

for _ in pairs(s.find_entities_filtered{}) do

vs hacks which add an extra step like

Code: Select all

for _ in pairs(s.find_entities_filtered{} or {}) do

Code: Select all

local ents = s.find_entities_filtered{}
if s then
  for _, in pairs(s) do

[Therax]

No, this does not work, because the act of destroying spawners creates new biters - not in vanilla, but many mods, including NatEvo and NauvisDay do this - and because new biters can and often will run in from the surrounding area (killing spawners seems to send a 'call for help').

Could you do a hybrid: have the first shot look for all entities, and followup shots look only for type = "unit". This would get biters that enter or spawn in the area, without having to iterate buildings, particles, projectiles, etc.