Extreme memory leak, possibly my (mod) fault (help wanted)

[Reika]

Whenever I try to use my orbital strikes, the game RAM usage spikes enormously and never recovers. Ten or twelve strikes and Windows is frantically terminating processes to keep the OS stable. It may well be my fault, such as an ever-expanding global data table, but I have checked and not found such a leak. Another possible cause is the near-simultaneous creation of several dozen to several hundred FX entities - explosions, biter blood spatter, etc - which just "pile up", though the fact the RAM usage never drops again implies otherwise.

Extensive debugging seems to point to calling the entity.die() function - if you call that on 50+ entities at once, you get a massive RAM spike, and little recovery. Replace that with entity.destroy() and the problem mostly evaporates. Obviously, however, that is not acceptable for actual game use.

My relevant code is here:
https://github.com/ReikaKalseki/Endgame ... trikes.lua

[keyboardhack]

Can't reproduce it. Downloaded mod from your git with the latest commit being update 35. After a few hundred manual and 20 or so automatic orbital strikes, the game still only uses 1gb of ram.

[Nexela]

local entities = surface.find_entities_filtered({area = {{target.position.x-32, target.position.y-32}, {target.position.x+32, target.position.y+32}}})

This could be one problem area each subsequent call is finding more and more entities (mostly particles/smoke/etc is my guess) to filter through which will increase ram usage.

[Reika]

local entities = surface.find_entities_filtered({area = {{target.position.x-32, target.position.y-32}, {target.position.x+32, target.position.y+32}}})

This could be one problem area each subsequent call is finding more and more entities (mostly particles/smoke/etc is my guess) to filter through which will increase ram usage.

Should the list not be garbage collected as soon as the code is finished executing (mere microseconds later), though?

Can't reproduce it. Downloaded mod from your git with the latest commit being update 35. After a few hundred manual and 20 or so automatic orbital strikes, the game still only uses 1gb of ram.

My game uses 4GB of RAM normally, so something else must be affecting it.

[Nexela]

Should the list not be garbage collected as soon as the code is finished executing (mere microseconds later), though?

viewtopic.php?f=23&t=39498

[keyboardhack]

Can't reproduce it. Downloaded mod from your git with the latest commit being update 35. After a few hundred manual and 20 or so automatic orbital strikes, the game still only uses 1gb of ram.

My game uses 4GB of RAM normally, so something else must be affecting it.

Can you upload the save where it happens? Just tried it with my own save that uses ~3gb of ram and wasn't able to reproduce it there either.

[Reika]

Should the list not be garbage collected as soon as the code is finished executing (mere microseconds later), though?

viewtopic.php?f=23&t=39498

OK, what do I do?

Can't reproduce it. Downloaded mod from your git with the latest commit being update 35. After a few hundred manual and 20 or so automatic orbital strikes, the game still only uses 1gb of ram.

My game uses 4GB of RAM normally, so something else must be affecting it.

Can you upload the save where it happens? Just tried it with my own save that uses ~3gb of ram and wasn't able to reproduce it there either.

Here is a download. Note that the pack on mediafire is a couple days old, so use the public releases of EndgameCombat and FTweaks instead of the ones here. As for where to target, try a couple of the massive biter bases north of the western ore fields.
http://www.mediafire.com/file/wra6qga4x ... pand-3.zip
http://www.mediafire.com/file/a67qrbb5k ... 7_2017.rar

[keyboardhack]

I could never get the game to run out of memory but the game did lag a lot with your save.

The reason the game lags is because the function fireOrbitalWeaponManually goes through a lot of redundant entities. The first time it's called it goes through these enemies.

(Sorry about formatting but the forums won't allow two spaces after each other)

behemoth-biter 15 enemy
big-biter 22 enemy
big-worm-turret 5 enemy
biter-spawner 9 enemy
behemoth-worm-turret 4 enemy
spitter-spawner 2 enemy
orbital-bombardment-crater 1 neutral
big-spitter 2 enemy
explosion-remnants-particle 20 neutral
dead-grey-trunk 4 neutral
orbital-bombardment-shockwave 1 neutral
orbital-bombardment-explosion 1 neutral
orbital-bombardment-firing-sound 1 neutral
orbital-manual-target-effect 1 neutral
orbital-manual-target-secondary 2 player
medium-spitter 2 enemy
medium-biter 1 enemy

Only a few entities are redundant as you want to modify most of them so the list seems fine.
The issue is with the last strike. Below is the list of entities that the function goes through with the last strike.

blood-particle 71597 neutral
item-on-ground 151 neutral
behemoth-biter-corpse 17 neutral
blood-fountain 8 neutral
medium-biter-corpse 26 neutral
big-biter-corpse 85 neutral
big-worm-corpse 5 neutral
small-biter-corpse 25 neutral
small-spitter-corpse 2 neutral
smoke-explosion-particle 4630 neutral
explosion-remnants-particle 160 neutral
orbital-bombardment-crater 8 neutral
biter-spawner-corpse 9 neutral
huge-worm-corpse 4 neutral
spitter-spawner-corpse 2 neutral
big-spitter-corpse 14 neutral
orbital-bombardment-shockwave 3 neutral
orbital-bombardment-explosion 8 neutral
orbital-bombardment-firing-sound 2 neutral
orbital-manual-target-secondary 2 player
dead-grey-trunk 4 neutral
orbital-manual-target-effect 1 neutral
medium-spitter-corpse 3 neutral

Of all the ~75k entities you only want to do something to the 4 dead-grey-trunk.

Made a pull request that excludes the neutral force in that function(except trees)which should prevent all the useless entities from taking all the ram. With this, it's now bearable for me to shoot ~10 orbital shots at once.

[Reika]

I could never get the game to run out of memory but the game did lag a lot with your save.

The reason the game lags is because the function fireOrbitalWeaponManually goes through a lot of redundant entities. The first time it's called it goes through these enemies.

(Sorry about formatting but the forums won't allow two spaces after each other)

behemoth-biter 15 enemy
big-biter 22 enemy
big-worm-turret 5 enemy
biter-spawner 9 enemy
behemoth-worm-turret 4 enemy
spitter-spawner 2 enemy
orbital-bombardment-crater 1 neutral
big-spitter 2 enemy
explosion-remnants-particle 20 neutral
dead-grey-trunk 4 neutral
orbital-bombardment-shockwave 1 neutral
orbital-bombardment-explosion 1 neutral
orbital-bombardment-firing-sound 1 neutral
orbital-manual-target-effect 1 neutral
orbital-manual-target-secondary 2 player
medium-spitter 2 enemy
medium-biter 1 enemy

Only a few entities are redundant as you want to modify most of them so the list seems fine.
The issue is with the last strike. Below is the list of entities that the function goes through with the last strike.

blood-particle 71597 neutral
item-on-ground 151 neutral
behemoth-biter-corpse 17 neutral
blood-fountain 8 neutral
medium-biter-corpse 26 neutral
big-biter-corpse 85 neutral
big-worm-corpse 5 neutral
small-biter-corpse 25 neutral
small-spitter-corpse 2 neutral
smoke-explosion-particle 4630 neutral
explosion-remnants-particle 160 neutral
orbital-bombardment-crater 8 neutral
biter-spawner-corpse 9 neutral
huge-worm-corpse 4 neutral
spitter-spawner-corpse 2 neutral
big-spitter-corpse 14 neutral
orbital-bombardment-shockwave 3 neutral
orbital-bombardment-explosion 8 neutral
orbital-bombardment-firing-sound 2 neutral
orbital-manual-target-secondary 2 player
dead-grey-trunk 4 neutral
orbital-manual-target-effect 1 neutral
medium-spitter-corpse 3 neutral

Of all the ~75k entities you only want to do something to the 4 dead-grey-trunk.

Made a pull request that excludes the neutral force in that function(except trees)which should prevent all the useless entities from taking all the ram. With this, it's now bearable for me to shoot ~10 orbital shots at once.

Your solution involves turning one find_entities_filtered into many, which I avoided due to the - I was told - extreme performance cost of ever calling that, let alone repeatedly. However, if it can be called several times without issue, would it not be more effective to just call it type-by-type (unit, turret, etc)?

EDIT: Never mind, I forgot about the fact it also damages infrastructure, and thus many many different types.

Still, excluding the neutral force is an undesired behavior, as there are - with some mods, such as Ruins - hostile neutral structures, some of which contain turrets. And I could just start adding more scans, but again, if there are 75k entities on the surface in the area, does that not mean that a find_entities_filtered will have to iterate through all of them every time?

[keyboardhack]

Your solution involves turning one find_entities_filtered into many, which I avoided due to the - I was told - extreme performance cost of ever calling that, let alone repeatedly. However, if it can be called several times without issue, would it not be more effective to just call it type-by-type (unit, turret, etc)?

Also, excluding the neutral force is a mildly undesired behavior, as there are - with some mods, such as Ruins - hostile neutral structures, some of which contain turrets.

For the most part it's better with fewer calls to find_entities_filtered but in this case it isn't the call itself that is slow. It's iterating over all entities in the returned list that is so slow. I assume the mod wants to damage all entities within the area which has health but there is no easy way to do that. My solution isn't perfect(i didn't think about neutral structures) but i believe(haven't tested) calling find_entities_filtered for each type will be just as slow as the current solution.

I think the best solution will be to request a new mod API function. Something like find_health_entities_filtered that only returns entities that has health, although it will probably first be implemented in 0.16.

[Reika]

Your solution involves turning one find_entities_filtered into many, which I avoided due to the - I was told - extreme performance cost of ever calling that, let alone repeatedly. However, if it can be called several times without issue, would it not be more effective to just call it type-by-type (unit, turret, etc)?

Also, excluding the neutral force is a mildly undesired behavior, as there are - with some mods, such as Ruins - hostile neutral structures, some of which contain turrets.

For the most part it's better with fewer calls to find_entities_filtered but in this case it isn't the call itself that is slow. It's iterating over all entities in the returned list that is so slow. I assume the mod wants to damage all entities within the area which has health but there is no easy way to do that. My solution isn't perfect(i didn't think about neutral structures) but i believe(haven't tested) calling find_entities_filtered for each type will be just as slow as the current solution.

I think the best solution will be to request a new mod API function. Something like find_health_entities_filtered that only returns entities that has health, although it will probably first be implemented in 0.16.

All of these hinge on the internal implementation of find_entities type functions; if they work like Minecraft, where the check is only used when testing to add an entity to a list, it will be extremely slow just to call the function, no matter how few entities are returned. Also, in the above linked thread, the "memory fragmentation" may apply in both cases as well.

I have asked about the internal implementation to see if this is true: viewtopic.php?f=49&t=52298

[Reika]

I just tested a slightly modified version of your code, and though the memory leak looks to be fixed, given the lack of a large spike each strike on the task manager, I still ran out of RAM after several dozen, including a Factorio CTD and Windows closing half a dozen other open programs. Oddly, however, this was accompanied by the first spike on the RAM graph:


I suppose now is a good time to mention that I am on a Windows 7 system with 12GB of RAM and, importantly, no paging file.

[Koub]

I am on a Windows 7 system with 12GB of RAM and, importantly, no paging file.

Have you considered reenabling the system paging file, and let Windows work the way it was designed ?

[Reika]

I am on a Windows 7 system with 12GB of RAM and, importantly, no paging file.

Have you considered reenabling the system paging file, and let Windows work the way it was designed ?

I have a 120GB SSD. Enabling paging will both fill up all remaining space on the drive and dramatically shorten the drive's life due to extreme read/write use.

[Koub]

You don't have to create your pagefile on the SSD. You can create it on your storage disk. Or are you telling me you have only a 120 GB SSD, and no additional storage space on a HDD ?

[impetus maximus]

i've disabled the page file system in the past and for some games it's just a bad idea.
even if you have more than enough physical memory. buy a small cheap SSD for it if you don't want it on your main drive(s).

[torne]

I have a 120GB SSD. Enabling paging will both fill up all remaining space on the drive and dramatically shorten the drive's life due to extreme read/write use.

While the space concern is a great point, unless your SSD is from the early days of SSDs being a thing (pre-2011 or so) then paging to it is not going to make any measurable difference to its lifetime, especially on a machine with 12GB of RAM (so paging is relatively light and infrequent). (also.. it's sadly still the case that a large proportion of SSD failures are not due to wear/usage at all, but bugs in the controller implementation causing the drive to become unusable under specific/random conditions unrelated to wear).

[Reika]

I have a 120GB SSD. Enabling paging will both fill up all remaining space on the drive and dramatically shorten the drive's life due to extreme read/write use.

While the space concern is a great point, unless your SSD is from the early days of SSDs being a thing (pre-2011 or so) then paging to it is not going to make any measurable difference to its lifetime, especially on a machine with 12GB of RAM (so paging is relatively light and infrequent). (also.. it's sadly still the case that a large proportion of SSD failures are not due to wear/usage at all, but bugs in the controller implementation causing the drive to become unusable under specific/random conditions unrelated to wear).

This drive (OCZ-AGILITY3) was purchased in mid 2013, I believe. However the drivers date from 2006, so the design might predate my purchase by a couple of years.

You don't have to create your pagefile on the SSD. You can create it on your storage disk. Or are you telling me you have only a 120 GB SSD, and no additional storage space on a HDD ?

That is exactly what I mean...almost. I do have an external 1TB HDD, but it is USB-connected and I have been told repeatedly that that connection is far too slow to even consider use for a pagefile.

i've disabled the page file system in the past and for some games it's just a bad idea.
even if you have more than enough physical memory. buy a small cheap SSD for it if you don't want it on your main drive(s).

I have a second SSD, but have not been able to install it because I lack the motor control to do it myself and am not interested in taking the entire computer into a shop, having to wait at least one full work day to get it back, being charged $400 for the install, and finding out that they have spent most of that time - which I was still billed for - rifling through the filesystem (example).

[keyboardhack]

I just tested a slightly modified version of your code, and though the memory leak looks to be fixed, given the lack of a large spike each strike on the task manager, I still ran out of RAM after several dozen, including a Factorio CTD and Windows closing half a dozen other open programs. Oddly, however, this was accompanied by the first spike on the RAM graph:

I suppose now is a good time to mention that I am on a Windows 7 system with 12GB of RAM and, importantly, no paging file.

At this point it's quite obvious that this is an optimization issue and not a memory leak. There is plenty ways to optimize the code and i am sure you can do it quite easily. For future reference, creating a 75k+ table of entities in lua every 5 ticks is not recommended if you want decent time and ram usage.