Performance problem with pathing

[cliff_build]

During a recent Biter Battles game, performance completely cratered for ~30 seconds at one point (everything went to ~10 UPS for a while, and all clients with slower computers were eventually dropped for inability to keep up). I grabbed a save in the middle of this: https://drive.google.com/file/d/1CjDLvw ... sp=sharing and the slowness appears to all be in "entity update".

There are no mods loaded, and this is running the custom Biter Battles scenario code. I would love to know if this is some underlying issue with the pathfinder, or if we have somehow configured things poorly in the scenario.

The slowness is all coming from one group of biters (verified by deleting those biters in /editor mode) that is in the north-east under the fog of war. The biters are trying to navigate around/through some water and wall to get to some turrets.
I will note that we do slightly change some pathfinder settings, as visible here: https://github.com/Factorio-Biter-Battl ... /502/files Specifically, for an attack group like this, we are setting "unit.ai_settings.path_resolution_modifier = -1"

Group of biters that cause slowness

Screenshot of biter group.png (9.44 MiB) Viewed 1312 times

.

Requested/required screenshot for performance issues

Screenshot 2024-05-11 at 4.21.58 PM.png (2.94 MiB) Viewed 1312 times

I am also including a screenshot with "show-detailed-info", which at least to me implies that the pathfinder is not doing huge amounts of work (we have had issues in the past where "Pather total: calls: xxx" shows a number in the millions that corresponds with lots of slowdown, but that doesn't seem to be happening here.

show detailed info.png (167.56 KiB) Viewed 1312 times

[ness056]

Hello,
I just looked at the save. The problem is not the path finder, it is the stickers (taking 70ms per tick on averge on my computer). I first thought that the fire-spread would cause the lag, but disabling it with a mod doesn't change much. So I tried to make the damage interval bigger (to 90 ticks instead of 10) and it did reduce the lag. However, by playing the game tick per tick, the stickers were still making lag, either 0ms or a multiple of 7ms depending on the tick, this makes me think that each individual damage tick is taking 7ms to compute, tho I have absolutely no idea why it would take that long to apply damage to the biters. It would be nice if someone with deeper understanding of how the stickers work internally of the game engine can take a look.

Thank you in advance.

[Rseding91]

This looks like it might be related to the scenario you're using and it changing path-finder settings. Every time one of the biters is damaged it calls for reinforcements which triggers a re-path.

[cliff_build]

So it sounds like it could in fact be the pathfinder in some capacity, if the stickers are causing damage, and that damage is causing new path requests.

While I agree that, in theory, this could be due to the changes that our scenario makes to the pathfinder settings, if you actually load this save and watch it, the biters are completely stuck and performance is terrible, and then magically after some number of seconds things recover and the biters start moving. I don't even understand how changing the pathfinder settings could result in behavior like this. I also tried changing many of the pathfinder settings back to their default values in the save and it didn't appear to make any difference to the performance.

So I don't really understand how I can fix this, and it really feels to me like the underlying engine has some issue preventing pathfinding from working properly (the biters are just staying 100% still while they take damage).

[Rseding91]

I’ll look into it more tomorrow but from what I saw it seems like the cache count has been increased by a factor of 100, and then every time the biters are damaged they request a new path to the target. But before the new path can be completed they are damaged again and the old request is thrown out and a new one placed.

The slow part comes from the cache being so large it has to go over each cached entry to see if one is valid to use.

[cliff_build]

The slow part comes from the cache being so large it has to go over each cached entry to see if one is valid to use.

Thank you for this explanation! We are moving our settings to be closer to vanilla: https://github.com/Factorio-Biter-Battl ... s/pull/527 We will likely still struggle when lots of biters all start pathing towards flamethrowers (which, in at least some tests, the larger cache size really helped with), but so it goes.

[Rseding91]

I finally got around to looking at this in the profiler in more detail and can confirm my initial findings are accurate.

There are some possibilities for improvement but it will still be quite slow with the amount of large paths cached. I also need to look into why the biters keep re-pathing when they already have an active request.

[cliff_build]

Thanks for the continued insights!

If you want an easy-to-reproduce case of this slowdown, we actually have a benchmark/performance testing framework that is currently demonstrating this behavior reliably.

If you use ./benchmarking/run_benchmark.sh from https://github.com/Factorio-Biter-Battl ... er-Battles it will demonstrate it reproducibly/quickly (here is a video of proof https://drive.google.com/file/d/1p5IzAa ... sp=sharing I'm sorry that the video is 600MB, I don't know an easy way to make it smaller) It shows my machine getting 15 UPS while the crazy pathfinding behavior is ongoing, and then goes back to ~60 UPS afterward.

If you look at benchmarking/main.lua, you can see the code that sets up the scenario/triggers lots of attacks.

Anyhow, you don't need to do any more looking/investigation - I just thought that I would pass this reproduction on in case it was helpful to you.