[Rseding91] [2.0.15] Demolishers take a disproportionate amount of cpu time

[djugei]

Space Age has somewhat bad performance on my device, so i investigated a bit.
the show-entity-time-usage debug view shows that a disproportionate amount of time is used for AsteroidCollector and SmokeWithTrigger, additionally a big amount of time is used for Inserter.
This is about SmokeWithTrigger. Investigating the game files i found a rather low amount of mentions of smoke-with-trigger. Specifically found some related to demolishers.
To test my theory of demolisher involvement i killed all demolishers using a command. this made SmokeWithTrigger take 0ms in updates and made factorio --benchmark go from 15 to 13 ms.

It therefore seems that demolishers cause a big amount of cpu work even while not useful.

proposals to fix:
* only spawn the smoke if a player or player-controlled entity is close by, alternatively spawn graphical (no-effect) smoke all the time and then switch it once players get close to preserve visuals.
* reduce the amount of time the checks take. right now i assume that each smoke cloud checks each entity for being an enemy and not having the slowdown effect yet. specifically this would consider each segment of the demolisher. to fix that either some faster/better filtering could be used, or, since unlike poison capsules the effect does not stack, a single check could be done for each group of smokes, or both.

[Sakari119]

An extension of the proposal is as follows;
A hacky solution is that the generation of the volcanic ash and magma trail is not needed when there are no player controlled entities.

11-08-2024, 17-42-13.png (112.47 KiB) Viewed 1459 times

Example of behavior where the blue boxed star represents a player structure, and red represents active worms with trails. Grey represents worms without trails been created.

[Rseding91]

Can you provide a save file that I can test with?

[Wadiyatalkinabeet]

Hi Rseding91,

I have noticed this happening on my save (attached)
After killing all demolishers with a command, the ms/tick drops from ~13 to ~8

11-10-2024, 10-29-14.png (644.42 KiB) Viewed 1275 times

->

11-10-2024, 10-30-10.png (619.16 KiB) Viewed 1275 times

[djugei]

thanks for providing an example, here is the original one, it still has the same issue even though vulcanus is more round.

first upload did not work out, so i deleted surfaces and blueprints, its about half the size now and the upload worked.

[Caddrel]

Very useful thread, I was also wondering also what "SmokeWithTrigger" was.

On our game it's taking up a non-trivial amount of update time: viewtopic.php?f=49&t=121399

(admittedly there are lots of other things taking far more)

Save file attached to that thread. We also have the other culprits mentioned in the OP (inserters and asteroid collectors).

SmokeWithTrigger is taking up the same time as all the assemblers we have going, apparently.

[GregoriusT]

Still the case in 2.0.20 by the way. One of my Friends is having this exact Issue causing Lag.

[OvermindDL1]

Currently [2.0.20], SmokeWithTrigger is eating around 1.2ms when normal in the game, and spikes to over 3ms (over 5ms in a few cases) when someone decides to go attack a demolisher or more and persists until a minute or so after they're dead. That's enough to cause it to drop below 60UPS for some clients, impacting their framerate down to the low single-digits before the UPS itself starts faltering and they build substantial buffer until the game has to be paused for them to catch up or the source of this is killed off and then wait a minute after, though by that point there's a lot of 'wait' built up and a large buffer so interacting with game is rather painful and jerky and extremely delayed when they are trying to, say, run for their life and their character doesn't want to move. This is on computers that handled, for example, a couple of dozen settled planets with very large active factories on ~12 of them with ~40 active ships in Factorio 1.1 Space Exploration without dipping below 60UPS. Even when not under attack the UPS is sustaining at 60UPS on those clients, however their FPS is dropping to between 15 to 35 FPS to keep the UPS at 60 (16 core systems at that, though it seems only 2 cores are in decently heavy use by factorio).

We've been trying to go on a long distance rampage to kill demolishers (attracting via artillery and killing at our base) without spawning more territory to help, and it does seem to be doing though, albeit slowly (there's multiple dozens of demolishers active on Vulcanus currently due to earlier exploration). The server is keeping up (barely it seems, but it is), just some clients aren't even though they aren't on that old of systems (as indicated above).

Closely watching the timings is what led us to the SmokeWithTrigger values spiking at certain times that correlated with the intense lag, and searching on these forums led us to that being the 'cloud' effects at the demolishers, which is what led to us starting to slaughter them from a distance (to not spawn more) and confirming that them using those clouds in force substantially effects UPS. Considering the persistent SmokeWithTrigger costs happening when one is not being attacked, I'm theorizing that they have such an effect around them in general?

SmokeWithTrigger is not the only high value, but it is the highest. The next highest entity timing is inserters and then assemblers at around 1.5ms and 1.3ms, but they don't jerk around and are fairly consistent. Electricity network timing is also up there at 1.2ms if I recall right (we know to keep each surface to 1 electrical network only as spamming electrical networks is a surefire way to lag the game to death, at least in 1.1 and before).

[TheCherry]

Currently [2.0.20], SmokeWithTrigger is eating around 1.2ms when normal in the game, and spikes to over 3ms (over 5ms in a few cases) when someone decides to
....

We have exactly the same issues. Yesterday we reached vulcano and explored the planet, if we come now near to a big demolisher, my FPS drops to 5.
Overall is the performance very bad atm.

11-19-2024, 11-02-11.png (11.44 KiB) Viewed 930 times

Added our savefile

[TheCherry]

I looked into the code, I think you should give the demolisher a extra simple hitbox, this clouds with all that triggers and low cooldowns is very overshot.
I am not a game dev and I am not deep in the factorio modding, but this cloud stuff with expending and repeats - combined with a nested collision actions seems not a good way to design it. I think you have to refactor this part and add very simple hitboxes around that enemy that just stick to it.

I don't know if that helps. Sry if not.

[masafi]

+1 for the bug, I'm playing on a laptop and smoke with trigger takes around the same time as inserters (~1.5ms) (while I'm on navius too...), it tanks my already poor performance a lot

[Rseding91]

We’ve identified the reason they’re taking as much CPU time as they are but we haven’t had time to work on them yet. Crashes, desyncs, and other game features not working correctly have taken priority.

[BlueTemplar]

[...]
We've been trying to go on a long distance rampage to kill demolishers (attracting via artillery and killing at our base) without spawning more territory to help, and it does seem to be doing though, albeit slowly (there's multiple dozens of demolishers active on Vulcanus currently due to earlier exploration). The server is keeping up (barely it seems, but it is), just some clients aren't even though they aren't on that old of systems (as indicated above).
[...]

Meanwhile, sounds like despawning chunks (hopefully taking demolishers with them) might help here ?
mods or commands

[OvermindDL1]

Sadly no mods or commands here, it's a completely vanilla server with no admin access to /c (to keep achievements for those that like going for them).

And yeah, as demolishers are killed, more ones spawn further out, and they all load back in when the game is loaded (didn't it used to be just 'active' chunks loaded back in, like ones with pollution or 'near to player entities', not "all" like vulcanus seems to do?).

EDIT: But yeah, I can't really play well on it anymore, my 'buffer' size just kind of constantly grows as my FPS drops to 6fps (guessing 10% is a cutoff internally of reduction?) while UPS hovers around like 58UPS until I eventually drop. Which is funny because the computer running the client is both more powerful and has more cores (and is much newer) than the dedicated server (which I picked specifically because it was slow just so it would slow down before dropping other people, which isn't happening as it keeps up fine...)

[djugei]

We’ve identified the reason they’re taking as much CPU time as they are but we haven’t had time to work on them yet. Crashes, desyncs, and other game features not working correctly have taken priority.

hey totally understand that priorization, but just to be clear: this is almost the entire entity update time early game (~60 spm, nauvis, vulcanus, gleba).
lategame it is about 10%. so this is not just some minor performance issue.

[manishearth]

Just to add a datapoint for prioritization: This bug is equivalently as game-breaking as a crash for me, since my FPS on our multiplayer world is consistently <5 (it's 60fps when loading and it rapidly drops). At this level the inputs start overlapping so almost any click I make will get executed seconds later with all of its sounds overlapping each other, and often at the wrong time with respect to other inputs. Basically, I have to be really careful what I do in case I mess something up (because I can't necessarily know what the screen state will be when the input is actually processed).

I'm at best able to join Factorio to _look_ at what my friends have built (even that is painfully laggy, but hard to get wrong), but large lags on input make it basically impossible to do anything useful. I've not really played for the last two weeks because of this.

It's possible that the FPS issue is a confluence of multiple issues including this demolisher one, but the demolishers are clearly a large part of it from the debug menu.

[Rseding91]

You can simply have your friends kill all of the demolishers if it's that big of an issue for you. Or slow down the game speed to allow your computer to keep up.

The root issue is just that *they take a lot of CPU time* because they always exist once uncovered until killed. That was part of their design doc so I can't simply just make them not do that or it breaks how they were intended to work (slowly patrol their area until killed). Some people explored on Vulcanus and generated a lot more worms and so have a lot of them going at all times. That wasn't thought about in their design and that's the issue that needs to be re-thought about and handled - if it even can be.

[GregoriusT]

You can simply have your friends kill all of the demolishers if it's that big of an issue for you. Or slow down the game speed to allow your computer to keep up.

The root issue is just that *they take a lot of CPU time* because they always exist once uncovered until killed. That was part of their design doc so I can't simply just make them not do that or it breaks how they were intended to work (slowly patrol their area until killed). Some people explored on Vulcanus and generated a lot more worms and so have a lot of them going at all times. That wasn't thought about in their design and that's the issue that needs to be re-thought about and handled - if it even can be.

1. Isn't the Issue technically the Collideable Clouds being spammed by the Demolishers (which they do unconditionally), and not the mere existence of Demolishers themselves?

2. You basically must have Debug Overlays on to detect the Demolishers in those non-radar-ed chunks (radar would generate even more), and attack them with Artillery to prevent more Chunks from being generated. Which if you only needed one shot to aggro them would be fine, but you need to focus on where the demolisher goes and shoot it more once it loses aggro again, otherwise it wont run into your Trap.