[MOD 0.18] Robot Army. v0.4.4

[kyranzor]

Congrats on being post #500 Gazer!

It looks like your game update is dominated by "entity update" and "factorissimo", taking ~7.5ms between them (5.5 from entity update, 2.5 from factorissimo), and during the "lagging" condition, robot army is adding an additional 2ms to the update time. When "not lagging" robot army is only 120 microseconds or so.

Basically, your game has reached a "tipping point", where fluctations in ANY mod, will begin to encroach on the ideal <= 16ms update time for your game to maintain 60 UPS.


As was mentioned earlier, updating my AI spread out over 60 ticks instead of doing ALL squad AI updates in a single tick, every so often, will help deal with "spikes", however there will come a time when the load just gets too great for whatever computer is running the game itself + all the other mods added (including mine).

I will be trialing the "update only a few squads per tick" method soon, and when it's ready on the development branch i'll let you guys know.

[iceman_1212]

is there a game command would let me delete all rally posts across the map? am playing with your (awesome awesome) mod for the first time and didn't quite understand how rally posts worked initially and now my robots chill out at the posts instead of attacking biters

[kyranzor]

is there a game command would let me delete all rally posts across the map? am playing with your (awesome awesome) mod for the first time and didn't quite understand how rally posts worked initially and now my robots chill out at the posts instead of attacking biters

Try this:
/c for i,j in pairs(game.player.surface.find_entities_filtered({name="rally-beacon", force=game.player.force})) do if (j and j.valid) then j.destroy() end end

[iceman_1212]

is there a game command would let me delete all rally posts across the map? am playing with your (awesome awesome) mod for the first time and didn't quite understand how rally posts worked initially and now my robots chill out at the posts instead of attacking biters

Try this:
/c for i,j in pairs(game.player.surface.find_entities_filtered({name="rally-beacon", force=game.player.force})) do if (j and j.valid) then j.destroy() end end

ty sir, worked like a charm

[dauphin]

Finally got a chance to try out this mod, and I don't think I can go back to the way things used to be. It's just so much more fun to automate war than it is to fight it in person.

Question: are there any known issues that would cause large groups of terminator bots to basically just sit in one location for 10 minutes or more? They seem to move *eventually*, but they're definitely not continually searching out new targets. Also, it seems like sometimes they stop revealing their location on the map.

At one point I got up to >500 Terminator bots, and this caused some pretty significant lag spikes about once a second. I see you're working on splitting the pathfinding across all 60 ticks instead of doing it every 60th - sounds like a major improvement.

I don't think the pathfinding lag is necessarily the cause of the overly passive behavior, but I figured I may as well mention it.


I'm a software developer, and though I don't have any experience with Factorio modding, I bet I could help with debugging things if you wanted to point me in the right direction.

[kyranzor]

Finally got a chance to try out this mod, and I don't think I can go back to the way things used to be. It's just so much more fun to automate war than it is to fight it in person.

Question: are there any known issues that would cause large groups of terminator bots to basically just sit in one location for 10 minutes or more? They seem to move *eventually*, but they're definitely not continually searching out new targets. Also, it seems like sometimes they stop revealing their location on the map.

At one point I got up to >500 Terminator bots, and this caused some pretty significant lag spikes about once a second. I see you're working on splitting the pathfinding across all 60 ticks instead of doing it every 60th - sounds like a major improvement.

I don't think the pathfinding lag is necessarily the cause of the overly passive behavior, but I figured I may as well mention it.


I'm a software developer, and though I don't have any experience with Factorio modding, I bet I could help with debugging things if you wanted to point me in the right direction.

were the squads deployed far away from enemies? they will sit around for a while until they see an enemy enter their hunt radius (centered on the squad). The next issue can be no more biters - if they have cleared out all biters on that part of the map, unless you personally go over to them, no new chunks will generate in that area which means there will be no new biters to kill. You can literally genocide the entire map as long as you don't keep expanding and exploring.

Another thing that i've noticed can cause long delays and strange behaviour in squads is if one of the squad members is stuck on something or on your base walls or in conveyor belts. if a squad member is stuck the squad will have problems - disbanding and reforming, not leaving for their destination (because they want to form up /regroup/go in formation)

[Optera]

Another thing that i've noticed can cause long delays and strange behaviour in squads is if one of the squad members is stuck on something or on your base walls or in conveyor belts. if a squad member is stuck the squad will have problems - disbanding and reforming, not leaving for their destination (because they want to form up /regroup/go in formation)

Is there a fix for that behavior? One of my maps has a lot of medium to large lakes with very rough coastlines. Squads almost always get stuck trying to reach the other side.
Maybe add giving up after x tries to reach target and return to base.

[kyranzor]

Well, it's a problem with the base game's pathfinder - Oxyd supposedly fixed biters (and therefore, droids) pathfinding issues around lakes and other obstacles but perhaps there are still lingering issues with it.

I can make sure that "start_moving()" command is always called after giving a squad a move command so that it skips trying to assemble the squad before moving to the destination. But the coherency of the squad is important because if the members spread out too far, the algorithms I wrote that use the unit group's "position" gets less and less accurate. The "position" of the unit group is a bit crazy even in normal circumstances, but if the squad members are scattered all over the place, the position that is reported is extremely bad quality.

For moving from A to B over large distances though, the position accuracy doesn't really matter much.

So Optera, there are two major issues I can think of - 1 is the pathfinding in general around things like lakes, and Oxyd (Developer) supposedly fixed this in some recent patches. 2 is the fact that if a squad member gets stuck, the rest of the squad may attempt to wait for them and will not move until they have caught up. This can be fixed with forcing the squad to move using start_moving() function after set_command function in my code, which I will check out and put on the Development branch for testing.

I am working on some rather large code changes for performance (the squad AI spread out over 60 ticks instead of doing all updates every n'th tick) so I can't say when i'll have a stable development version for you to test.

edit: i just had a look at my code, and i'm using start_moving everywhere already, so this can't be the issue. damn!

[kyranzor]

Could i get people to try the development branch and test for performance improvements with my first attempt at the squad AI tick schedule method? https://github.com/kyranf/robotarmyfact ... opment.zip

You'll need to change the folder of the Robot Army mod to 0.2.4

A migration script will run and move any existing squads over into the new tick update table system.

[dauphin]

were the squads deployed far away from enemies? they will sit around for a while until they see an enemy enter their hunt radius (centered on the squad). The next issue can be no more biters - if they have cleared out all biters on that part of the map, unless you personally go over to them, no new chunks will generate in that area which means there will be no new biters to kill. You can literally genocide the entire map as long as you don't keep expanding and exploring.

Another thing that i've noticed can cause long delays and strange behaviour in squads is if one of the squad members is stuck on something or on your base walls or in conveyor belts. if a squad member is stuck the squad will have problems - disbanding and reforming, not leaving for their destination (because they want to form up /regroup/go in formation)

Having played a few more hours with all of this, I can confirm that both of these things are affected the squads. I did increase the hunt range all the way to 10000, but apparently even that wasn't quite large enough for some of my squads. Also, the squads really get confused if they get near belts. But the mod description does warn about their allergy to pasta, so that's on me.

Also, at one point they very nearly did wipe out all of the biters. I'm using RSO and the default expansion parameters are waaaay too slow. So I manually set them to respawn between 5 minutes and 15 minutes and the biters came back with a vengeance.


The somewhat unrelated things I've noticed:
1) sometimes a squad gets down to one or two members, and they end up just kinda stranded in the middle of nowhere. I usually go around the map and pick these guys up after a while.
2) every once in a while a biter spawns on an island. (This is annoying on its own, but...) Then it seems like the squads will sometimes get 'stuck' trying to path their way to that island. I would have thought the default pathfinding algorithm would've been smart enough to fail at that... but maybe your code that picks the next target is overriding and continuing to order them to try to get there. Is is possible to check for the availability of a path to the target before actually picking them as a target?
3) Lastly, as more of a feature request - is there any way the targeting algorithm could be something of a hybrid? Instead of just targeting the very closest biters, would it be possible for the squads to prioritize biters that are closer to player structures/bases than the squad currently is? Sometimes the squads end up plowing a path deep into the wilds, but in the meantime there are biters building closer and closer to my factories. I know this might be complicated, so it's just a thought.


I will definitely download that development branch and I'll see if I can push past the ~150 droid mark, which is where the pathfinding starts to slow down on my computer.

This is easily the best mod I've played with so far in terms of the value added to play. Coupled with RSO and some more aggressive biter settings, this is the endgame I've been looking for since I started playing. Keep up the good work, and let me know if you ever want a dev partner. Though.... it's been a while since I worked on code that uses tabs for indentation.

[kyranzor]

The somewhat unrelated things I've noticed:
1) sometimes a squad gets down to one or two members, and they end up just kinda stranded in the middle of nowhere. I usually go around the map and pick these guys up after a while.
2) every once in a while a biter spawns on an island. (This is annoying on its own, but...) Then it seems like the squads will sometimes get 'stuck' trying to path their way to that island. I would have thought the default pathfinding algorithm would've been smart enough to fail at that... but maybe your code that picks the next target is overriding and continuing to order them to try to get there. Is is possible to check for the availability of a path to the target before actually picking them as a target?
3) Lastly, as more of a feature request - is there any way the targeting algorithm could be something of a hybrid? Instead of just targeting the very closest biters, would it be possible for the squads to prioritize biters that are closer to player structures/bases than the squad currently is? Sometimes the squads end up plowing a path deep into the wilds, but in the meantime there are biters building closer and closer to my factories. I know this might be complicated, so it's just a thought.


I will definitely download that development branch and I'll see if I can push past the ~150 droid mark, which is where the pathfinding starts to slow down on my computer.

This is easily the best mod I've played with so far in terms of the value added to play. Coupled with RSO and some more aggressive biter settings, this is the endgame I've been looking for since I started playing. Keep up the good work, and let me know if you ever want a dev partner. Though.... it's been a while since I worked on code that uses tabs for indentation.

So yes, if you deploy too far away, they will stand around (causing more lag because they are actively searching that huge radius every time their update is called). If a squad finds a target and moves towards it, you get much less CPU strain because i don't try to search for new targets or re-assign them if they already have a task.

1) perhaps they don't have a droid assembler to retreat to? The retreat setting is 2 i think, by default, so perhaps they cannot find a path or a valid droid assembler to return to. This can happen if you only ever deploy by hand, or if you deployed by assembler but then removed the droid assemblers afterwards (maybe for relocating?) but either way as soon as there is a valid one (no distance requirement) they should go towards it if "retreating".
2) the mod API gives no way to check if the pathfinder subsystem can find a path to an island or not.. that is unfortunate, and a case I didn't think about. I'm not sure what to do about it. There might be a way I can detect this but doing it efficiently and without gross logic might be hard.
3) This plays in to the issue talked about earlier on this thread about how instead of the hunt radius being squad-centered, it could be centered on the assembler that produced the squad. Then it's kind of like an aggressive "guard" assembler. Current behaviour of the squads is literally to just blindly slaughter/rampage their way through the nearest enemies until they die, which indeed if they have a target-rich environment, may cause them to delve deep into enemy territory.


I was saying before that a complete re-write of the mod's method of deployment and organisation/AI of the squads so that it's assembler-centric at all times would solve this, at least a little bit - and maybe even disable the player's ability to place droids by hand altogether. It might be more intuitive that way.

I can have upwards of 1200 droids actively slaughtering biters, before I see any performance impact.. maybe my computer is just a beast? I hope the improved AI handling helps your performance

"This is easily the best mod I've played with so far in terms of the value added to play" Thanks, and supposedly my mod has sparked conversation at Wube HQ so maybe they'll implement it (and better!) in the base game??

Lua using indenting is disgusting, I hate it. If this was python (but, compiled, for speed) I would be much happier. I am always interested in having help on the mod, there is plenty of code clean-up, refactoring, new concepts, and integrating the work that another guy has done for me recently on the new technology tree for Robot Army that makes the tech mod-unique and has multiple levels of damage tech for each droid type.

[Optera]

I was saying before that a complete re-write of the mod's method of deployment and organisation/AI of the squads so that it's assembler-centric at all times would solve this, at least a little bit - and maybe even disable the player's ability to place droids by hand altogether. It might be more intuitive that way.

I would love to see assembler-centered hunting radius.
One thing keeping me from changing my defense completely to bots is their habit of plowing forever into biter territory, ignoring nests close to my borders.
Another thing is their knowledge of biters in areas i haven't revealed. They should only attack bases they can know of aka being shown on map.
Perhaps add a new exploration mode where they simply wander around their assembler looking for prey (though radars probably are still better at that)

[Nexela]

I was saying before that a complete re-write of the mod's method of deployment and organisation/AI of the squads so that it's assembler-centric at all times would solve this, at least a little bit - and maybe even disable the player's ability to place droids by hand altogether. It might be more intuitive that way.

I would love to see assembler-centered hunting radius.
One thing keeping me from changing my defense completely to bots is their habit of plowing forever into biter territory, ignoring nests close to my borders.
Another thing is their knowledge of biters in areas i haven't revealed. They should only attack bases they can know of aka being shown on map.
Perhaps add a new exploration mode where they simply wander around their assembler looking for prey (though radars probably are still better at that)

I also think being deployer centric (or player or player position centric when placed by hand) would be the better choice also.

[dauphin]

So yes, if you deploy too far away, they will stand around (causing more lag because they are actively searching that huge radius every time their update is called). If a squad finds a target and moves towards it, you get much less CPU strain because i don't try to search for new targets or re-assign them if they already have a task.

This may be part of it. But I was under the impression that 10,000 tiles is a huge distance, so I keep assuming that those squads can at least see the biters I can see. Maybe not, though. I've often wished for an on-map measuring tool.

1) perhaps they don't have a droid assembler to retreat to? The retreat setting is 2 i think, by default, so perhaps they cannot find a path or a valid droid assembler to return to. This can happen if you only ever deploy by hand, or if you deployed by assembler but then removed the droid assemblers afterwards (maybe for relocating?) but either way as soon as there is a valid one (no distance requirement) they should go towards it if "retreating".

There are a few of these at the edges of my remote outposts. I have deployed some squads by hand, but there are still places for them to return to in theory. And if there's no distance limit... then that's confusing.

3) This plays in to the issue talked about earlier on this thread about how instead of the hunt radius being squad-centered, it could be centered on the assembler that produced the squad. Then it's kind of like an aggressive "guard" assembler. Current behaviour of the squads is literally to just blindly slaughter/rampage their way through the nearest enemies until they die, which indeed if they have a target-rich environment, may cause them to delve deep into enemy territory.

I was saying before that a complete re-write of the mod's method of deployment and organisation/AI of the squads so that it's assembler-centric at all times would solve this, at least a little bit - and maybe even disable the player's ability to place droids by hand altogether. It might be more intuitive that way.

This sounds like it might work. Though I have to admit, I'd prefer to still be able to place them by hand. I find destroyed capsules to be pretty frustrating (their behavior that requires you to continually run at/through the biter base just seems dumb from a warfare aspect), so I prefer to be able to defend myself "in the wilds" just by putting down a few 'guard' droids at a moment's notice. The fact that I lose control over their targeting once they've defended me from the initial threat is not generally a problem.

I can have upwards of 1200 droids actively slaughtering biters, before I see any performance impact.. maybe my computer is just a beast? I hope the improved AI handling helps your performance

Oh, wow. I was initially running on my MacBook Pro, which is admittedly only a 2.5 GHz processor. But then I moved to my desktop rig, and I found that I was still getting the once-per-second hitches/freezes until the numbers dropped below about 200. At the time I was at 500, but that's still nowhere near 1500. I did end up overclocking that CPU to about 4.4 GHz, and it's seemed better since then, but it was still occasionally a little glitchy. I also reduced the hunt distance from 10000 (which I tried because I had too many squads that were just being lazy) back to around 6000, and I'm sure that made a difference as well.

Lua using indenting is disgusting, I hate it. If this was python (but, compiled, for speed) I would be much happier. I am always interested in having help on the mod, there is plenty of code clean-up, refactoring, new concepts, and integrating the work that another guy has done for me recently on the new technology tree for Robot Army that makes the tech mod-unique and has multiple levels of damage tech for each droid type.

I guess if you know Python and can stomach Lua's indenting, then so can I. I'll give the dev version a test today and I'll get back to you. And I'll see if I can get a chance to really look at the code again. Looks like control.lua at least tells me what each of the main, important files are.

[kyranzor]

Well actually, Lua uses logic blocks and "end" not indenting. Python is actually the horrible indenting maniac language, but I still prefer it for some reason.

Doing lots of scripting in Factorio mods can be hard because you have no real feedback except for when things go wrong - if you have no syntactical errors and no nil references pop up, it can be hard to tell if your code is running as intended. I have to litter my code with debug print or log statements to make sure the inner workings are as intended. Plus the visual feedback in the game but that can often-times be confusing or useless lol.

Last night, was debugging the following: A squad is standing still after having worked fine for a while, and not doing anything and the unit_group disbands randomly causing the units to scatter and start wandering - why!? Well it turns out my AI function call checks if the unit group is valid, and sometimes it ISN'T valid and normally my other code goes and re-creates the unit_group and re-adds the members - but what if the AI function is called on exactly the tick that the unit_group disbands? It never runs the AI function, and the group never gets remade! So I added a call to my re-group function before checking if it's valid or not, and now the AI functions get called and everything works

[b38917ce-3ad3-4fae-b]

I can have upwards of 1200 droids actively slaughtering biters, before I see any performance impact.. maybe my computer is just a beast?

I'm curious about this: do you mind trying the save I linked in this post and comparing your timings to mine?

[kyranzor]

Sure, I will do this tonight.

[dauphin]

Really good news - the new "binned" squad update approach works absolute wonders.

I don't know what the 'units' are, but the time_used_percent (okay, maybe they're percent? ) dropped from a fairly consistent 1.5 for Robot Army with 0.2.3, to 0.06 with the alpha of 0.2.4.

This also pushed my "median" UPS back up to 60, from around 54.

Probably more importantly, even when the UPS drops lower (sometimes as low as 50 on my current map where the save file is ~70 MB), the game no longer 'hitches' once per second as Robot Army does its calculations. Instead, the 'slowness' (coming from other parts of the game) is appropriately spread out across the entire second, so while the framerate isn't perfect, it still 'feels' smooth.

This is a major step forward for Robot Army.

Oh, btw, I had to change the line in info.json to 0.2.4 to make it match the folder name.


-----

However, I have some bad news as well. It seems that shortly after the migration script runs, the 'radar vision' of the robots disappears. Where previously I could find my squads by looking at the map for areas that were revealed and filled with little blue moving dots, that visually-revealed area on the map disappears within the first few seconds of the migration running. The bots are still there, as I can travel to their location and see them moving around. I can still pick them up, and they still seem to be on my 'force'. But I don't think I had observed this behavior in 0.2.3. So it seems like there may have been a regression as well.

[kyranzor]

Hey, thanks for the feedback!

The migration script may have failed to properly load the existing squads into the new tick table method of running AI updates, so it's possible that your (old) squads are actually not getting any more AI updates! AI updates are their hunting/guarding logic, scanning around them for artifacts, and revealing the map/chunks around them. If they do not seem to be collecting artifacts, the map isn't updating, or they have stopped finding new targets to hunt, it means they did indeed fail to be ported over to the new system, and they are effectively dead-weights.

I need a save game with existing hunting squads prior to the application of the migration script. In my current test save I only had guard squads lol so it made it awkward to test the transition.

Can you make some new squads and see if they find targets and reveal the map as expected?

A peak script time value of 1.5 (units are milliseconds by the way) every n'th tick from the old way of doing things should now be ~60 times lower, so 0.025 milliseconds for the AI update, but as a constant time rather than a pulsed/peak CPU load.

[dauphin]

Yep, I did my own math after I posted and realized that the 'goal' time would be 0.025. The fact that it's slightly higher than that doesn't surprise or bother me too much - I'm guessing the O(1) costs are factoring in a bit more obviously now that the O(n) costs are lower. Anecdotally, I can report that it does actually get as high as 0.195 with my roughly 150 existing robots when they're actually running around doing things.

Which brings me to the second observation. Yes, it's just pre-0.2.4 squads that are misbehaving. I went and collected a large squad (seemed like about 50 robots), and then spawned some new ones. They immediately went out to hunt, and their radar signature is back to normal. So this isn't a major regression - probably just a minor issue with the migration, as you suggest.

Let me know if I can do anything additional to test. I anticipate trying to play a 2 person multiplayer game with 0.2.4 enabled here in a little bit. If anything goes awry, I will let you know.

Oh, and can you make things easy on me and point me to the actual code that does the per-tick update? I don't know much about Factorio modding, and in particular I don't know the 'entry points' from the core game into the mod. I'd be interested in looking a little more closely at the code to see if it sparks any ideas, particularly on the performance side. Since I tend to play on fairly large, late-game maps, I am interested in squeezing as much performance as possible out of things. Never mind. Found onTickHandler().