Page 1 of 1

[0.17.79] server not responding when placing landfill

Posted: Tue Feb 11, 2020 9:45 pm
by ThaPlayah
I have a savefile (with mods) with around 200 hours (150MB currently) on my linux server and recently (without any changes or new major builds) things started to get crazy.
The thing I want to report here is that every time we did not use landfill (or waterfill mod) for a few minutes and then using it again, we get "server not responding" popups for up to 30 seconds.
On the screenshot you can see a waterfill beeing placed next to my cursor and it looks weird - like not finished - during the freeze
What I found out is that UPS and FPS drop to 58 for a second when placing that land/waterfill. The server CPU usage is up to 100% on 1 of the 6 cores.
Most of the CPU cores are below 25% load when playing normally.
Before and after that freeze, UPS and FPS are 60.
After the popup disappears, we can continue to use land and waterfill without lags.
I did not find any suspicious log entries in the server log nor the client log (server on verbose). I also watched it during the freeze and there was nothing except for nat punches.

Any help debugging the issue is greatly appreciated

Other things that seem weird and started happening just recently:
  1. Our savegame was around 45MB on the server of the friend I'm playing with. For some reason his server went low on memory and crashed a few times. The game server crashing actually made using the VM where it was running on impossible and he was forced to reboot. I saved the backup when I was dropped and it became large in size (120MB). Since then, the autosaves go randomly between 60MB and 160MB and I don't know why
  2. The "server not sponding" popup sometimes happens woithout any contex, when just running around. In that case i see 100% on 1 single CPU core aswell.
link to savefile: https://drive.google.com/file/d/1geZ18- ... sp=sharing

Re: [0.17.79] server not responding when placing landfill

Posted: Wed Feb 12, 2020 5:00 pm
by ThaPlayah
Okay, here is an update:
I have spent some time better understanding the debug overlay and what I found is that shortly after the waterfill/landfill is placed,
the Pathfinder is going crazy and it seems that this is happening on server and on client side.
Not sure if it is important but I already placed a few million landfill on the map. At the beginning there was no problem at all.

So the things that I want to find out now is:
  • what parts of the game use this pathfinder (biters, robots?, trains?)
  • can I narrow down what exactly is causing the high CPU usage? Is there any additional debug info/tool that could help here?
  • can I fix it by changing the world
Once again, any help would be greatly appreciated.

Re: [0.17.79] server not responding when placing landfill

Posted: Wed Feb 12, 2020 8:34 pm
by Rseding91
This should be better in the next experimental 0.18 release. Additionally Oxyd is looking into it more and might find other things to improve on.

Re: [0.17.79] server not responding when placing landfill

Posted: Thu Feb 13, 2020 12:19 am
by ThaPlayah
Okay I was able to find the root cause by learning more about the different debugging overlaysand fix it.
With the waterfill mod we surrounded our base with a ring of water. That mean biters can't come close and attack us.
Unfortunately we left one single tile open by accident. The biters pathfinding found this "bridge" and created multiple paths that were around 20k tiles(?) long and reached around the whole map. Every time we placed a new waterfill or a landfill, some algorithm apparently tried to reevaluate the paths which probably cause the high CPU usage.
I fixed the problem by closing the waterfill gap so now the biters can't find any path. The savegame file is also back to 47MB now.
Took me around 10 hours to figure out I think :D

In the end it still seems weird to me that the savegame file became so big.

Re: [0.17.79] server not responding when placing landfill

Posted: Thu Feb 13, 2020 10:48 am
by Rseding91
ThaPlayah wrote: Thu Feb 13, 2020 12:19 am Okay I was able to find the root cause by learning more about the different debugging overlaysand fix it.
With the waterfill mod we surrounded our base with a ring of water. That mean biters can't come close and attack us.
Unfortunately we left one single tile open by accident. The biters pathfinding found this "bridge" and created multiple paths that were around 20k tiles(?) long and reached around the whole map. Every time we placed a new waterfill or a landfill, some algorithm apparently tried to reevaluate the paths which probably cause the high CPU usage.
I fixed the problem by closing the waterfill gap so now the biters can't find any path. The savegame file is also back to 47MB now.
Took me around 10 hours to figure out I think :D

In the end it still seems weird to me that the savegame file became so big.
Well, just turn off biters if you want to cheese them into not being able to fight. Otherwise you get stuff like this.