Page 1 of 1

Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 3:50 am
by AndrolGenhald
Pic says it all, a little over 20 GiB just by loading the save. Mods attached, save is too large to upload, caused nginx error page, it's at http://www.AndrolGenhald.com/factorio_ram_bug.zip for the time being.
RAM.png
RAM.png (62.58 KiB) Viewed 5240 times
Wow, I didn't realize how bad XFCE's default screenshot software was till I used it on a moving graph.

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 4:17 am
by Rseding91
I can't reproduce the issue. I loaded the save file on my computer and it used 21~ GB of RAM but when I clicked "exit" and it went back to the main menu it released all of it.

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 4:39 am
by AndrolGenhald
Well since there's no crash I don't see anything useful in the log. Is there a way to force verbose logging, or is there anything else I can do to help?
Regardless of the memory leak though, might there be anything that's using excess RAM as well? I realize the uncompressed save is almost a gig, and I expect it to use quite a bit of RAM to run it, but 20 GiB seems kind of high. If it can't be fixed I'm going to have to figure out how to run Factorio in a cgroup so the thrashing doesn't kill my system until I build a new one with 64 GiB RAM.

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 5:43 am
by Smarty
Install this mod https://mods.factorio.com/mods/orzelek/undecorator
I was able to reduce save file size by almost 66%
It might take some loading time for the mod to do its thing

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 12:13 pm
by Twinsen
Are you sure it's not some OS caching? Does loading the large save again cause your memory usage to reach maximum or make the game or computer unusable?

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 1:58 pm
by Oxyd
It's glibc keeping free'd memory to itself, instead of making it available for the rest of the OS.

Fixed in 0.14.9.

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 6:01 pm
by AndrolGenhald
You guys are great! :D

I take it that the amount of RAM being used is not currently considered an issue though?
Since exploration seems to be the largest factor in increased file size I'm assuming that it's also a significant factor for RAM usage? Could passive entities be unloaded after RAM reaches an optional limit based on LRU or furthest from the player (I'm thinking trees and decorations that might only be used when drawing them on the screen)? I realize it could end up being a lot of work, so I don't expect much, just throwing some thoughts out there.

Re: Extreme RAM usage and memory leak

Posted: Thu Sep 22, 2016 6:26 pm
by AndrolGenhald
Figured out why I couldn't get cgroups to work last night, apparently even when running Factorio from a shell, it's the steam process that actually starts it, so I was just putting the messenger in the cgroup instead of Factorio itself. After putting Steam in a cgroup and fiddling around with memory.limit_in_bytes I can get it to run pretty well with a few minor lag spikes due to paging with as low as 2 GiB memory and the rest in swap.

Edit: I've increased the limit to 16 GiB so that autosaves don't take forever, but I think the fact that it runs reasonably well with 2 GiB shows that most of the memory is used very infrequently.