[0.18.36] Parallel sprite loading consumes too much memory

[mrvn]

When I start factorio it creates an awful number of atlas bitmaps and then when loading sprites it consumes all of my free 10G of ram and then fails with a Bus Error when it runs out.

I played with a lot of mods in 0.17 before and never saw that many atlas bitmaps and all nearly the same size being used.

Note: vanilla loads fine. The mods seems to push it over the edge.

[Loewchen]

Post a save that includes the mods to synchronize please.

[mrvn]

I just downloaded 0.18.36 and added the mods and the problem appeared on the restart. I haven't even started a game yet and since the game crashes on start it's hard to make a save, right?

Anyway: Somehow factorio screwed up Xorg. Restarting factorio would show the progressbar for loading but never update it. The log showed progress but nothing got rendered. I had to shut down and restart XOrg completely.

After that factorio loaded fine with using "just" 1.2G ram and having a lot fewer atlases. I added a savegame for easier mod sync now that I can start factorio again.

Note: I relaced LTN-combinator with the LTN-combinator-fix mod but I doubt that has an affect.

[posila]

I just downloaded 0.18.36 and added the mods and the problem appeared on the restart.

Did you copy config.ini from different computer? You don't have default graphics settings for your HW configuration.

You can probably launch the game if you use --single-thread-loading command line parameter. Then I recommend to set texture compression level back to Low quality, which will make sprite atlases take up 33% less memory. If that is not enough, you might need to drop sprite resolution to normal due to mods you are using tripling amount of graphical assets. Then you should be able to launch the game without --single-thread-loading argument.

[mrvn]

I copied over the config dir from my 0.17 version of factorio, which comes from even older versions. Maybe I will delete that and create a fresh one. Maybe that helps.

The Bus Error problem is still there though. Today when I started factorio it happened again. I retried with top running (so I could monitor memory availability) and it worked. I did see it come pretty close to exhausting memory though. So I might just be right at the edge and sometimes it has enough and sometimes not.

What I saw was that during the loading the memory use exploded but when the Main menu popped up it went down to 3G virtual and 1.2G resident. Is that normal?

[Rseding91]

What I saw was that during the loading the memory use exploded but when the Main menu popped up it went down to 3G virtual and 1.2G resident. Is that normal?

Yes, it's using your available system resources to load as fast as it can.

[mrvn]

Would it be possible to add an option "--sprite-threads arg" to allow using a few threads for sprite loading but not all cores? Hopefully that would reduce the amount of temporary ram used without going totally single threaded. Or will "--threads" affect sprite loading too already?

[posila]

There is max-sprite-loading-threads in config.ini (you can set it from the game by Ctrl + Alt + clicking on Settings and then going to "The reset" options). The value is maximum, it will get reduced to minimum of (number of HW threads, floor((RAM_size - 2.5GB) / (1GB * HW_thread_count))

[mrvn]

That is odd then. I removed the config.ini so none of the old settings might screw things up. I assume it will then use the formula you mentioned. Doing the math (16G-2.5G)/(1G * 8 cores) = 1.6875 so I should get 1 thread.

In the log file I see:

32.128 Texture processor created (2048). GPU accelerated compression Supported
: yes, Enabled: yes/yes. Test passed. YCoCgDXT PSNR: 35.83, BC3 PSNR: 33.82
35.707 Parallel Sprite Loader initialized (threads: 7)

Not 1 as it should.

Setting it to 4 seem to do the trick for me with the high volume of graphics pyadon mods have. This has solved the problem for me but I think there is something wrong with the default as well as the formula.

The formula doesn't make much sense to me. If I have 16GB ram and 4 cores then 4 threads are used. But with 8 cores only 1 thread. With the same ram the more cores you have the less are to be used? The RAM should limit the number of thread as should the actual number of cores. But the formula would indicate the ram usage is about 1GB per core per thread. Why would each thread use more ram because the system has more cores?

I would expect something like min(cores, (RAM - 2.5G) / (4 * size of graphics to load)).

[posila]

My mistake, the formula is just floor((RAM_size - 2.5GB) / 1GB)