[0.18.36] Parallel sprite loading consumes too much memory

Anything that prevents you play the game properly. Do you have issues with paying for the game, downloading it or properly running it on your computer? Let us know here.
Post Reply
mrvn
Smart Inserter
Smart Inserter
Posts: 4183
Joined: Mon Sep 05, 2016 9:10 am
Contact:

[0.18.36] Parallel sprite loading consumes too much memory

Post by mrvn »

When I start factorio it creates an awful number of atlas bitmaps and then when loading sprites it consumes all of my free 10G of ram and then fails with a Bus Error when it runs out.

I played with a lot of mods in 0.17 before and never saw that many atlas bitmaps and all nearly the same size being used.

Note: vanilla loads fine. The mods seems to push it over the edge.
Attachments
mod-list.json
(1.97 KiB) Downloaded 7 times
factorio-current.log
(28.26 KiB) Downloaded 11 times
Last edited by mrvn on Fri Jul 31, 2020 12:18 am, edited 1 time in total.

Loewchen
Global Moderator
Global Moderator
Posts: 6105
Joined: Wed Jan 07, 2015 5:53 pm
Contact:

Re: [0.18.36] Bus error after consum ing all memory on start

Post by Loewchen »

Post a save that includes the mods to synchronize please.

mrvn
Smart Inserter
Smart Inserter
Posts: 4183
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: [0.18.36] Bus error after consum ing all memory on start

Post by mrvn »

I just downloaded 0.18.36 and added the mods and the problem appeared on the restart. I haven't even started a game yet and since the game crashes on start it's hard to make a save, right?

Anyway: Somehow factorio screwed up Xorg. Restarting factorio would show the progressbar for loading but never update it. The log showed progress but nothing got rendered. I had to shut down and restart XOrg completely.

After that factorio loaded fine with using "just" 1.2G ram and having a lot fewer atlases. I added a savegame for easier mod sync now that I can start factorio again.

Note: I relaced LTN-combinator with the LTN-combinator-fix mod but I doubt that has an affect.
Attachments
pyblock-000.zip
(2.17 MiB) Downloaded 5 times
factorio-current.log
(36.75 KiB) Downloaded 6 times
mod-list.json
(1.97 KiB) Downloaded 5 times

posila
Factorio Staff
Factorio Staff
Posts: 4660
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.18.36] Bus error after consum ing all memory on start

Post by posila »

mrvn wrote:
Sun Jul 26, 2020 6:09 pm
I just downloaded 0.18.36 and added the mods and the problem appeared on the restart.
Did you copy config.ini from different computer? You don't have default graphics settings for your HW configuration.

You can probably launch the game if you use --single-thread-loading command line parameter. Then I recommend to set texture compression level back to Low quality, which will make sprite atlases take up 33% less memory. If that is not enough, you might need to drop sprite resolution to normal due to mods you are using tripling amount of graphical assets. Then you should be able to launch the game without --single-thread-loading argument.

mrvn
Smart Inserter
Smart Inserter
Posts: 4183
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: [0.18.36] Bus error after consuming all memory on start

Post by mrvn »

I copied over the config dir from my 0.17 version of factorio, which comes from even older versions. Maybe I will delete that and create a fresh one. Maybe that helps.

The Bus Error problem is still there though. Today when I started factorio it happened again. I retried with top running (so I could monitor memory availability) and it worked. I did see it come pretty close to exhausting memory though. So I might just be right at the edge and sometimes it has enough and sometimes not.

What I saw was that during the loading the memory use exploded but when the Main menu popped up it went down to 3G virtual and 1.2G resident. Is that normal?

Rseding91
Factorio Staff
Factorio Staff
Posts: 10995
Joined: Wed Jun 11, 2014 5:23 am
Contact:

Re: [0.18.36] Bus error after consuming all memory on start

Post by Rseding91 »

mrvn wrote:
Tue Jul 28, 2020 7:02 pm
What I saw was that during the loading the memory use exploded but when the Main menu popped up it went down to 3G virtual and 1.2G resident. Is that normal?
Yes, it's using your available system resources to load as fast as it can.
If you want to get ahold of me I'm almost always on Discord.

mrvn
Smart Inserter
Smart Inserter
Posts: 4183
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: [0.18.36] Bus error after consum ing all memory on start

Post by mrvn »

Would it be possible to add an option "--sprite-threads arg" to allow using a few threads for sprite loading but not all cores? Hopefully that would reduce the amount of temporary ram used without going totally single threaded. Or will "--threads" affect sprite loading too already?

posila
Factorio Staff
Factorio Staff
Posts: 4660
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.18.36] Parallel sprite loading consumes too much memory

Post by posila »

There is max-sprite-loading-threads in config.ini (you can set it from the game by Ctrl + Alt + clicking on Settings and then going to "The reset" options). The value is maximum, it will get reduced to minimum of (number of HW threads, floor((RAM_size - 2.5GB) / (1GB * HW_thread_count))

mrvn
Smart Inserter
Smart Inserter
Posts: 4183
Joined: Mon Sep 05, 2016 9:10 am
Contact:

Re: [0.18.36] Parallel sprite loading consumes too much memory

Post by mrvn »

That is odd then. I removed the config.ini so none of the old settings might screw things up. I assume it will then use the formula you mentioned. Doing the math (16G-2.5G)/(1G * 8 cores) = 1.6875 so I should get 1 thread.

In the log file I see:
32.128 Texture processor created (2048). GPU accelerated compression Supported
: yes, Enabled: yes/yes. Test passed. YCoCgDXT PSNR: 35.83, BC3 PSNR: 33.82
35.707 Parallel Sprite Loader initialized (threads: 7)
Not 1 as it should.

Setting it to 4 seem to do the trick for me with the high volume of graphics pyadon mods have. This has solved the problem for me but I think there is something wrong with the default as well as the formula.

The formula doesn't make much sense to me. If I have 16GB ram and 4 cores then 4 threads are used. But with 8 cores only 1 thread. With the same ram the more cores you have the less are to be used? The RAM should limit the number of thread as should the actual number of cores. But the formula would indicate the ram usage is about 1GB per core per thread. Why would each thread use more ram because the system has more cores?

I would expect something like min(cores, (RAM - 2.5G) / (4 * size of graphics to load)).

posila
Factorio Staff
Factorio Staff
Posts: 4660
Joined: Thu Jun 11, 2015 1:35 pm
Contact:

Re: [0.18.36] Parallel sprite loading consumes too much memory

Post by posila »

My mistake, the formula is just floor((RAM_size - 2.5GB) / 1GB)

Post Reply

Return to “Technical Help”

Who is online

Users browsing this forum: No registered users