[0.17.64] Crash allocating memory while RAM available

[marco_]

I'm experiencing a crash when I start the game. It starts loading and at about 41% I get "Unexpected Error occured....". When I continue trying to start the game different things happen. Either it just stops loading at around 41% or the screen flickers before it crashes. After the 20th try or something it then managed to load completely. It seems to be Memory related, since when I close some programs there's no issue starting the Game.

[boskid]

Not a Bug/Technical Help.

It looks like you have not enough free RAM

Code: Select all

 108.177 Error CrashHandler.cpp:384: Unhandled exception type: .?AVBadAllocation@@
 108.177 Error CrashHandler.cpp:398: Unhandled exception: memory allocation of 2178799 bytes failed! RAM: 9749/16325, Page: 16819/16825.

-- edit:
I think this could be related to this (even if there is free RAM available, system may use free ram for buffers and not reuse them immediately)

Memory allocation errors can be caused by slow page file growth

Cause
Memory allocation failures can occur due to latencies that are associated with growing the size of a page file to support additional memory requirements in the system. A potential cause of these failures is when the page file size is configured as “automatic.” Automatic page-file size starts with a small page file and grows automatically as needed.

[Loewchen]

Ref.: 73841

[marco_]

I see... why does it Swap when I have plenty of free memory?

[Rseding91]

The log file is showing that your page file is almost completely used before the game even starts working on loading everything.

To me, that sounds like you have load of other programs running which have allocated memory, and been paged to disk. The page file fills to 100% and the OS fails a normal allocation because it can't store *anything* in the page file.

At least, that's my theory.

Do you have this problem after a fresh restart?

[marco_]

No... when I haven't so many programs in the background everything works fine. I've extended page file now and it works...
It's just that I don't completely understand why windows pages if there is still 7GB Ram available... but that's another question I guess

[BlueTemplar]

A decade or so ago, it was recommended to set a fixed size page file, twice the size of your RAM...

But since then, the limit of 2 GB per program went away, 64-bit became the norm rather than the exception, and (relatively) low capacity SSD's have arrived.
So I guess that this advice is obsolete now ?

[slippycheeze]

No... when I haven't so many programs in the background everything works fine. I've extended page file now and it works...
It's just that I don't completely understand why windows pages if there is still 7GB Ram available... but that's another question I guess

Because the value of a page of memory sitting there doing nothing is lower than the value of a page of actively used data from disk, or that another program is actively using. Main memory is very, very far away from the perspective of the CPU, but disk ... might as well be another planet, far away. So if you can use that memory for something else and avoid touching disk that'd be literally 1,000 times faster.

Since it is also a frequently asked question in this context: you want zero "free" memory, full stop. Because "free" memory means it is sitting there doing nothing at all of value. Caching disk content, etc, are much, much more valuable uses of your limited memory.

...uh, and please remember that this is an area where magical thinking and snake oil abound. Like the "latency fixers" that do less than nothing, lots of people take advantage of ignorance -- or act from it -- to try and solve problems around swap without understanding them. You shouldn't fiddle with the defaults, by default, especially not if something says it'll improve performance.

A decade or so ago, it was recommended to set a fixed size page file, twice the size of your RAM...

But since then, the limit of 2 GB per program went away, 64-bit became the norm rather than the exception, and (relatively) low capacity SSD's have arrived.
So I guess that this advice is obsolete now ?

Not for the reason you think, but yes. The reasons for having swap are still there, and still just as valid today as they were way back then.

What changed is that you probably don't want 64GB of swap on a 32GB computer, because that turns "ran out of memory" into swapping forever. You only want a few GB -- 2, 4, maybe 8 depending on your workload. Desktop computers tend to the last -- they often have a pile of unused stuff running, like those background chrome tabs you have not gotten back to yet.

You really don't want to go much beyond 8GB under any circumstances, though. That said ... letting the OS figure it out is almost always the right choice today. They do a fine job, because they are built to make this technical complexity just go away for all the folks who expect them to just work™ for them.

PS: there are circumstances where more than 8GB swap are appropriate, but if you don't know what they are, they don't apply to you. This is in the realm of "specialized setup for very specific purposes" configuration.

[posila]

No... when I haven't so many programs in the background everything works fine. I've extended page file now and it works...
It's just that I don't completely understand why windows pages if there is still 7GB Ram available... but that's another question I guess

Because the value of a page of memory sitting there doing nothing is lower than the value of a page of actively used data from disk, or that another program is actively using. Main memory is very, very far away from the perspective of the CPU, but disk ... might as well be another planet, far away. So if you can use that memory for something else and avoid touching disk that'd be literally 1,000 times faster.

Since it is also a frequently asked question in this context: you want zero "free" memory, full stop. Because "free" memory means it is sitting there doing nothing at all of value. Caching disk content, etc, are much, much more valuable uses of your limited memory.

...uh, and please remember that this is an area where magical thinking and snake oil abound. Like the "latency fixers" that do less than nothing, lots of people take advantage of ignorance -- or act from it -- to try and solve problems around swap without understanding them. You shouldn't fiddle with the defaults, by default, especially not if something says it'll improve performance.

Does it mean those 7 GB of RAM that OS reports as free is actually being used as filesystem cache to speed up file IO? And is it normal that such use of "free" RAM by OS would make further memory allocation to fail?

[slippycheeze]

No... when I haven't so many programs in the background everything works fine. I've extended page file now and it works...
It's just that I don't completely understand why windows pages if there is still 7GB Ram available... but that's another question I guess

Because the value of a page of memory sitting there doing nothing is lower than the value of a page of actively used data from disk, or that another program is actively using. Main memory is very, very far away from the perspective of the CPU, but disk ... might as well be another planet, far away. So if you can use that memory for something else and avoid touching disk that'd be literally 1,000 times faster.

Since it is also a frequently asked question in this context: you want zero "free" memory, full stop. Because "free" memory means it is sitting there doing nothing at all of value. Caching disk content, etc, are much, much more valuable uses of your limited memory.

...uh, and please remember that this is an area where magical thinking and snake oil abound. Like the "latency fixers" that do less than nothing, lots of people take advantage of ignorance -- or act from it -- to try and solve problems around swap without understanding them. You shouldn't fiddle with the defaults, by default, especially not if something says it'll improve performance.

Does it mean those 7 GB of RAM that OS reports as free is actually being used as filesystem cache to speed up file IO? And is it normal that such use of "free" RAM by OS would make further memory allocation to fail?

Executive summary:

• I'm rusty at the deep internals here.
• It may or may not include file system cache, depending how you asked the OS, but it probably does.
• That number just means "pages we can use immediately without writing them to disk."
• That number has approximately nothing to do with allocating new memory, or getting an "out of memory" error.

My comments were mostly general, because it has been more than a decade since I looked that closely at the internals of the Windows memory allocator. It looks like those numbers are generally not going to include the filesystem cache any longer when Task Manager etc show them. I guess Microsoft got sick of the complaints and reclassified that memory. Your code probably does count the cache in the value though.

Checking around, it seems things haven't changed that much, but keep in mind I'm a *cough* bit out of date here. Windows still does strict overcommit in 10, so you can't allocate more virtual memory than the sum of physical RAM and page file size. Since the user had a fixed page file size, and heavy use of it, it'd guess they simply couldn't get more memory because Windows won't risk running out.

So ... I think in this case, yes, the system was "out" of memory, in the sense that it promised all the memory to something, and wasn't gonna gamble on being unable to make good that promise even if there was "free" memory right now.

The confusion is because "free" and "used" are talking about different ways to measure stuff. The "free" memory in the graph and summary is not the same as the "free" memory needed when Factorio tries to allocate more memory. The amount that matters there is the "committed" memory, and especially the number on the right, in the task manager. If that caps out, you can't allocate more memory, regardless of how much "free" memory the task manager is showing.

I can't say for sure what the numbers from CrashHandler.cpp use, but there are probably not enough of them to have a full view of why this happened. Assuming the world hasn't moved on too far, I think you are probably logging the `ullAvailPageFile`and `ullTotalPageFile` fields from GlobalMemoryStatusEx, which might also be less than the amount available to the whole system -- just to add more confusion to the whole thing.

This document is declared outdated, but I think still has a good summary of how things map from the system calls to the Task Manager UI. Which the short version of is "really, really confusingly", thanks Microsoft.


Uh, I hope that all helped, rather than adding confusion.

[BlueTemplar]

Desktop computers tend to the last -- they often have a pile of unused stuff running, like those background chrome tabs you have not gotten back to yet.

I haven't used Chrome in a while, but doesn't it "unload" unused tabs when it senses that you start to run out of memory?

[posila]

Thanks for the knowledge dump.

I can't say for sure what the numbers from CrashHandler.cpp use, but there are probably not enough of them to have a full view of why this happened. Assuming the world hasn't moved on too far, I think you are probably logging the `ullAvailPageFile`and `ullTotalPageFile` fields from GlobalMemoryStatusEx, which might also be less than the amount available to the whole system -- just to add more confusion to the whole thing.

Yeah, we log (ullTotalPhys - ullAvailPhys), ullTotalPhys, (ullTotalPageFile - ullAvailPageFile) and ullTotalPageFile. We could also add results of GetProcessMemoryInfo and GetPerformanceInfo, but it seems to me we can't do anything but reduce our memory usage, to fix these issues to make this kind of crash happen less often.

[slippycheeze]

Yeah, we log (ullTotalPhys - ullAvailPhys), ullTotalPhys, (ullTotalPageFile - ullAvailPageFile) and ullTotalPageFile. We could also add results of GetProcessMemoryInfo and GetPerformanceInfo, but it seems to me we can't do anything but reduce our memory usage, to fix these issues to make this kind of crash happen less often.

Yeah, those are really all that matters. If you felt enthused you could dump `HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagingFiles` which contains the configured sizes, to differentiate between "ran out of space" and "didn't expand fast enough."

...but you can't really do much about it anyway, exactly as you say. In theory retrying the memory allocation might work, in practice I doubt it'd ever work in a way that mattered, because the user gets no feedback other than application failures or whatever, so the memory isn't likely to come free.

Desktop computers tend to the last -- they often have a pile of unused stuff running, like those background chrome tabs you have not gotten back to yet.

I haven't used Chrome in a while, but doesn't it "unload" unused tabs when it senses that you start to run out of memory?

Not last time I bothered to check. Even if so, not enough that it solves my procrastination tab management problem.

...but you can substitute anything else for Chrome which was, frankly, a concrete example I picked by virtue of it being exactly what I was looking at right now, not because it is special in any other way compared to, eg, keeping open Word and Excel, or whatever, which I have no idea why people do.