trying to make a mod, accidentally crashed my whole computer

[prg]

And what do you want Factorio to do about this? What could it possibly do? It can't know how much memory the system has to spare, knowing this is the job of the kernel. So Factorio requests some memory from the system. If this works, Factorio uses this memory (why shouldn't it?); if it doesn't, a "cannot execute command: out of memory" error message is displayed. This is working properly.

Now if the kernel hands out more memory than it should there is a problem.

On Linux this can happen if overcommitting is enabled, which tends to be the default. This means the kernel might be optimistic and grant memory allocations even if there isn't enough memory available, hoping the requested memory won't actually be used (some processes like doing this apparently). Now if the memory does get used, a process needs to die to free up some memory. This will be whatever uses a lot of memory, likely Factorio in this example but it might also hit Firefox first if you have a ton of tabs open. This is a kernel issue, if you don't want to see random processes dying, you can disable overcommitting with sysctl vm.overcommit_memory=2. Then the kernel will just deny allocations when the system is out of memory and you will see the proper error message in Factorio.

Not sure how this works on Windows, but if the kernel shits itself before it starts denying memory allocations, that's a problem with the kernel.

So... if a memory allocation fails, Factorio displays the proper error message and gets on with life. If memory gets allocated successfully and the system still gets into trouble, there isn't anything Factorio can do about it.

[ArderBlackard]

I guess Factorio needs to include (besides the actual gameplay) also a fully-functional Lua code analysis (to detect and prevent all possible modding errors like endless loops and memory overcomitting from the inside) along with a bunch of system-monitoring tools (to control any issues from the outside).

Well, why stop at this? Why not create a FactorioOS that will guarantee the game behavior to be predictable and controllable without any external factors? Just a few more dozen years to wait for the game to be released...

Or maybe modders could just check their code a bit before launching it? I've encountered a variety of different funny and not-so-funny consequences of programming errors during my career but I've never tried to blame anyone for "why didn't they prevent the mistakes I had made?"

[Divran]

That's not necessary.

You are missing the point.
You encountered a bug.
You have to offer if possible (without redundancy)

• a crash log file
• a screenshot
• a description or material how to reproduce it (done)
• ...

I don't think taking a screenshot would work very well when my computer is crashing (except for a VM, but all you'd see is a black screen, not helpfup). Maybe a picture with my smartphone? But again, all you'd see is a black screen for a few seconds before it restarts.
I'm not sure if a log is created at all when your entire computer crashes. But I could check when I get back home.

The devs have already tested your issue and they don't have that issue, so the ball is on your side once again.
Now you need prove that the issue exists on other systems than your own.

I've already tested it in three different systems (my pc, and two VMs) and in all three it was reproducable. The only OS where someone has tested it where it seems to work properly is windows 7. A developer said he left it to run for a few minutes without it crashing. I don't know if he did anything else after that, though - such as attempting to open a new game + another application while factorio had already eaten all the memory - so his test probably passed because of that.

Maybe the issue is in the mod you created and can only produced in a specific way, that the devs didn't use during testing. Who knows? Thats why I asked you to provide the mod so that others can confirm the bug and to give the devs a chance to look at it.

What's so difficult about making an infinite loop? This thread has 4 pages talking about how to reproduce the issue. But if you insist

Code: Select all

local derp = {}
while true do derp[#derp+1] = "derp" end

or this

Code: Select all

local derp = {herp=1}
for k,v in pairs( derp ) do derp[k.."herp"] = v end

In code there are "sideeffects" that "magically" do some stuff you don't expect, but without code its hard to reproduce them.

There's no side effects or magic in a simple infinite loop that just eats your memory

And what do you want Factorio to do about this? What could it possibly do? It can't know how much memory the system has to spare, knowing this is the job of the kernel. So Factorio requests some memory from the system. If this works, Factorio uses this memory (why shouldn't it?); if it doesn't, a "cannot execute command: out of memory" error message is displayed. This is working properly.

[... more text ...]

I'm sorry, I'm not a low level developer so I don't know exactly how to solve the problem. But we now know that this works in windows 7 already (a developer tested it there, and he said that he was unable to crash his computer.). In linux and windows 10 however, it does not work (linux is forced to step in and force close it, while windows 10 crashes the entire computer).

I also know that most games out there already has this exact check in place, regardless of whether or not they have modding support. The reason is probably that the developers themselves know that they can make mistakes, and they don't want the game or computer to crash in the middle of development. So judging from this, it can't be too hard to solve.

Take Garry's Mod for example. It has Lua modding support, and it runs in the Source engine. The source engine has fully working memory usage checks in place. If you attempt to make something that eats your memory in Garry's mod, the Source engine itself will terminate its own process long before it becomes a problem. Long before Linux can come in and close it, even.

[... more stuff ...] Or maybe modders could just check their code a bit before launching it? I've encountered a variety of different funny and not-so-funny consequences of programming errors during my career but I've never tried to blame anyone for "why didn't they prevent the mistakes I had made?"

Haha very funny. Obviously you haven't read a single thing I've said in this entire thread. This is extremely simple and I don't understand why nobody understands this. Let me repeat this for the 10th time: Of course it's your own fault if you make an infinite loop. But that should not crash your entire computer. At most, it should crash the factorio process.

If you people want to be as unbelievably ignorant as you are, be my guest. Explaining a blatantly obvious feature over and over again is growing tiresome.

[daniel34]

What's so difficult about making an infinite loop? This thread has 4 pages talking about how to reproduce the issue. But if you insist

Code: Select all

local derp = {}
while true do derp[#derp+1] = "derp" end

derp_0.0.1.zip

(644 Bytes) Downloaded 97 times

Using that mod (modified to use 500 characters instead of 4 to fill memory faster) on a system with 16 GB RAM (Win 7 x64) the savegame will load for ~3 minutes, then show a messagebox "not enough memory" without crashing Factorio or the OS, using ~6 GB according to the task manager. I didn't even expect that Factorio would catch the infinite loop / prevent a crash due to memory overload.

EDIT: Tested on a laptop with Windows 10 Home (64-bit, 6GB RAM). After around 4-5GB usage it begins to swap (Task Manager shows >50MB/s harddisk usage), a little while (1min) later it will stop with the same "not enough memory" dialog. I used Chrome while doing that test and didn't encounter any slowdown there.

Unless you can provide a mod that actually crashes Factorio or the operating system your statement from the first post (quoted below) is false.

It's never okay for a process to be able to crash your whole computer, regardless of what caused it. The fact that you can do so from inside a mod makes it even worse.

[Divran]

What's so difficult about making an infinite loop? This thread has 4 pages talking about how to reproduce the issue. But if you insist

Code: Select all

local derp = {}
while true do derp[#derp+1] = "derp" end

derp_0.0.1.zip

Using that mod (modified to use 500 characters instead of 4 to fill memory faster) on a system with 16 GB RAM (Win 7 x64) the savegame will load for ~3 minutes, then show a messagebox "not enough memory" without crashing Factorio or the OS. I didn't even expect that Factorio would catch the infinite loop / prevent a crash due to memory overload.

Unless you can provide a mod that actually crashes Factorio or the operating system your statement from the first post (quoted below) is false.

It's never okay for a process to be able to crash your whole computer, regardless of what caused it. The fact that you can do so from inside a mod makes it even worse.

Now to be perfectly clear, if I am proven wrong here and it does actually work, and the issue is only reproducable on my computer, it would make me happy, not angry. I want factorio to work just as much as the next guy.

Now..
Did you test it on windows 10 home 64 bit? It's possible the issue only occurs there. This is the only OS I've been able to crash the computer in. In linux, all that happened when factorio's own popup appeared was that the process was terminated by the OS.

Did you try clicking "Ok" on the "out of memory" popup, and then immediately creating a new game? Did the popup appear this time?
If it still hasn't crashed, try launching a new application (it crashed for me when I tried opening task manager) while Factorio, with inf loop, is running at the same time.

When it happens has been slightly different for me each time, and I've tested it three times. One time (windows 10), it crashed about 5 seconds after clicking the generate map button. The next time (linux), the popup showed, so I tried again (without restarting factorio). This time, Linux stepped in and closed the process immediately after clicking the generate map button. The third time (windows 10), the popup showed, then it didn't crash, so I tried to open task manager to see how much memory it was using. Opening task manager didn't work, as windows told me my swap space was full. I tried opening task manager three times, and the third time, it crashed everything.

I'm going to break open my old as f laptop (with windows 7) to see if I can reproduce it there, on entirely different hardware. I could also test it on my newer laptop, for a third set of hardware, but that's running the same OS as my desktop.
Will report back as soon as I can on this

[daniel34]

Did you test it on windows 10 home 64 bit? It's possible the issue only occurs there. This is the only OS I've been able to crash the computer in. In linux, all that happened when factorio's own popup appeared was that the process was terminated by the OS.

Did you try clicking "Ok" on the "out of memory" popup, and then immediately creating a new game? Did the popup appear this time?
If it still hasn't crashed, try launching a new application (it crashed for me when I tried opening task manager) while Factorio, with inf loop, is running at the same time.

I edited in the test of Windows 10 directly after I made my post and tested it (~10 minutes after) and before reading yours, because you mentioned it only happening on Win 10 in your previous post.

It was the same behavior on 7 and 10, it shows the popup at around 4-6GB memory usage. Clicking OK on it and trying to load the save again gives the same result. On the 16 GB machine it keeps some of memory allocated before, increasing total memory used to 8 GB on the second run (compared to 6 GB on the first run), on the 6 GB machine the memory used stays the same. I suspect the garbage collection does its job there, whereas its not needed as much on the other machine.

Now to be perfectly clear, if I am proven wrong here and it does actually work, and the issue is only reproducable on my computer, it would make me happy, not angry. I want factorio to work just as much as the next guy.

I agree with you. If there's an actual problem with infinite loops / memory usage that crashes the whole OS or impacts other applications it would be bad and should be solved. But in my testing Factorio stops the savegame load after a specific amount of memory was used and doesn't impact other applications or crash the OS.

How much memory does your system have, and how much did you give the virtual machines?

[ArderBlackard]

Obviously you haven't read a single thing I've said in this entire thread. This is extremely simple and I don't understand why nobody understands this. Let me repeat this for the 10th time: Of course it's your own fault if you make an infinite loop. But that should not crash your entire computer. At most, it should crash the factorio process.

Actually I have, honestly. But still don't agree with you. The application trying to consume more memory than it is possible should be terminated by the OS. If a simple memory allocation over the limit crashes an OS - the problem is in the OS.

I've tried running your script in Factorio (I'm on Win10 Enterprise with 16 GB RAM):
* When launching from console - after using ~16GB of RAM Factorio responded that there was not enough memory and terminated the command.
* When running as a mod - after using ~16 GB I got a message box with a similar message. No crash occurred.
* [I'll turn off the pagefile completely now and try once again...]

[Divran]

So I tried it on my old laptop and my new laptop. Unfortunately my old laptop is so slow that I can't tell the difference between a freeze and map generation, haha

Anyway, I was able to reproduce the issue on my new laptop. However only with this code:

Code: Select all

local derp = {derp=1}
for k,v in pairs( derp ) do derp[k.."derpderpderpderpderpderp"] = v end

I was not able to reproduce it with the mod that daniel posted.

When I ran this code, my laptop didn't immediately crash. Here's what happened:
After about half a second, ram usage went up to 99%. Immediately after, factorio's ram usage went down to 50mb. Then up to about 350 mb, then back down to 50mb, and then it kept jumping all over the place to random memory usages. All of Window's processes also jumped up and down like mad. It was as though windows could no longer give task manager the correct memory usage values for any application. After sitting around waiting for a couple more seconds, with nothing happening, I decided to try to check the other tab in the task manager. When I went to click it, I noticed the cursor was moving sluggishly. It also took a few seconds for the task manager page to load its contents. This thing was bringing my laptop to its knees (and my laptop is no weakling. i7 processor, 8 gb ram and SSD). Next, I clicked the other tab in task manager to switch back, and that's when the entire screen went black. Note, my laptop didn't auto-restart like my desktop did. This was some kind of different crash. The screen was just black... Nothing I did could bring the screen back. I tried hitting CTRL+ALT+DEL, Alt+F4 (in an attempt to close the factorio process), nothing worked. After waiting for a while, I held down the power button and restarted the computer.
EDIT: Oh and Daniel, you mentioned you loaded a save game. I didn't. I'm creating a new game every time.

And the entire time, my SSD was sitting at 100% usage, probably due to crazy thrashing.

So yeah, this is strange. Also, I have no idea why one of the two loops would cause this and not the other.

Also, my old laptop has finished loading the first code style. The popup had appeared. Time to edit the code to use the method I showed in this post to see if windows 7 crashes the same way.

Actually I have, honestly. But still don't agree with you. The application trying to consume more memory than it is possible should be terminated by the OS. If a simple memory allocation over the limit crashes an OS - the problem is in the OS.

Already answered this on the second page viewtopic.php?f=25&t=25583&start=10#p161613

EDIT: Old laptop is doing the thing now. Gonna take a while since it's such a slow POS

[bNarFProfCrazy]

I tested it on my maschine with the given derp mod.

Win 7 x64 8GB RAM -> no crash

Here is an overview from my task manager showing the memory consumption.

Memory

MemoryUsage.png (12.87 KiB) Viewed 4856 times

Without anything running my system takes 1GB of memory.
Browser takes usually 300MB on start.

1) Launch of Factorio with mod installed
2) Loading scenario stuck at 100% + Open Task manager (Game takes 3.7GB Memory (32bit limit?) + 2 GB free)
A) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there (Garbage collector? Memory shuffling for better space usage?)
B) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there (Factorio took 7.3 GB of my RAM before that)
3) Launch browser, it wasn't fast, but that was to be expected without free memory
4) Close browser
C) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there
5) Launch browser again
6) Take a screenshot and open paint
7) Save the screenshot in paint (because system is dead slow, but still useable)
8) Close browser
D) Loading savegame failed out of memory popup shows up and try loading the next game + start browser.
9) Kill factorio/End of test

I'm sorry that I cannot place the markers more precisely.

The test took ~5-10 minutes. AFAICT no memory has been to my hard disk during the entire test.

I don't have any other systems ready to test the mod elsewhere.

[Divran]

I tested it on my maschine with the given derp mod.

Win 7 x64 8GB RAM -> no crash

Here is an overview from my task manager showing the memory consumption.

Memory

MemoryUsage.png

Without anything running my system takes 1GB of memory.
Browser takes usually 300MB on start.

1) Launch of Factorio with mod installed
2) Loading scenario stuck at 100% + Open Task manager (Game takes 3.756GB Memory (32bit limit?) + 2 GB free)
A) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there (Garbage collector? Memory shuffling for better space usage?)
B) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there (Factorio took 7.3 GB of my RAM before that)
3) Launch browser, it wasn't fast, but that was to be expected without free memory
4) Close browser
C) Random event, that frees memory. I didn't do anything at that time so I'm not sure what happened there
5) Launch browser again
6) Take a screenshot and open paint
7) Save the screenshot in paint (because system is dead slow, but still useable)
8) Close browser
D) Loading savegame failed out of memory popup shows up and try loading the next game + start browser.
9) Kill factorio/End of test

I'm sorry that I cannot place the markers more precisely.

The test took ~5-10 minutes. AFAICT no memory has been to my hard disk during the entire test.

I don't have any other systems ready to test the mod elsewhere.

Can you test it with the code I posted above your post? Did you test it with factorio 64 bit?

[ArderBlackard]

Actually I have, honestly. But still don't agree with you. The application trying to consume more memory than it is possible should be terminated by the OS. If a simple memory allocation over the limit crashes an OS - the problem is in the OS.

Already answered this on the second page viewtopic.php?f=25&t=25583&start=10#p161613

EDIT: Old laptop is doing the thing now. Gonna take a while since it's such a slow POS

Well, I don't mean that there is something wrong with Windows in general. It seems that there is some problem on your side (hardware?). Because I didn't manage to crash my OS despite all the attempts. I tried running your code (you have lost closing end btw) without pagefile at all. All that I got - the message box in Factorio and another one from Windows about running out of memory. After that I just closed Factorio and everything continued running without even any glitch. And other users here also state that they don't get any troubles with the issue. OSes as well as Factorio behave in general as it can be expected.
So it's not about "Factorio is bad written - it crashes operating systems", it seems to be more likely "sometimes overcommitting memory with Factorio causes operating systems to crash" but the reasons are not so obvious.

P.S. Just another idea: did you try to consume all the memory on your computer using some other program? Will it crash the same way?

[Divran]

Ok so I gave up testing on my old laptop (the one with windows 7) because it's just so slow it'll probably take all day to replicate the results of my new laptop. That is, if it's ever going to crash, since no one with windows 7 has been able to reproduce it so far.

Actually I have, honestly. But still don't agree with you. The application trying to consume more memory than it is possible should be terminated by the OS. If a simple memory allocation over the limit crashes an OS - the problem is in the OS.

Already answered this on the second page viewtopic.php?f=25&t=25583&start=10#p161613

EDIT: Old laptop is doing the thing now. Gonna take a while since it's such a slow POS

Well, I don't mean that there is something wrong with Windows in general. It seems that there is some problem on your side (hardware?). Because I didn't manage to crash my OS despite all the attempts. I tried running your code (you have lost closing end btw) without pagefile at all. All that I got - the message box in Factorio and another one from Windows about running out of memory. After that I just closed Factorio and everything continued running without even any glitch. And other users here also state that they don't get any troubles with the issue. OSes as well as Factorio behave in general as it can be expected.
So it's not about "Factorio is bad written - it crashes operating systems", it seems to be more likely "sometimes overcommitting memory with Factorio causes operating systems to crash" but the reasons are not so obvious.

P.S. Just another idea: did you try to consume all the memory on your computer using some other program? Will it crash the same way?

Did you try creating a new game immediately after clicking the "ok" button on factorio's out of memory popup? Without restarting factorio in between. That's what I had to do in my Linux VM before Linux stepped in and closed the process automatically. Also I've never received a message from windows that I'm out of memory in any of my tests. (Which OS are you running? Let me guess, windows 7. Nevermind, I read your other post, and it's windows 10. That's weird though) Nobody seems to be able to reproduce it on windows 7. I have, however, tested it on windows 10 three times now: my desktop, my desktop's VM, and my newer laptop all have windows 10. All three crashed.

[bNarFProfCrazy]

Next try with the code you posted:

Code: Select all

local derp = {}
while true do derp[#derp+1] = "derp" end

Still no crash.

Infact after the console comand failed to execute, the scenario was automatically closed and i could load a new game, and all memory got freed.

Memory

See

MemoryUsage.png (8.35 KiB) Viewed 4952 times

1) Launch of Factorio + execute command in console
2) Game looks frozen + open task manager
A) Random event that freed memory
B) Random event that freed memory
3) Start browser
C) Random event that freed memory
4) Close browser
D) Scenario got killed, Factorio still running + load a game
5) Launch browser to report the results

EDIT:

I have, however, tested it on windows 10 three times now: my desktop, my desktop's VM, and my newer laptop all have windows 10. All three crashed.

Maybe you have some bad/broken applications installed.
Like a anti virus sandbox. Similar to the Firewall MITM attacks that allowed invalid certificates to be accepted by the browser or execute root commands.
Sometimes they are the biggest PITA available.

[Divran]

stuff and things

Are you using factorio 32bit or 64bit though

[bNarFProfCrazy]

stuff and things

Are you using factorio 32bit or 64bit though

0.12.33 (Build 18092, win64, steam)

I don't have any other.

[ArderBlackard]

Did you try creating a new game immediately after clicking the "ok" button on factorio's out of memory popup? Without restarting factorio in between. That's what I had to do in my Linux VM before Linux stepped in and closed the process automatically.

After about 8 messages in a row (Not enough memory -> OK -> Generate -> Not enough memory -> OK -> Generate ... and so on ) Factorio just crashed with some message mentioning Boost library error. The OS has still resisted (let me remind that I'm running OS without a pagefile so no backup memory is available).
At the same time I've managed to crash my system with quickly written program that just consumes memory and closes if the system replies with OutOfMemory error (after multiple runs)
So from my point of view it looks like currently Factorio handles memory shortage pretty well (at least not worse than a majority of software) in quite a predictable way - by notifying about this fact and failing to execute the operation. All the crashes seem to be OS-related in the case it for some reasons cannot survive memory shortage properly.

[Divran]

Well it is possible that these crashes are not directly caused by factorio. Instead, factorio uses all the ram, and then windows itself and/or a different application is the cause of the crash when it tries to use some more ram. In which case, we'll have to bitch at microsoft to fix their broken OS I guess?

[Rockstar04]

Well it is possible that these crashes are not directly caused by factorio. Instead, factorio uses all the ram, and then windows itself and/or a different application is the cause of the crash when it tries to use some more ram. In which case, we'll have to bitch at microsoft to fix their broken OS I guess?

You don't need to worry about Microsoft at all, Linux is free and available to everyone.

[prg]

And what do you want Factorio to do about this? What could it possibly do? It can't know how much memory the system has to spare, knowing this is the job of the kernel. So Factorio requests some memory from the system. If this works, Factorio uses this memory (why shouldn't it?); if it doesn't, a "cannot execute command: out of memory" error message is displayed. This is working properly.

[... more text ...]

I'm sorry, I'm not a low level developer so I don't know exactly how to solve the problem. But we now know that this works in windows 7 already (a developer tested it there, and he said that he was unable to crash his computer.). In linux and windows 10 however, it does not work (linux is forced to step in and force close it, while windows 10 crashes the entire computer).

There isn't anything regarding the Linux situation that needs to be fixed by Factorio. This is a kernel issue. If you want to prevent processes from getting killed due to the system running out of memory, you need to configure the kernel to not overcommit memory. This has been explained in the [... more text ...] part.

[Divran]

Well it is possible that these crashes are not directly caused by factorio. Instead, factorio uses all the ram, and then windows itself and/or a different application is the cause of the crash when it tries to use some more ram. In which case, we'll have to bitch at microsoft to fix their broken OS I guess?

You don't need to worry about Microsoft at all, Linux is free and available to everyone.

That doesn't mean I'm going to use it. Linux might work better than windows, but that's only when it actually works. Every time I've given linux a chance, it doesn't f'n work. Believe me, I want to agree that linux is the better OS, but the truth is that it isn't. My opinion on linux is irrelevant. Ignore this.

And what do you want Factorio to do about this? What could it possibly do? It can't know how much memory the system has to spare, knowing this is the job of the kernel. So Factorio requests some memory from the system. If this works, Factorio uses this memory (why shouldn't it?); if it doesn't, a "cannot execute command: out of memory" error message is displayed. This is working properly.

[... more text ...]

I'm sorry, I'm not a low level developer so I don't know exactly how to solve the problem. But we now know that this works in windows 7 already (a developer tested it there, and he said that he was unable to crash his computer.). In linux and windows 10 however, it does not work (linux is forced to step in and force close it, while windows 10 crashes the entire computer).

There isn't anything regarding the Linux situation that needs to be fixed by Factorio. This is a kernel issue. If you want to prevent processes from getting killed due to the system running out of memory, you need to configure the kernel to not overcommit memory. This has been explained in the [... more text ...] part.

Do I need to link to page 2 again? I've already said that I agree with this. It should be the OS's job to not crash if an application uses all of the memory. In my tests, however, it doesn't work in windows, so my argument was to have Factorio handle it as well, as an additional safety measure. I've also already mentioned that every other game that I've ever played which has run out of memory has done so long before using all of the computer's memory. I assume these games set their own memory cap and check against that, while factorio checks when it can no longer allocate any memory. If the OS keeps giving it more even when it can't (like you said), then obviously factorio's checks would do absolutely nothing, and that's the OS's fault. Maybe Factorio could learn a thing or two from every other game ever and instead set a cap for itself? Not sure if that would work for factorio,because you can't know how much memory all of the mods are going to use in advance (gmod seems to handle this just fine, though). If there is a way to get how much memory the computer has in total, you may be able to set the memory cap to something like 75% of the total memory, I guess?