[0.16.3][Linux] Video hang on game exit

[talchas]

On exiting the game in 0.16 (.0 through .3) sometimes the nvidia driver hangs, freezing the screen and all input, though the system still works underneath (as long as you don't touch anything too involved in the video card or X or anything like that). 0.15 worked fine. As a workaround, kill -9 instead of a clean exit works fine.

This is obviously at root an nvidia driver issue, is probably ~impossible for you to debug on top of that, and is quite possibly system specific, but hey, maybe you'll find it useful or someone else will have this issue too, and I was bored and did some digging.

The hang in the one crash I went through the trouble of looking through occurred in glXDestroyContext, as called by xdpy_destroy_display_hook_default in backtraces.log. Just LD_PRELOADing that with an empty replacement wasn't sufficient, and neither was patching xdpy_destroy_display_hook_default to immediately ret. LD_PRELOADing glXDestroyContext to kill(getpid(), 9) does appear to be fine though.

The attached core dump is a bit hacked up - gdb didn't work on a thread hung in kernel, so the core has the general registers from the presumably-interesting thread patched in; I didn't bother with fp/sse or updating the tid metadata so those are all wrong. The delta-<pid> files are xdelta3 patches to the core file for the other threads.

Vsync was irrelevant, the sprite cache was irrelevant, nvidia drivers 384.90, 387.22, 387.34 all fail, nvidia.modeset=0 or 1 fails, gentoo-sources kernel 4.4.39 and 4.12.12 both fail. I haven't tested lower graphics settings and am planning on playing the game now rather than repeatedly crashing my system

[thecatlover1996]

On 0.16.10 and 0.16.12, the game also hangs for me when I click exit or the "close window" button on the top-right.
Using Ubuntu 16.04 with XFCE, Linux kernel 4.10-42, NVIDIA drivers 384.90.
Doesn't stop me from playing Factorio though, I'll just make sure I close all other programs before I click exit in Factorio and have to kill my PC
Didn't try `kill -9` yet, but thanks for the tip, I'll try it next time I play

[thecatlover1996]

Nope, using `kill -9 $(pidof factorio)` still makes my PC hang.
I can't switch to a TTY (like talchas said, input is frozen), but I can log in with SSH and see that Factorio is using an entire core of the CPU.
I could try to run Factorio with a debugger via SSH. It probably won't give much more information, since taichas already mentioned that it's a kernel thread that hangs.

[thecatlover1996]

Magic! Today my Nvidia drivers updated to 384.111, and as experiment I tried the regular exit button in the main menu after playing Factorio, and nothing crashed! (and now thumbs up and hope that this wasn't a one-time lucky shot )

@talchas (if you're still reading this): can you also try updating your drivers?

EDIT: Never mind, I just tried again and my system froze again. So it was a one-time lucky shot

[thecatlover1996]

I managed to attach GDB to Factorio and generate a core dump right before I clicked Exit. It's probably useless, but I can send a link in PM if you want.

After I clicked Exit, I couldn't manage to pause the debugger (or attach a new one) anymore in order to generate the core dump. Factorio was running at 100% CPU on one core, and so was one of my Chrome processes (my guess is that that's the GPU process, but I didn't check).

If there's any steps I can try in order to get closer to the cause of the system hang, please let me know

[Oxyd]

Since the entire system hangs, this is probably a problem either in the driver or the kernel. We're also rewriting the renderer for 0.17, so there's a chance it'll be fixed by that.

[thecatlover1996]

Thanks for your reply, I already read it in the FFF.
Good luck with the new engine!

[DiRaven]

Having exactly the same problem. Sincerely hoping it gets fixed in .17.

It's worth mentioning though that the problem is not always there. The game sometimes exits just fine. The issue tends to arise more often when the play session is long enough, or a few mod reloads happened (due to the new mods installed or deleted, for example).

[Muffindrake]

This has in fact not been resolved in 0.17 and is an issue isolated to the Nvidia closed source binary driver. The OP claims that pkill -9 lets you quit the game without freezing your system, but it is not so - it's a 100% certainty that your entire video/display will freeze if you launch it and the only way to recover from that is rebooting your system (try SysRq + REISUB).

This is a really serious problem that renders the game a risky business for anyone with an nvidia card and no way to use other graphics cards. I tested with 418.43 and kernel 4.19.20. I can still quit the game just fine if I run my entire session on my integrated Intel card rather than the dedicated Nvidia one, but isn't an acceptable workaround.

Note: this doesn't cause a kernel panic, but the nvidia driver completely locks up any display (I was still able to move my mouse here)

Since this issue was introduced in 0.16, please review your GPU cleanup routines and check what functions are being registered as signal handlers for cleanup purposes - I suspect the issue lies there.

[Mike5000]

FWIW I've never experienced this on my previous Nvidia nor my current NVidia Quadro P2000 - currently using driver 390.87 and kernel 4.19.0 in Linux Devuan Ascii using the TDE desktop.

Hard to guess what is causing this but hopefully more data points will help.

EDIT: Desktop.

[torment]

I was hoping this was fixed on 0.17, but it still happens

I have updated SO (Linux Mint), updated kernel, updated Nvidia drivers, and about 80% of game exits completelly freezes my computer

It is really annoying because it's not only when you click exit, but when you install/update mods, change mods settings, anything that makes the game restart, and sometimes you forget about this and lose anything not saved on workspace

And to get worse, when you change mod settings and freezes, sometimes when you reboot the change is not made, so you have to try again, resulting in 3 - 4 consecutive reboots..

[torment]

FWIW I've never experienced this on my previous Nvidia nor my current NVidia Quadro P2000 - currently using driver 390.87 and kernel 4.19.0 in Linux Devuan Ascii.

Hard to guess what is causing this but hopefully more data points will help.

maybe its something related to notebooks graphic cards?

Mine is GeForce GTX 960M

[talchas]

Yeah, the kill -9 was not actually sufficient. I thought I'd edited the post, but apparently not. And yeah, mine is also a 960M, and "run it in intel" does work. It didn't crash when exiting 0.17.1 for me on nvidia, but it was never 100% consistent and I only tried the once, so I'll make sure to go back to running it in intel.

[Muffindrake]

My graphics card are a nvidia 950m and an intel HD 530, for completeness sake.

[talchas]

Hmmm, doing like 10 quick attempts this morning after I'd lost power anyways, I got no crashes on 0.17.2, nvidia 418.43, xorg 1.20.3 and kernel 4.14.52-gentoo. I guess I'll go back to doing it on nvidia and see if it crashes again for me after longer playtimes.

[roothorick]

I've played on a desktop 660Ti, 760, 1070 and now 1080Ti on numerous binary driver versions and had no issues. This is probably hybrid graphics related. My only laptop is an Intel-only model so I can't do meaningful testing myself.

[Muffindrake]

So what I just did (I triple-checked my X11 session is running on the 950M) is play factorio for an hour and then quit the game (of course after 'sync'ing all my files first), and I didn't encounter a driver crash. This was done with game version 0.17.4.

I then launched the game (after Steam was done synchronizing my save files with a measly 60KiB/s upload) and immediately quit it. This used to trigger a crash. I will try twenty more times.

I didn't manage to trigger the crash a single time. I suspect this may have to do with graphics settings, as I've changed them to bare minimum due to having played on the weaker integrated card.



I then changed the graphics settings to nigh-maximum.



Starting the game with this setup already consumes 2230MiB of VRAM in the main menu (my graphics card has 4GB in total) - while the low-spec mode above consumed around 60MiB. Unlike the low-spec setup, these settings have me crash my video driver every single time I attempt to quit the game. I have a hunch the culprit is the "All" setting for VRAM usage - it would at least be nice to know how much VRAM the other settings are designed to use.

Of the other processes using any VRAM on my NVIDIA GPU, there's X11 at 251MiB, Chromium at 74MiB and Steam at 36MiB. They were also running every time I triggered the crash and I was not doing anything graphics-intensive like watching video while attempting to quit Factorio.

[thecatlover1996]

Seems to be fixed for me, based on one time playing 0.17.4 and shutting it down normally after that Using kernel 4.15.0-45 and Nvidia 390.77.

[torment]

Still happens for me, i believe you have to enable a lot of mods to trigger, because i disabled all mods and on this restart the game freezes, but without any mod, a restarted a couple of times and it's ok, enabled the mods again, and happened again..

I tried to lower the memory usage, to high and it did not work too



EDIT:
After put memory usage on medium and decrease 1 thread, i played a few hours yesterday without freezes on exit

[talchas]

Starting the game with this setup already consumes 2230MiB of VRAM in the main menu (my graphics card has 4GB in total) - while the low-spec mode above consumed around 60MiB. Unlike the low-spec setup, these settings have me crash my video driver every single time I attempt to quit the game. I have a hunch the culprit is the "All" setting for VRAM usage - it would at least be nice to know how much VRAM the other settings are designed to use.

Of the other processes using any VRAM on my NVIDIA GPU, there's X11 at 251MiB, Chromium at 74MiB and Steam at 36MiB. They were also running every time I triggered the crash and I was not doing anything graphics-intensive like watching video while attempting to quit Factorio.

Hmm, I've had high quality texture compression on, but otherwise was at max quality, this gave me 822 MiB of VRAM usage (and no longer hanging). I haven't tried no-compression or mods yet; I don't know what the memory usage was like when I was testing it in 0.16, I assume I at least tested vanilla and not only the big modpack I spent a lot of time playing. I think 0.16 defaulted to no texture compression though, and my leftover installs do have it disabled.