[raiguard][1.1.77] Crash when using paint bucket tool in editor

[ptx0]

I replaced an ocean with concrete using the editor and received this crash:

Code: Select all

40921.253 Player 0 ran command: game.player.force.chart_all()
40958.728 Player 0 ran command: game.player.force.chart_all()
40987.917 Player 0 ran command: game.player.force.chart_all()
43116.001 Player 0 ran command: game.player.force.chart_all()
Factorio crashed. Generating symbolized stacktrace, please wait ...
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (313): Logger::writeStacktrace(FileWriteStream*, StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (322): Logger::logStacktrace(StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (184): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7f3e62671f5f
0x7f3e626c0cac
gsignal
abort
0x7f3e62ca4bb2
0x7f3e62c95151
0x7f3e62c972c2
0x7f3e62c97847
0x7f3e62c9847a
0x7f3e62c993a9
0x7f3e62c9b630
0x7f3e62c9c0e2
0x7f3e62c9e1b8
../../../../libstdc++-v3/libsupc++/new_op.cc (50): operator new(unsigned long)
/home/build/gcc-11.2/include/c++/11.2.0/bits/basic_string.tcc (219): void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) [clone .constprop.0]
/home/build/gcc-11.2/include/c++/11.2.0/bits/basic_string.h (539): std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (75): TimeTravelLogBuffer::flush(Logger&)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (342): Logger::flushTimeTravelLogs()
/tmp/factorio-build-rF4Jmx/src/Util/Logging.cpp (76): Logging::flushFutureLogs()
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (626): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7f3e62671f5f
0x7f3e626c0cac
gsignal
abort
0x7f3e62ca4bb2
0x7f3e62c95151
0x7f3e62c972c2
0x7f3e62c98121
0x7f3e62ca2b3d
0x7f3e62ca5aa4
0x7f3e626bc2d0
0x7f3e626bee34
0x7f3e6273f0fb
0xffffffffffffffff
Stack trace logging done
43283.121 Error Util.cpp:97: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.

[ptx0]

There was about 3GB of memory free after the game crashed so it might have run out and not handled that very well.

[ptx0]

I free'd a bunch of memory before starting the game back up. I tried again to paintbucket the ocean. It worked.

Then, I tried to copy a large section of an outpost, and received:

Code: Select all

 462.421 Info AppManagerStates.cpp:1910: Saving finished
 588.445 Quitting: window closed.
 589.636 Time travel logging:
  69.422 Player 0 ran command: game.player.force.chart_all()
 363.115 Player 0 ran command: game.player.force.chart_all()
 448.098 Player 0 ran command: game.player.force.chart_all()
Factorio crashed. Generating symbolized stacktrace, please wait ...
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (313): Logger::writeStacktrace(FileWriteStream*, StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (322): Logger::logStacktrace(StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (184): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7f929a2e2f5f
0x7f929a331cac
gsignal
abort
0x7f929a915bb2
0x7f929a906151
0x7f929a9082c2
0x7f929a909121
0x7f929a913b3d
0x7f929a916aa4
0x7f929a32d2d0
0x7f929a32fe34
0x7f929a3b00fb
0xffffffffffffffff
Stack trace logging done
 589.766 Error Util.cpp:97: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.
 592.482 Uploading log file
 592.521 Error CrashHandler.cpp:635: Received SIGSEGV
Factorio crashed. Generating symbolized stacktrace, please wait ...
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (313): Logger::writeStacktrace(FileWriteStream*, StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (322): Logger::logStacktrace(StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (184): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7f929a2e2f5f
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (735): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::_M_mbegin() const
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (739): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::_M_begin()
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (1270): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::lower_bound(int const&)
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_map.h (1259): std::map<int, ChildProcessAgent::ChildRecord, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::lower_bound(int const&)
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_map.h (497): std::map<int, ChildProcessAgent::ChildRecord, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::operator[](int const&)
/tmp/factorio-build-rF4Jmx/src/ChildProcessAgent.cpp (109): ChildProcessAgent::fork(std::function<void (int)> const&, std::function<void (int, ChildProcessAgent::ProcessStatus)> const&)
/tmp/factorio-build-rF4Jmx/src/Util/SystemUtil.cpp (776): SystemUtil::runProcessDontWait(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (296): CrashHandler::uploadLogFile()
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (255): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7f929a2e2f5f
0x7f929a331cac
gsignal
abort
0x7f929a915bb2
0x7f929a906151
0x7f929a9082c2
0x7f929a909121
0x7f929a913b3d
0x7f929a916aa4
0x7f929a32d2d0
0x7f929a32fe34
0x7f929a3b00fb
0xffffffffffffffff
Stack trace logging done
 592.560 Error Util.cpp:97: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.
 592.560 Uploading log file

[Rseding91]

Thanks for the report. In both cases it ran out of RAM (or so the OS told it). When the game tries to allocate but is told "no" there's nothing we can do except abort the process. Typically places that are trying to allocate are right in the middle of game logic and we can't simply continue without the new RAM the logic needs.

[ptx0]

Thanks for the report. In both cases it ran out of RAM (or so the OS told it). When the game tries to allocate but is told "no" there's nothing we can do except abort the process. Typically places that are trying to allocate are right in the middle of game logic and we can't simply continue without the new RAM the logic needs.

I ran that again with `free -m` printing to the factorio log every 5 seconds and it had 10GB free before and in between the game showing 'Quitting: window closed' and the LogHandler trying to upload.

[ptx0]

Code: Select all

               total        used        free      shared  buff/cache   available
Mem:           60119       49092       10009         535        1017        9847
Swap:              0           0           0
2158.703 Quitting: window closed.
2160.671 Time travel logging:
 354.281 Player 0 ran command: game.player.force.chart_all()
 406.367 Player 0 ran command: game.player.force.chart_all()
 537.992 Player 0 ran command: game.player.force.chart_all()
 575.106 Player 0 ran command: game.player.force.chart_all()
 605.344 Player 0 ran command: game.player.force.chart_all()
 882.994 Player 0 ran command: game.player.force.chart_all()
Factorio crashed. Generating symbolized stacktrace, please wait ...
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (313): Logger::writeStacktrace(FileWriteStream*, StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (322): Logger::logStacktrace(StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (184): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7fd883aa9f5f
0x7fd883af8cac
gsignal
abort
0x7fd8840dcbb2
0x7fd8840cd151
0x7fd8840cf2c2
0x7fd8840d0121
0x7fd8840dab3d
0x7fd8840ddaa4
0x7fd883af42d0
0x7fd883af6e34
0x7fd883b770fb
0xffffffffffffffff
Stack trace logging done
2160.738 Error Util.cpp:97: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.
               total        used        free      shared  buff/cache   available
Mem:           60119       49121        9965         542        1033        9807
Swap:              0           0           0
2164.466 Uploading log file
2164.495 Error CrashHandler.cpp:635: Received SIGSEGV
Factorio crashed. Generating symbolized stacktrace, please wait ...
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (313): Logger::writeStacktrace(FileWriteStream*, StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/Logger.cpp (322): Logger::logStacktrace(StackTraceInfo*)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (184): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7fd883aa9f5f
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (735): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::_M_mbegin() const
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (739): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::_M_begin()
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_tree.h (1270): std::_Rb_tree<int, std::pair<int const, ChildProcessAgent::ChildRecord>, std::_Select1st<std::pair<int const, ChildProcessAgent::ChildRecord> >, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::lower_bound(int const&)
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_map.h (1259): std::map<int, ChildProcessAgent::ChildRecord, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::lower_bound(int const&)
/home/build/gcc-11.2/include/c++/11.2.0/bits/stl_map.h (497): std::map<int, ChildProcessAgent::ChildRecord, std::less<int>, std::allocator<std::pair<int const, ChildProcessAgent::ChildRecord> > >::operator[](int const&)
/tmp/factorio-build-rF4Jmx/src/ChildProcessAgent.cpp (109): ChildProcessAgent::fork(std::function<void (int)> const&, std::function<void (int, ChildProcessAgent::ProcessStatus)> const&)
/tmp/factorio-build-rF4Jmx/src/Util/SystemUtil.cpp (776): SystemUtil::runProcessDontWait(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > const&)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (296): CrashHandler::uploadLogFile()
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (255): CrashHandler::writeStackTrace(CrashHandler::CrashReason)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (636): CrashHandler::commonSignalHandler(int)
/tmp/factorio-build-rF4Jmx/src/Util/CrashHandler.cpp (642): CrashHandler::SignalHandler(int)
0x7fd883aa9f5f
0x7fd883af8cac
gsignal
abort
0x7fd8840dcbb2
0x7fd8840cd151
0x7fd8840cf2c2
0x7fd8840d0121
0x7fd8840dab3d
0x7fd8840ddaa4
0x7fd883af42d0
0x7fd883af6e34
0x7fd883b770fb
0xffffffffffffffff
Stack trace logging done
2164.519 Error Util.cpp:97: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.
2164.519 Uploading log file
               total        used        free      shared  buff/cache   available
Mem:           60119       41253       17837         536        1029       17679

[ptx0]

I've killed everything except the display server and Factorio, which leaves me 47GB free memory before starting the game, and then pasting the same large blueprint with "instant blueprint" enabled shows:

Code: Select all

7769.254 Info AppManagerStates.cpp:1910: Saving finished
7965.319 Info AppManagerStates.cpp:1910: Saving finished
8115.779 Quitting: window closed.
8117.562 Goodbye

there is no stack trace then.

[ptx0]

I'm using OS defaults and running Factorio without any env vars set - no (non-default) Wayland support enabled, no memory allocator changes, no libraries being overloaded.

Just normal `./bin/x64/factorio`

[ptx0]

I've done more testing. This is definitely a bug in the recently-added Wayland support.

Under Plasma, with Wayland enabled:

• 1.1.76 - Can paste extremely large blueprints
• 1.1.77 - Once the blueprint finishes pasting, the game quits with "Quitting: window closed."

Under Plasma + X11:

• 1.1.76 - Can paste extremely large blueprints
• 1.1.77 - Can paste extremely large blueprints

Under "Weston" (Wayland demo compositor):

• 1.1.76 - Can paste extremely large blueprints
• 1.1.77 - Once the blueprint finishes pasting, the game quits with "Quitting: window closed."

Sway unfortunately doesn't support NVIDIA devices unless I uninstall my graphics drivers.

[Rseding91]

I'll forward this to the wayland person.

[ptx0]

I'll forward this to the wayland person.

Thank you! It'll be very worthwhile once it's working fully. I get higher fps on that codepath. For anyone else hitting this issue, just try disabling instant blueprint for now, and use Raiguard's "Editor Extensions" construction bots to build.

[raiguard]

Thanks for the report. Unfortunately, this isn't something that can be easily fixed on Factorio's side. Essentially, when Factorio is frozen while placing a blueprint, input events will pile up in the Wayland connection buffer. This buffer is only 4096 bytes long, and when it fills up, the display server immediately kills the window. By contrast, X11 will expand and re-allocate the input buffer when it overflows, so it doesn't have this problem.

There is a merge request open to allow clients to increase the buffer size. Once that is merged and implemented, I will be able to add the requisite call to expand the buffer. But until then, there is nothing I can do.

To avoid this happening, try not to move your mouse on the window too much when the game is frozen. If too many mouse movement events pile up, Wayland will kill the window.

Wayland merge request: https://gitlab.freedesktop.org/wayland/ ... quests/188
Wayland issue: https://gitlab.freedesktop.org/wayland/ ... issues/159
SDL issue: https://github.com/libsdl-org/SDL/issues/5536

[ptx0]

hm, it looks like it even affects Xwayland. That's incredibly unfortunate, and it describes why the game still closes even without the new wayland support enabled? Can't figure out why it didn't happen before.

[raiguard]

Are you 100% certain that it happens on xwayland too? I can't reproduce it through xwayland.

[ptx0]

Are you 100% certain that it happens on xwayland too? I can't reproduce it through xwayland.

well, it might be that Xwayland is never getting used, even if the wayland support is disabled.

Basically, i don't set any env flags to cause the crash. but the Wayland support is supposed to be disabled then. it's possible the detection is broken, and it gets enabled unconditionally.

the crash doesn't happen when i'm in pure Xorg so that's why I assumed that Xwayland crashes, too.

[raiguard]

well, it might be that Xwayland is never getting used, even if the wayland support is disabled.

A way to know with 100% certainty is to check in the log. If you're on xwayland, it will show two XWAYLAND displays in place of your actual displays. For example, on my machine, it looks like this:

Code: Select all

   0.021 Available displays: 2
   0.021  [0]: XWAYLAND0 32" - {[2560,0], 2560x1440, SDL_PIXELFORMAT_RGB888, 60Hz}
   0.021  [1]: XWAYLAND2 25" - {[0,0], 2560x1440, SDL_PIXELFORMAT_RGB888, 60Hz}

Additionally, I just took a second look at the merge request, and it appears that they are also adding dynamic resizing of the input buffer, just like Xorg. So I shouldn't even need to add any code to Factorio to fix it. I will attempt to build libwayland with the merge request so I can verify this information.

[ptx0]

well, it might be that Xwayland is never getting used, even if the wayland support is disabled.

A way to know with 100% certainty is to check in the log. If you're on xwayland, it will show two XWAYLAND displays in place of your actual displays. For example, on my machine, it looks like this:

Code: Select all

   0.021 Available displays: 2
   0.021  [0]: XWAYLAND0 32" - {[2560,0], 2560x1440, SDL_PIXELFORMAT_RGB888, 60Hz}
   0.021  [1]: XWAYLAND2 25" - {[0,0], 2560x1440, SDL_PIXELFORMAT_RGB888, 60Hz}

Additionally, I just took a second look at the merge request, and it appears that they are also adding dynamic resizing of the input buffer, just like Xorg. So I shouldn't even need to add any code to Factorio to fix it. I will attempt to build libwayland with the merge request so I can verify this information.

It's also easy enough to do here, as I can drop it into `/etc/portage/patches/dev-libs/wayland`

And the old logs for my game, show:

Code: Select all

   0.022 Available displays: 1
   0.022  [0]: XWAYLAND0 16" - {[0,0], 2560x1600, SDL_PIXELFORMAT_RGB888, 60Hz}

though i wouldn't take this at face value, because it's from 1.1.74 instead. I will re-test later on 1.1.77 without the env var set for Wayland to see if this changes there.

[raiguard]

I built and installed the merge request, but it did nothing. After looking into it even more, it appears that the connection buffer size is completely controlled by the display server - clients can't "request" a larger one. So unfortunately, this is completely out of my hands, and the behavior may vary between compositors.