[14.21] SIGSEGV in game then on loading save

[willsk]

I encountered a crash today while playing, then again on loading a recent save from the same session. I can reproduce this on two different computers, both running Linux (Fedora 25).

Code: Select all

   0.000 2017-01-25 19:09:27; Factorio 0.14.21 (build 25327, linux64, alpha)
   0.000 Operating system: Linux
   0.000 Program arguments: "Dropbox/factorio/bin/x64/factorio" 
   0.000 Read data path: /home/willsk/Dropbox/factorio/data
   0.000 Write data path: /home/willsk/Dropbox/factorio
   0.000 Binaries path: /home/willsk/Dropbox/factorio/bin
   0.018 System info: [CPU:         Intel(R) Core(TM) i5-2400 CPU @ 3.10GHz, 4 cores, RAM: 7966MB]
   0.019 Display options: [FullScreen: 0] [VSync: 1] [UIScale: 100%] [MultiSampling: OFF] [Screen: 255]
   0.076 Available display adapters: 1
   0.076  [0]: resolution 1920x1080px at [0,0]
   0.076 Create display on adapter 0. Size 1280x720 at position [310, 162].
   0.235 Initialised OpenGL:[0] GeForce GT 440/PCIe/SSE2; driver: 4.5.0 NVIDIA 375.26
   0.235     Video memory size (dedicated/total available/current available): 1024/1024/785 MB
   0.265 Graphics options: [Graphics quality: normal] [Video memory usage: high] [Light scale: 32%] [DXT: auto]
   0.374 Loading mod core 0.0.0 (data.lua)
   0.378 Loading mod base 0.14.21 (data.lua)
   0.459 Loading mod PickerExtended 1.0.3 (data.lua)
   0.514 Loading mod rso-mod 2.3.3 (data.lua)
   0.570 Loading mod TheFatController 1.0.2 (data.lua)
   0.626 Loading mod advanced-logistics-system 0.4.0 (data.lua)
   0.694 Loading mod alien-science 0.1.2 (data.lua)
   0.749 Loading mod auto-research 2.2.1 (data.lua)
   0.819 Loading mod autofill 1.4.6 (data.lua)
   0.875 Loading mod AutoTrash 1.0.1 (data.lua)
   0.932 Loading mod Bottleneck 0.6.2 (data.lua)
   1.008 Loading mod bullet-trails 0.1.2 (data.lua)
   1.064 Loading mod CharcoalBurner 1.0.5 (data.lua)
   1.122 Loading mod Crafting_Speed_Research 0.1.4 (data.lua)
   1.204 Loading mod EvoGUI 0.4.108 (data.lua)
   1.263 Loading mod Expanded_Robot_Tech 0.4.2 (data.lua)
   1.323 Loading mod FARL 0.7.4 (data.lua)
   1.401 Loading mod Foreman 1.1.5 (data.lua)
   1.460 Loading mod Larger Inventory 1.0.2 (data.lua)
   1.520 Loading mod Laser_Beam_Turrets 0.1.8 (data.lua)
   1.583 Loading mod ModuleInserter 1.0.0 (data.lua)
   1.666 Loading mod MoreLight 1.1.1 (data.lua)
   1.727 Loading mod radarplus for rso 0.0.3 (data.lua)
   1.786 Loading mod RailTanker 1.4.0 (data.lua)
   1.849 Loading mod ScoreExtended 1.0.22 (data.lua)
   1.943 Loading mod upgrade-planner 1.2.14 (data.lua)
   2.009 Loading mod YARM 0.7.108 (data.lua)
   2.086 Loading mod rso-mod 2.3.3 (data-updates.lua)
   2.163 Loading mod bullet-trails 0.1.2 (data-updates.lua)
   2.242 Loading mod FARL 0.7.4 (data-updates.lua)
   2.307 Loading mod FARL 0.7.4 (data-final-fixes.lua)
   2.397 Loading mod long-reach 0.0.6 (data-final-fixes.lua)
   2.462 Loading mod longer-belts-redux 0.14.3 (data-final-fixes.lua)
   2.538 Loading mod ModuleInserter 1.0.0 (data-final-fixes.lua)
   2.607 Loading mod RailTanker 1.4.0 (data-final-fixes.lua)
   2.671 Loading mod tree_collision 0.4.0 (data-final-fixes.lua)
   2.756 Checksum for core: 1291735968
   2.756 Checksum for mod base: 786624017
   2.756 Checksum for mod PickerExtended: 2390657668
   2.756 Checksum for mod rso-mod: 922899904
   2.756 Checksum for mod TheFatController: 3783390293
   2.756 Checksum for mod advanced-logistics-system: 2163846892
   2.756 Checksum for mod alien-science: 3782378071
   2.756 Checksum for mod auto-research: 1737942890
   2.756 Checksum for mod AutoDeconstruct: 0
   2.756 Checksum for mod autofill: 3086359458
   2.756 Checksum for mod AutoTrash: 845706699
   2.756 Checksum for mod Bottleneck: 4008105457
   2.756 Checksum for mod bullet-trails: 3097468026
   2.756 Checksum for mod CharcoalBurner: 2811892453
   2.757 Checksum for mod Crafting_Speed_Research: 3928230558
   2.757 Checksum for mod EvoGUI: 1040096666
   2.757 Checksum for mod Expanded_Robot_Tech: 2542801217
   2.757 Checksum for mod FARL: 3325049299
   2.757 Checksum for mod Foreman: 889675380
   2.757 Checksum for mod InitialScan: 0
   2.757 Checksum for mod Larger Inventory: 32563986
   2.757 Checksum for mod Laser_Beam_Turrets: 3880469623
   2.757 Checksum for mod long-reach: 3674062281
   2.757 Checksum for mod longer-belts-redux: 883706356
   2.757 Checksum for mod ModuleInserter: 1557794654
   2.757 Checksum for mod MoreLight: 3037131012
   2.757 Checksum for mod radarplus for rso: 2162590730
   2.757 Checksum for mod RailTanker: 1260574978
   2.757 Checksum for mod ScoreExtended: 2617688541
   2.757 Checksum for mod tree_collision: 3231141521
   2.757 Checksum for mod upgrade-planner: 696037660
   2.757 Checksum for mod YARM: 4206050284
   3.165 Info PlayerData.cpp:55: Local player-data.json available, timestamp 1485314343
   3.165 Info PlayerData.cpp:62: Cloud player-data.json unavailable
   3.333 Initial atlas bitmap size is 16384
   3.336 Created atlas bitmap 16384x3851
   3.337 Created atlas bitmap 4096x4080
   3.339 Created atlas bitmap 4096x4076
   3.340 Created atlas bitmap 4096x3992
   3.340 Created atlas bitmap 4096x1040
   3.341 Created atlas bitmap 4096x3656
  14.210 Sprites loaded
  14.212 Convert atlas 4096x4080 to: compressed 
  15.010 Convert atlas 4096x4076 to: compressed 
  15.907 Convert atlas 4096x3992 to: compressed 
  16.554 Convert atlas 4096x1040 to: trilinear-filtering 
  16.611 Convert atlas 4096x3656 to: mipmap 
  16.775 Loading sounds...
  18.341 Custom inputs active: 17
  18.373 Factorio initialised
  51.431 Loading map /home/willsk/Dropbox/factorio/saves/_autosave1.zip
  51.432 Error FileUtil.cpp:284: Can't remove /home/willsk/Dropbox/factorio/temp/currently-playing: No such file or directory
  51.448 Info Scenario.cpp:152: Map version 0.14.21-0
  51.775 Error CrashHandler.cpp:230: Received SIGSEGV
Factorio crashed. Generating symbolized stacktrace, please wait ...
  52.413 Warning Logger.cpp:369: Symbols.size() == 13, usedSize == 10
#0  0x000000000085c24d in Logger::logStacktrace(StackTraceInfo*) at /tmp/factorio-ZKDrW7/src/Util/Logger.cpp:391
#1  0x000000000063251b in CrashHandler::writeStackTrace() at /tmp/factorio-ZKDrW7/src/Util/CrashHandler.cpp:105
#2  0x00000000006327b4 in CrashHandler::SignalHandler(int) at /tmp/factorio-ZKDrW7/src/Util/CrashHandler.cpp:235
#3  0x00007fccfa74c990 in ?? at ??:0
#4  0x000000000091117c in load at /tmp/factorio-ZKDrW7/src/Map/Tile.cpp:254
#5  0x0000000000c1fed9 in load at /tmp/factorio-ZKDrW7/src/Surface/Chunk.cpp:347
#6  0x0000000000c2162b in load at /tmp/factorio-ZKDrW7/src/Surface/Surface.cpp:594
#7  0x00000000011a6bc0 in Map::Map(MapDeserialiser&, ProgressObserver*, bool) at /tmp/factorio-ZKDrW7/src/Map/Map.cpp:527
#8  0x00007fccfc4e66ca in loadFactory at /tmp/factorio-ZKDrW7/src/Scenario/Scenario.cpp:161
#9  0x00007fccfa81ef7f in ParallelScenarioLoader::doLoad(ParallelScenarioLoader*, MapDeserialiser*, InputType, InputHandler*, MultiplayerManagerBase*, bool) at /tmp/factorio-ZKDrW7/src/Scenario/ParallelScenarioLoader.cpp:121
  53.027 Warning Logger.cpp:369: Symbols.size() == 12, usedSize == 9
  53.027 Error Util.cpp:77: Unexpected error occurred. If you're running the latest version of the game you can help us solve the problem by posting the contents of the log file on the Factorio forums.
Please also include the save file(s), any mods you may be using, and any steps you know of to reproduce the crash.

I have attached a tarball of the abrt directory sans coredump (2.0Gb / ~150Mb xz file, available on request), and one containing the offending save and a few other files. These crashes are from loading the saved game. (an older save-backup also causes the crash), tomorrow morning (US EST time) I can get the stack trace / logs for the crash mid-game if needed. Please let me know if there is anything else I can provide!

EDIT: I just noticed these two similar recent reports: viewtopic.php?f=49&t=39467 and viewtopic.php?f=49&t=38488 not sure if Tile:254 is meaningful...

[Rseding91]

The saves are corrupt. It's trying to load the corrupt save and it ends up crashing.

This issue should be resolved for 0.15 but for now the only thing you can do is load an older save.

[willsk]

Right, I figured; interesting that all 3 autosaves are bad, so the game itself crashed long after writing several bad saves, any ideas on what causes it? (I doubt RAM corruption as I've got ECC RAM on one of the computers, but it's possible). Thanks for the reply, I'll post the other crash tomorrow if it looks interesting...

[Rseding91]

Right, I figured; interesting that all 3 autosaves are bad, so the game itself crashed long after writing several bad saves, any ideas on what causes it? (I doubt RAM corruption as I've got ECC RAM on one of the computers, but it's possible). Thanks for the reply, I'll post the other crash tomorrow if it looks interesting...

We've never been able to reproduce it and have only seen it in bug reports but we suspect it's related to chunk generation that runs in a threaded setup. I found a few issues with how chunks were being generated and fixed them for 0.15 but we still don't know for sure if it would trigger this issue.

So, as I said: it *should* be fixed for 0.15 but as we can't reproduce it we don't know for sure. It's always the same thing: 1 whole chunk is saved with tile IDs of 0 (invalid - no ID) which then crashes when loaded trying to use them to rebuild the chunk data.

[willsk]

Just in case this is interesting, below is the initial crash that happened first, before the one loading the corrupted save (if it's a result of rendering a corrupted map then probably isn't).

Code: Select all

1436.357 Error CrashHandler.cpp:230: Received SIGSEGV
Factorio crashed. Generating symbolized stacktrace, please wait ...
1437.284 Warning Logger.cpp:369: Symbols.size() == 14, usedSize == 9
#0  0x000000000085c24d in Logger::logStacktrace(StackTraceInfo*) at /tmp/factorio-ZKDrW7/src/Util/Logger.cpp:391
#1  0x000000000063251b in CrashHandler::writeStackTrace() at /tmp/factorio-ZKDrW7/src/Util/CrashHandler.cpp:105
#2  0x00000000006327b4 in CrashHandler::SignalHandler(int) at /tmp/factorio-ZKDrW7/src/Util/CrashHandler.cpp:235
#3  0x00007f4b576d9990 in ?? at ??:0
#4  0x000000000096dc3f in prepareRow at /tmp/factorio-ZKDrW7/src/Graphics/EntityRenderer.cpp:210
#5  0x0000000000636e6b in threadJob at /tmp/factorio-ZKDrW7/src/Graphics/EntityRenderer.cpp:161
#6  0x00000000011a6bc0 in swap<std::_Any_data> at /usr/include/c++/5/bits/move.h:186
#7  0x00007f4b594736ca in swap at /usr/include/c++/5/functional:2160
#8  0x00007f4b577abf7f in std::function<void ()>::function(std::function<void ()>&&) at /usr/include/c++/5/functional:2035
1437.762 Warning Logger.cpp:369: Symbols.size() == 13, usedSize == 8
1437.762 Error CrashHandler.cpp:106: Map tick at moment of crash: 4288014

[posila]

Btw. do you have some working autosave? If not we might be able to recover the broken save.

[willsk]

No, if you can that would be awesome! Attached is the latest autosave.

Thanks!

[posila]

I made a short attempt to fix the save but it is more complicated than usual. There are about 30 bytes with garbage data in them, its hard to figure out what valid data should be. I'll give it another try tomorrow.

[posila]

I managed to fix the save. Here you go.

[willsk]

You people are awesome, thank you!

[posila]

You're welcome Glad to help.

You said one of your computers have ECC RAM; is the one that produced those corrupted autosaves? (I am trying to lower probability of "cosmic-ray" explanation)

[willsk]

Yes, it was, although that doesn't rule out other sources of HW corruption; however I've noticed no other instabilities...