[kovarex] [0.16.47] Crash: "Shooter::update"

[veedub1955]

mod-list.json-latest versions of all mods

Code: Select all

{
  "mods": [
    {
      "name": "base",
      "enabled": true
    },
    {
      "name": "5dim_automatization",
      "enabled": true
    },
    {
      "name": "5dim_core",
      "enabled": true
    },
    {
      "name": "5dim_energy",
      "enabled": true
    },
    {
      "name": "5dim_logistic",
      "enabled": true
    },
    {
      "name": "5dim_mining",
      "enabled": true
    },
    {
      "name": "5dim_trains",
      "enabled": true
    },
    {
      "name": "5dim_transport",
      "enabled": true
    },
    {
      "name": "5dim_vehicle",
      "enabled": true
    },
    {
      "name": "aai-programmable-vehicles",
      "enabled": true
    },
    {
      "name": "aai-signals",
      "enabled": true
    },
    {
      "name": "aai-vehicles-chaingunner",
      "enabled": true
    },
    {
      "name": "aai-vehicles-hauler",
      "enabled": true
    },
    {
      "name": "aai-vehicles-miner",
      "enabled": true
    },
    {
      "name": "angelsaddons-pressuretanks",
      "enabled": true
    },
    {
      "name": "auto-research",
      "enabled": true
    },
    {
      "name": "AutoDeconstruct",
      "enabled": true
    },
    {
      "name": "autofill",
      "enabled": false
    },
    {
      "name": "AutoTrash",
      "enabled": false
    },
    {
      "name": "beltSorter",
      "enabled": false
    },
    {
      "name": "BigBags",
      "enabled": false
    },
    {
      "name": "Big_Brother",
      "enabled": true
    },
    {
      "name": "BlackMarket",
      "enabled": false
    },
    {
      "name": "bobassembly",
      "enabled": true
    },
    {
      "name": "bobinserters",
      "enabled": true
    },
    {
      "name": "boblibrary",
      "enabled": true
    },
    {
      "name": "boblogistics",
      "enabled": true
    },
    {
      "name": "bobmodules",
      "enabled": false
    },
    {
      "name": "bobpower",
      "enabled": true
    },
    {
      "name": "bobwarfare",
      "enabled": true
    },
    {
      "name": "Bottleneck",
      "enabled": true
    },
    {
      "name": "bullet-trails",
      "enabled": true
    },
    {
      "name": "clock",
      "enabled": true
    },
    {
      "name": "Crafting_Speed_Research",
      "enabled": true
    },
    {
      "name": "DoubleFurnace",
      "enabled": true
    },
    {
      "name": "Electric Furnaces",
      "enabled": true
    },
    {
      "name": "ElectricTrain",
      "enabled": true
    },
    {
      "name": "Enhanced_Map_Colors",
      "enabled": true
    },
    {
      "name": "even-distribution",
      "enabled": true
    },
    {
      "name": "Expanded_Robot_Tech",
      "enabled": true
    },
    {
      "name": "Factorissimo2",
      "enabled": true
    },
    {
      "name": "FARL",
      "enabled": true
    },
    {
      "name": "FastRemoveTiles",
      "enabled": true
    },
    {
      "name": "Flare Stack",
      "enabled": true
    },
    {
      "name": "Flow Control",
      "enabled": true
    },
    {
      "name": "Foreman",
      "enabled": true
    },
    {
      "name": "Geothermal",
      "enabled": true
    },
    {
      "name": "helmod",
      "enabled": true
    },
    {
      "name": "LB-Modular-Chests",
      "enabled": true
    },
    {
      "name": "LiquifyRawMaterials",
      "enabled": true
    },
    {
      "name": "LogisticTrainNetwork",
      "enabled": true
    },
    {
      "name": "miniloader",
      "enabled": true
    },
    {
      "name": "Nanobots",
      "enabled": true
    },
    {
      "name": "railloader",
      "enabled": true
    },
    {
      "name": "Regroup",
      "enabled": true
    },
    {
      "name": "research-queue",
      "enabled": true
    },
    {
      "name": "robotarmy",
      "enabled": true
    },
    {
      "name": "rso-mod",
      "enabled": true
    },
    {
      "name": "Senpais_Trains",
      "enabled": true
    },
    {
      "name": "shmfaststart",
      "enabled": true
    },
    {
      "name": "shmlamps",
      "enabled": true
    },
    {
      "name": "Side Inserters",
      "enabled": true
    },
    {
      "name": "Single-Splitter",
      "enabled": true
    },
    {
      "name": "SmartTrains",
      "enabled": true
    },
    {
      "name": "Squeak Through",
      "enabled": true
    },
    {
      "name": "TheFatController",
      "enabled": true
    },
    {
      "name": "UltimateBelts",
      "enabled": true
    },
    {
      "name": "upgrade-planner",
      "enabled": true
    },
    {
      "name": "Vehicle Wagon",
      "enabled": true
    },
    {
      "name": "VehicleSnap",
      "enabled": true
    },
    {
      "name": "WaiTex_Full",
      "enabled": true
    },
    {
      "name": "Warehousing",
      "enabled": true
    },
    {
      "name": "what-is-it-really-used-for",
      "enabled": true
    },
    {
      "name": "wireless-signals",
      "enabled": true
    },
    {
      "name": "YARM",
      "enabled": true
    }
  ]
}

Ever since I updated to 0.16.47 I've been experiencing random crashes, by random I mean totally random.. sometime it can be every 5 minutes and the next time can be 1 hour. I'm not doing the same thing before the crash (e.g. placing entities).

I'm running the latest version of all the mods I have installed.

If any additional information is required let me know.

[kovarex]

The bug is strange, as there is basically if for nullptr, and two lines later, the value is nullptr.
Could you try to do the steam check integrity thing on your install?

[veedub1955]

Hey kovarex. I've ran an integrity check on the Factorio installation - all files validated successfully.

I uninstalled the game completely, including the %APPDATA%/Factorio directory (backed up my saves) and reinstalled it with all my mods and I reproduce the error.

The issue started happening randomly when I updated to 0.16.47, I had the same set of mods for a while and didn't update any prior to the issue occurring, I have attached my save file if it helps.

[TruePikachu]

While I don't expect it to be relevant, have you managed to reproduce this _without_ any mods? My naïve guess right now is some sort of race condition, where the pointer is nulled out between the check and the dereference.

[Rseding91]

While I don't expect it to be relevant, have you managed to reproduce this _without_ any mods? My naïve guess right now is some sort of race condition, where the pointer is nulled out between the check and the dereference.

The game logic is single threaded and mods can't introduce race conditions since they're also single threaded.

Based off what Kovarex said the only time I've seen that logic in the past is when some external program (cheat engine in my case) had modified the in-memory version of the executable to add additional code between the if check and the use of the thing checked.

[TruePikachu]

Wasn't sure if some parts of the Lua runtime in the game multithreaded, which is why I suggested testing it without mods. Though, I also find it strange that OP would intentionally be modifying the executable's memory space and yet not suspect this issue was their own creation.

EDIT: Just checked the dump in WinDbg, this is a weird bug that does in fact look like image modification. Here's the ASM context:

Code: Select all

00007ff6`c8f03f99 488b157833c600  mov     rdx,qword ptr [factorio!PrototypeList<ItemPrototype>::sortedPrototypes (00007ff6`c9b67318)]
00007ff6`c8f03fa0 6685c0          test    ax,ax
00007ff6`c8f03fa3 740e            je      factorio!Shooter::update+0x233 (00007ff6`c8f03fb3)
00007ff6`c8f03fa5 0fb7c0          movzx   eax,ax
00007ff6`c8f03fa8 488b0cc2        mov     rcx,qword ptr [rdx+rax*8]

`Shooter::update+228`, the faulting IP, is the last line there.

The first line appears to get a pointer to the list of sorted prototypes. The second+third do the "null check" (though only on an offset into that array, strangely). The fourth and fifth attempt to read an element from the array, and trigger an invalid pointer read.

As recorded in the memory dump provided by OP, RAX is 0x5B, and RDX is 0xEB963FF7E0. This would result in an attempt to read 0xEB963FFAB8 (which looks well-defined), but the access violation recorded in the dump claims an attempted read of 0x55D26AD8. EDIT: Different addresses are written to the dump than the Factorio log. The log claims RAX=0x57 and RDX=0x55D26820 resulting in an address of 0x55D26AD8, which is what was reported as being problematic. I'd check `PrototypeList<ItemPrototype>::sortedPrototypes` for possible corruption; it wasn't null, but it also wasn't a valid address (and its contents aren't present in the dump file).

[veedub1955]

Thanks for everyone looking into this. Just to confirm I am not editing anything to do with the game other than installing mods, I am not using cheat engine or anything like that.

As explained earlier I also uninstalled and reinstalled the game and reproduced the error, the issue started randomly so if I try and reproduce it without mods it may take hours of in game time to replicate again.

[kovarex]

Wasn't sure if some parts of the Lua runtime in the game multithreaded, which is why I suggested testing it without mods. Though, I also find it strange that OP would intentionally be modifying the executable's memory space and yet not suspect this issue was their own creation.

EDIT: Just checked the dump in WinDbg, this is a weird bug that does in fact look like image modification. Here's the ASM context:

Code: Select all

00007ff6`c8f03f99 488b157833c600  mov     rdx,qword ptr [factorio!PrototypeList<ItemPrototype>::sortedPrototypes (00007ff6`c9b67318)]
00007ff6`c8f03fa0 6685c0          test    ax,ax
00007ff6`c8f03fa3 740e            je      factorio!Shooter::update+0x233 (00007ff6`c8f03fb3)
00007ff6`c8f03fa5 0fb7c0          movzx   eax,ax
00007ff6`c8f03fa8 488b0cc2        mov     rcx,qword ptr [rdx+rax*8]

`Shooter::update+228`, the faulting IP, is the last line there.

The first line appears to get a pointer to the list of sorted prototypes. The second+third do the "null check" (though only on an offset into that array, strangely). The fourth and fifth attempt to read an element from the array, and trigger an invalid pointer read.

As recorded in the memory dump provided by OP, RAX is 0x5B, and RDX is 0xEB963FF7E0. This would result in an attempt to read 0xEB963FFAB8 (which looks well-defined), but the access violation recorded in the dump claims an attempted read of 0x55D26AD8. EDIT: Different addresses are written to the dump than the Factorio log. The log claims RAX=0x57 and RDX=0x55D26820 resulting in an address of 0x55D26AD8, which is what was reported as being problematic. I'd check `PrototypeList<ItemPrototype>::sortedPrototypes` for possible corruption; it wasn't null, but it also wasn't a valid address (and its contents aren't present in the dump file).

That seems little bit off, as the corresponding code is (probably) this:

Code: Select all

 if (this->delayedShootingData)
  {
    // we cannot assert this or throw because this can actually happen
    // when a game is loaded with delayedShooting in progress but the gunID doesn't exist anymore
    if (!optionalShootingParameters && this->delayedShootingData->gunID.isZero())
      return this->clearDelayedShootingData(); // returns true so we can start shooting again
    const AttackParameters* attackParameters(this->delayedShootingData->gunID.isZero()
                                             ? optionalShootingParameters
                                             : static_cast<const GunPrototype*>(this->delayedShootingData->gunID.getPrototype())->attackParameters);

the this->delayedShootingData is null on the last line. Either the value was magicaly chagned, or the return didn't really return.

I loaded the save, and I'm going to let it run for some time to see if I could reproduce it as well.

[posila]

EDIT: Different addresses are written to the dump than the Factorio log. The log claims RAX=0x57 and R[DX=0x55D26820 resulting in an address of 0x55D26AD8, which is what was reported as being problematic. I'd check `PrototypeList<ItemPrototype>::sortedPrototypes` for possible corruption; it wasn't null, but it also wasn't a valid address (and its contents aren't present in the dump file).

asm.png (60.26 KiB) Viewed 4525 times

Not only RDX is invalid address, but RBX too. RBX should have been address of delayedShootingData (which is NULL), and was supposed to be used to read gunID into EAX, but if that happened, it would have crashed right there (on the line movzx eax, word ptr [rbx+4]) ... so given two registers have bogus value in them, which should have caused the crash earlier, it makes me think none of these instructions were ran and the execution jumped to this instruction (or one of the instructions before this one, but after reading address of sortedPrototypes).

Though, I also find it strange that OP would intentionally be modifying the executable's memory space and yet not suspect this issue was their own creation.

If it really is caused by bad exeuctable, it might not be intentional modification but random corruption of the executable (therefore OP should verify integrity of the game files in Steam)

[veedub1955]

If it really is caused by bad exeuctable, it might not be intentional modification but random corruption of the executable (therefore OP should verify integrity of the game files in Steam)

I have already checked the integrity, and reinstalled the game.

[kovarex]

I ran the save for 8 hours when I was away and it didn't crash.

I would suspect that something on your system might be faulty. Some antivirus or a virus making something nasty, system corruption, hardware problem? Do you have some other computer you could test on?

[TruePikachu]

I _did_ notice a DLL for HitmanPro attached in the loaded modules (`hmpalert`), which would be the first thing I'd disable for testing.

EDIT: If it's possible, could you attach another dump+log so I can see if the corruption follows a pattern (e.g. the second byte of RBX always having data instead of being null, or the crash site always being the same instruction)?

[veedub1955]

I played for over 2 hours and didn't experience a crash, I will try again tomorrow. (PS, I'm also running this on my laptop to test on that at the same time).

HitmanPro seems to be related to Sophos Anti-virus, which I have on my desktop and laptop so if it's my AV hopefully I'll be able to reproduce on my laptop too.