[2.0.76] Crash loading save (TrainManager::checkConsistency)

[Katara]

Link to save on dropbox:
https://www.dropbox.com/scl/fi/ue5e1wjv ... fwzub&dl=0

Log:

factorio-current.log

(13.8 KiB) Downloaded 16 times

Game version: 2.0.76 (Space Age)
Mods:
Auto Deconstruct, Biter Cleanup, Clean Floor, Disco Science, Even Distribution, Factorio Library, HD planets, Module Inserter Simplified, Rate Calculator, Remove Biter Soil & Nuke Scorch Marks, Show Max Underground Distance, Stats GUI, Text Plates, Unrestricted Achievements, Unrestricted Achievements (Space Age), Visible Planets in Space

Problem:
All saves from this world after a certain point crash on load with:
TrainManager::checkConsistency (!signal->reservedFor)

Details:
~100 autosaves all crash the same way
Older saves from before Space Age load fine
Other worlds load fine

Steps:
Load attached save
Game crashes during map load
Crashes the same way with all mods except official space age content disabled

Notes:
Seems related to train system consistency after Space Age upgrade.

[pioruns]

Your CPU belongs to the Raptor Lake family, which has known issues that can lead to permanent hardware degradation under certain conditions. If you're experiencing frequent crashes, especially during gaming or high-load scenarios, your CPU may already be damaged. This is not an isolated case, it's a widely reported and acknowledged problem.

More technical details and analysis can be found here:
https://machaddr.substack.com/p/extensi ... the-raptor

If your CPU is still under warranty, initiate an RMA process with Intel or your hardware vendor. Report the issue as repeated instability and crashing during games or other intensive workloads. Do not delay, as continued use might worsen the condition.

If an RMA isn't possible, the next step is to update your motherboard BIOS to the latest version, as several vendors have released firmware patches intended to mitigate this issue. While this won't fix already-present damage, it may help reduce instability going forward.

Other than that, there is nothing else you can do, permanent hardware damage cannot be fixed by the game or other software.

[Tertius]

Intel extended warranty to 5 years after date of purchase, so this CPU can be RMA'd. Don't hesitate to do this, your symptoms suggest your CPU took permanent damage from the manufacturing flaw. No Bios update afterwards is able to repair, the CPU is permanently damaged. Do update the Bios when you got a replacement.

Stability goals you can reach with proper hardware including undamaged Raptor Lake CPUs:
- Factorio crashes: never in 5 years
- Windows 11 BSOD: once per 1-2 years

[sushi_eater]

I've had consistent crashes that were blamed on the hardware:
viewtopic.php?t=121740

I'm pretty much certain that there was (perhaps is) a very rare, highly timing-dependent Factorio multi-threading bug responsible (I'm almost certain that there was/maybe is a Factorio bug with memory getting accessed after being freed in extremely rare, heavily timing-dependent scenarios).

The crashes were reproducible on my system. I haven't had ANY other crashes. The same crash doesn't happen anymore on the same hardware, a few Linux kernel/scheduler updates later (I tried going back to the same Factorio version/test save).

(I'm a software engineer with extremely extensive multi-threaded development experience.)

If you have issues at random locations (looking at the backtraces), it's almost certainly a hardware issue (e.g unstable CPU or RAM). If you experience issues at the same location, it's most likely a Factorio bug.

[pioruns]

Intel extended warranty to 5 years after date of purchase, so this CPU can be RMA'd. Don't hesitate to do this, your symptoms suggest your CPU took permanent damage from the manufacturing flaw. No Bios update afterwards is able to repair, the CPU is permanently damaged. Do update the Bios when you got a replacement.

Stability goals you can reach with proper hardware including undamaged Raptor Lake CPUs:
- Factorio crashes: never in 5 years
- Windows 11 BSOD: once per 1-2 years

With reliable hardware and if you move away from a toy operating system, you can achieve even better stability.
For example, in my setup using Linux, RAID1 for data, and ECC RAM:

• Factorio crashes: none in the last 5 years
• Linux kernel panic, crash, or hang: none in the last 5 years
• computer been running 24/7 for year

And of course move away from Raptor Lake CPUs as far as possible. Intel really shot itself in a foot with this one.

[Katara]

Thanks for all your replies. I've been running some CPU tests to help see if I can get RMAd. Through prime95 I did find some errors that quickly showed up due to my ram clock speed being a bit too high, but reverting to 4800mhz, zero errors after a 3.5 hour run. But this is defo something I'll be looking further into as I was under the impression for a while that it only affected the 14th gen line.

More interestingly, I actually tried loading up the save on another computer, (as I believe sushi_eater has tried too) runs on a Intel Ultra 7 155H and the saves fail to load on there as well.

The crashes were reproducible on my system. I haven't had ANY other crashes. The same crash doesn't happen anymore on the same hardware, a few Linux kernel/scheduler updates later (I tried going back to the same Factorio version/test save).

If you have issues at random locations (looking at the backtraces), it's almost certainly a hardware issue (e.g unstable CPU or RAM). If you experience issues at the same location, it's most likely a Factorio bug.

So to clarify you weren't able to get the save to load, and do you have any indication as to whether its a hardware or game issue?

Another piece of info which may or may not be useful. I have had saves before that would refuse to load in a similar way, which is why for the past year I do individual saves rather than overwriting saves. It still maybe occurred 2 or 3 times the past year, but that would mean I only lose a few hours of progress rather than days or the entire save. Also when the save would refuse to load, all the saves (including autosaves) made in that same previous game session would also refuse to load.

I've attached the log from the other computer that also crashed here:

factorio-current.log

(14.89 KiB) Downloaded 6 times

Any further pointers I would really appreciate. Even though the save is several hundred hours, I completely understand if this is something that can't be fixed and would be grateful if anyone knows how to prevent this happening again down the line.

[Loewchen]

More interestingly, I actually tried loading up the save on another computer, (as I believe sushi_eater has tried too) runs on a Intel Ultra 7 155H and the saves fail to load on there as well.

The save file is corrupted, it will not load on any hardware. The cause for the corruption is the part of interest but only if you could actually reproduce it.