[2.0.21] Crash interacting with partially loaded train (RollingStock::disconnectNeighborAndHandleRailStateChanges)

[mminkov]

First things first, a couple chunks on Vulcanus got... lost? They seem to have moved very far away on the map.
The chunks were moved to an unloaded area (moving close to them triggers the worms to start destroying the items in the chunks).
Note that the missing chunks were *completely empty*, but I rebuilt some of the rails before I figured out what was going on.

A part of a train was on a "lost" chunk. The locomotives and some wagsons are still where they were - but not all wagons. Interacting with this train in any way breaks the game. Can't be moved, wagons can't be disconnected.If I move close enough for the missing wagon(s) to be loaded again, it crashes.

Screenshots, video, savegame attached.

pics

Screenshot 2024-11-27 112236.png (6.59 MiB) Viewed 507 times

Screenshot 2024-11-27 112224.png (7.42 MiB) Viewed 507 times

Screenshot 2024-11-27 112208.png (7.59 MiB) Viewed 507 times

[Loewchen]

Post the log of a session with the crash please, see 3638.

[mminkov]

My bad, I forgot to add that file. I loaded the save and generated a new log just now.

I ended up using blueprints creatively to restore my progress since my last save before the chunk thing.

Since then the "Vulcanus chunks being in the wrong place" issue happened 2 more times, with 2 different chunks, both on Vulcanus. Thankfully, no crashes, just things being on top of each other. Should this be a separate bug report? Is it a known issue?

[boskid]

This type of a corruption reminds me of viewtopic.php?p=569694#p569694 where a huge set of entities was moved on a surface. Unfortunately this type of corruption can only happen when saving or when loading (and then saving) because when saving a surface, entities are saved in columns of chunks and for save file size optimization purposes entity positions are often stored as delta position of previous saved entity, so if one of such deltas gets corrupted when saving, a huge block of entities will get loaded at incorrect positions.

When looking at one of the roboports that was moved, it looks like its original position was {-161, 189} however corrupted one is at position {-1019.496, -830.902}. Such a large position difference would not use the delta approach on saving so there has to be something even more suspicious happening. It is extremally unlikely to be an issue caused by Factorio itself, i would suggest doing a memtest to verify if your PC runs stable, even more suspicious part is that given the log file it looks like you have Intel CPU while 13th and 14th gen cpus were involved in some serious stability issues that may require RMA of the faulty CPU if you would be able to narrow the issue down to cpu being at fault.

Given that there are no signs of factorio fault while there are extreme signs of hardware being at fault i am moving this to 1/0 magic.

[mminkov]

You are correct about my CPU, and I have to say that I've been worried about this exact issue myself. I've seen a couple bluescreens in the last few months, and it's been on my mind. But "in general", the system is stable and I have not seen anything to make me think that the CPU is done. For what it's worth, I'm on the latest microcode that "fixes" it.

This is a 70+ hour save that has had no issues so far. Moreover, the chunk thing has never happened on any of the other planets, only on Vulcanus.
The Nauvis base is several times larger, it gets modified more often, and not one chunk has ever moved there.

While the Intel 13/14 gen issue is serious, it might be a bit premature to write off any bugs that are on CPUs that have about 50% market share between them.

I'm not saying that "the magical Vulcanus worms ate my RAM", just giving more info in case other people have the same problem.

[boskid]

Well, i am not saying it is definitely a failing hardware, i just see that there is a bunch of entities that were all moved by exactly the same offset and offset being too large for the delta approach so if there was a fault in the code, it would have to be specifically when loading (this offset could not be saved as a valid delta) where there is "last loaded position" value stored so if loading sees a deltas, it would be able to compute original position of the following entity while only having a position delta. For this exact type of corruption to happen, this "last loaded position" variable would have to be corrupted exactly between loading of previous entity and loading of first corrupted entity in the block. All following entities were simply using correct delta relative to incorrectly recovered last position. This is possibly the second time where i saw this issue (and the previous one that i linked was possible to explain as memory bit flip as it was a consistent 4 chunks move down within range of saving positions using delta approach) while first being that severe with entities being in an absolutely bonkers position.

I may be biased towards the 13/14 gen issue because i myself had a 14900k bought in january this year, in may it started failing and i had to RMA it as having at least 5 different apps crashing on average 20 times total a day was a little bit too much for me.

[Rseding91]

Crashes typically fall into 1 of 3 categories:

1. Obvious from the call stack what went wrong -> we fix it without needing reproduction steps (commonly just from the auto-crash logs)
2. Not obvious from the call stack but clearly broken game logic -> we need reproduction steps to figure it out and fix it
3. Not obvious from the call stack but code that runs hundreds of thousands to millions of times per second on every players game every time it's run -> *RARELY* a game bug (I can remember 2 instances in the past 10 years of Factorio work) -> commonly players report bad RAM/other program crashes.

Your crash falls cleanly into #3