[Dominik][0.17.63] FluidSystem.cpp:171: connection.isConnected() was not true

[boskid]

Ref:
- 74252 - uses similar custom fluid fuzzer in control.lua
- 74116 - this issue may be reproduced by this script

Steps to reproduce:

1. Load "fluid_fuzzer-crash.zip" (more waiting, in case of early save corruption) or "fluid_fuzzer-crash-b.zip" (less waiting)

   fluid_fuzzer-crash.zip

   (169.13 KiB) Downloaded 129 times

   fluid_fuzzer-crash-b.zip

   (170.61 KiB) Downloaded 138 times

2. Increase speed to x64
3. Wait some time (65s for first save, 5s for second save)
4. Crash
   log

   Code: Select all

      0.000 2019-08-07 20:48:37; Factorio 0.17.63 (build 46739, win64, steam)

   Code: Select all

   Factorio crashed. Generating symbolized stacktrace, please wait ...
   c:\cygwin64\tmp\factorio-build-x93bju\src\fluid\fluidsystem.cpp (171): FluidSystem::addConnection
   c:\cygwin64\tmp\factorio-build-x93bju\src\fluid\fluidbox.cpp (1613): FluidBox::updateBlockedUndergroundConnections
   c:\cygwin64\tmp\factorio-build-x93bju\src\fluid\fluidsystem.cpp (433): FluidSystem::updateBlockedUndergroundConnections
   c:\cygwin64\tmp\factorio-build-x93bju\src\fluid\fluidsystem.cpp (307): FluidSystem::updateFluidIDAndFilters
   c:\cygwin64\tmp\factorio-build-x93bju\src\fluid\fluidmanager.cpp (286): FluidManager::moveOut
   c:\cygwin64\tmp\factorio-build-x93bju\src\entity\pipe.cpp (85): Pipe::onDestroy
   c:\cygwin64\tmp\factorio-build-x93bju\src\entity\entity.cpp (302): Entity::forceDestroy
   c:\cygwin64\tmp\factorio-build-x93bju\src\script\luaentity.cpp (683): LuaEntity::luaDestroy
   c:\cygwin64\tmp\factorio-build-x93bju\src\script\luabinder.hpp (326): LuaBinder<LuaLogisticContainerControlBehavior>::callWrapper
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\ldo.c (350): luaD_precall
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\lvm.c (710): luaV_execute
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\lapi.c (1027): f_call
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\ldo.c (137): luaD_rawrunprotected
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\ldo.c (623): luaD_pcall
   c:\cygwin64\tmp\factorio-build-x93bju\libraries\lua\lapi.c (1053): lua_pcallk
   c:\cygwin64\tmp\factorio-build-x93bju\src\util\luahelper.cpp (3520): LuaHelper::pCallWithStackTrace
   c:\cygwin64\tmp\factorio-build-x93bju\src\script\luagamescript.cpp (979): LuaGameScript::runNthTickHandler
   c:\cygwin64\tmp\factorio-build-x93bju\src\script\luaeventdispatcher.cpp (120): LuaEventDispatcher::dispatch
   c:\cygwin64\tmp\factorio-build-x93bju\src\scenario\scenario.cpp (1210): Scenario::update
   c:\cygwin64\tmp\factorio-build-x93bju\src\mainloop.cpp (1191): MainLoop::gameUpdateStep
   c:\cygwin64\tmp\factorio-build-x93bju\src\mainloop.cpp (1060): MainLoop::gameUpdateLoop
   c:\cygwin64\tmp\factorio-build-x93bju\src\util\workerthread.cpp (42): WorkerThread::loop

   Code: Select all

     84.723 Error FluidSystem.cpp:171: connection.isConnected() was not true

   Code: Select all

     84.723 Error CrashHandler.cpp:189: Map tick at moment of crash: 314372

   factorio-current.log

   (13 KiB) Downloaded 147 times

[boskid]

[0.17.64]

74269-manual-reproduction.gif (493.63 KiB) Viewed 3800 times

-- edit:
issue looks to be around building pipe-to-ground ghost (may be similar to 74252), where unconnected pipe-to-ground pair is now broken: left one connects to ghost but right one connects(not exactly because of fluid mixing) to left one. This state is not stable save-loadable, because after save-load, right pipe-to-ground will disconnect.

This also produces some heavy-mode desync when right fluid system is changed somehow (part of gif where i build additional pipe above existing one). With heavy-mode enabled, this may crash after building pipe above existing pipe with water, without heavy-mode it will crash on removing pipe with water on assembler output.

log of heavy mode crashing game

Code: Select all

 133.801 Error CrashHandler.cpp:462: Exception Code: c0000005, Address: 0x00007ff74ea40a7e
ModuleBase: 0x00007ff74db00000, ImageSize: 01760000, RelativeAddress: 00f40a7e
 133.801 Error CrashHandler.cpp:468: Access Violation: Read at address 0000000000000000

Code: Select all

f:\dd\vctools\crt\vcruntime\src\string\amd64\memcpy.asm (115): memcpy_repmovs
c:\cygwin64\tmp\factorio-build-6ights\src\util\streams\threadedbufferedwritestream.cpp (45): ThreadedBufferedWriteStream::write
c:\cygwin64\tmp\factorio-build-6ights\src\util\serialiserhelper.hpp (198): SerialiserHelper<MapSerialiser>::Saver<std::basic_string<char,std::char_traits<char>,std::allocator<char> >,void>::save
c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidbox.cpp (963): FluidBox::save
c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidsystem.cpp (111): FluidSystem::save
c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidmanager.cpp (66): FluidManager::save
c:\cygwin64\tmp\factorio-build-6ights\src\map\map.cpp (1328): Map::save
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1529): MainLoop::saveMapWithTags
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1544): MainLoop::checkHeavyModeCrc
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1145): MainLoop::gameUpdateStep
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1036): MainLoop::gameUpdateLoop
c:\cygwin64\tmp\factorio-build-6ights\src\util\workerthread.cpp (42): WorkerThread::loop

-- edit:
Another crash when testing pipe-to-ground ghost:

log of crash in entity get selection box

Code: Select all

c:\cygwin64\tmp\factorio-build-6ights\src\entity\entity.cpp (1497): Entity::getSelectionBox
c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidbox.cpp (327): FluidBox::drawUndergroundConnections
c:\cygwin64\tmp\factorio-build-6ights\src\entity\entity.cpp (2653): Entity::drawUndergroundConnections
c:\cygwin64\tmp\factorio-build-6ights\src\graphics\cursorrenderer.cpp (260): CursorRenderer::renderEntitySelection
c:\cygwin64\tmp\factorio-build-6ights\src\graphics\cursorrenderer.cpp (173): CursorRenderer::renderPlayerCursor
c:\cygwin64\tmp\factorio-build-6ights\src\graphics\cursorrenderer.cpp (78): CursorRenderer::prepare
c:\cygwin64\tmp\factorio-build-6ights\src\graphics\gamerenderer.cpp (150): GameRenderer::prepareJob
c:\cygwin64\tmp\factorio-build-6ights\src\graphics\gamerenderer.cpp (130): GameRenderer::prepare
c:\cygwin64\tmp\factorio-build-6ights\src\gameview.cpp (1193): GameView::prepareRender
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (971): MainLoop::prepare
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (632): MainLoop::mainLoopStep
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (377): MainLoop::run

-- edit:
And another:

crash in FluidBox notifyBlockedUndergroundConnectionsOfDestroy

Code: Select all

c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidbox.cpp (1625): FluidBox::notifyBlockedUndergroundConnectionsOfDestroy
c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidbox.cpp (1053): FluidBox::onDestroy
c:\cygwin64\tmp\factorio-build-6ights\src\entity\entity.cpp (302): Entity::forceDestroy
c:\cygwin64\tmp\factorio-build-6ights\src\manualstorage.cpp (215): ManualStorage::handleMinedEntity
c:\cygwin64\tmp\factorio-build-6ights\src\manualminer.cpp (124): ManualMiner::performMining
c:\cygwin64\tmp\factorio-build-6ights\src\controller\editorcontroller.cpp (258): EditorController::update
c:\cygwin64\tmp\factorio-build-6ights\src\map\map.cpp (1484): Map::updateEntities
c:\cygwin64\tmp\factorio-build-6ights\src\game.cpp (201): Game::update
c:\cygwin64\tmp\factorio-build-6ights\src\scenario\scenario.cpp (1215): Scenario::update
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1167): MainLoop::gameUpdateStep
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1036): MainLoop::gameUpdateLoop
c:\cygwin64\tmp\factorio-build-6ights\src\util\workerthread.cpp (42): WorkerThread::loop

-- edit:
And yet another one:

reproduction

74269-blocked-connection-mismatch-repro.gif (467.43 KiB) Viewed 3774 times

crash in consistency

Code: Select all

c:\cygwin64\tmp\factorio-build-6ights\src\map\map.cpp (2031): Map::checkConsistency
c:\cygwin64\tmp\factorio-build-6ights\src\script\luagamescript.cpp (6139): LuaGameScript::postUpdate
c:\cygwin64\tmp\factorio-build-6ights\src\scenario\scenario.cpp (1117): Scenario::postUpdate
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1168): MainLoop::gameUpdateStep
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1036): MainLoop::gameUpdateLoop
c:\cygwin64\tmp\factorio-build-6ights\src\util\workerthread.cpp (42): WorkerThread::loop
c:\program files (x86)\microsoft visual studio\2017\buildtools\vc\tools\msvc\14.15.26726\include\thr\xthread (230): std::_LaunchPad<std::unique_ptr<std::tuple<void (__cdecl TransferSource::*)(void),TransferSource *>,std::default_delete<std::tuple<void (__cdecl TransferSource::*)(void),TransferSource *> > > >::_Go
c:\program files (x86)\microsoft visual studio\2017\buildtools\vc\tools\msvc\14.15.26726\include\thr\xthread (209): std::_Pad::_Call_func
minkernel\crts\ucrt\src\appcrt\startup\thread.cpp (115): thread_start<unsigned int (__cdecl*)(void * __ptr64)>
ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 00007FF9F71C7BD4)
00007FF9F71C7BD4 (KERNEL32): (filename not available): BaseThreadInitThunk
ERROR: SymGetLineFromAddr64, GetLastError: 487 (Address: 00007FF9F8A4CE71)
00007FF9F8A4CE71 (ntdll): (filename not available): RtlUserThreadStart
Stack trace logging done
 127.539 Error FluidSystem.cpp:243: this->blockedConnectionCount == blockedConnectionCounter was not true

-- edit:
Factorio Pandora's Box:

reproduction

74269-compatible-connection-disconnected.gif (942.91 KiB) Viewed 3770 times

crash compatible connection is disconnected

Code: Select all

c:\cygwin64\tmp\factorio-build-6ights\src\fluid\fluidbox.cpp (578): FluidBox::checkConsistency
c:\cygwin64\tmp\factorio-build-6ights\src\map\map.cpp (2031): Map::checkConsistency
c:\cygwin64\tmp\factorio-build-6ights\src\script\luagamescript.cpp (6139): LuaGameScript::postUpdate
c:\cygwin64\tmp\factorio-build-6ights\src\scenario\scenario.cpp (1117): Scenario::postUpdate
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1168): MainLoop::gameUpdateStep
c:\cygwin64\tmp\factorio-build-6ights\src\mainloop.cpp (1036): MainLoop::gameUpdateLoop
c:\cygwin64\tmp\factorio-build-6ights\src\util\workerthread.cpp (42): WorkerThread::loop

Code: Select all

 173.746 Error FluidBox.cpp:578: Compatible connection is disconnected

[boskid]

Just LOL: new vanilla feature: underground connection from pipe-to-ground directly to pipe, and diagonal!

74269-dangling-pointer.gif (519.44 KiB) Viewed 3765 times

Most likely after removal of pipe-to-ground in center, right pipe-to-ground points to "related pipe-to-ground" that is already removed. New pipe that is placed when created gets same address in memory as removed pipe-to-ground and so connection is drawn diagonal to regular pipe

[steinio]

Can you please stop breaking the game

[boskid]

Can you please stop breaking the game

I will when this bug report gets assigned to someone (@Dominik?) - then i will know devs admit it is a bug with "enough reproduction" (and laugh) (and huge spike on crash tracker)

[Dominik]

Yes, this is very closely related to the 74252 but still a separate issue. I should have thought of these ghost things.

I was on a holiday. I come back and find this mayhem. Want to go back.

[Jap2.0]

I was on a holiday. I come back and find this mayhem. Want to go back.

This whole fluid system rework has been way more fun than you expected, hasn't it?