[Pending] Headless Server with Mods segfault

[malk0lm]

I'm sure this is mod related but here is the details...

Code: Select all

# gdb --args ./factorio --start-server AerodoDotMe060316_v2
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./factorio...done.
run
(gdb) run
Starting program: /srv/factorio/bin/x64/factorio --start-server AerodoDotMe060316_v2
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
   0.000 2016-06-07 14:25:49; Factorio 0.12.35 (Build 18124, linux64, headless)
   0.066 Operating system: Linux (Ubuntu 14.04)
   0.066 Program arguments: "/srv/factorio/bin/x64/factorio" "--start-server" "AerodoDotMe060316_v2"
   0.066 Read data path: /srv/factorio/data
   0.066 Write data path: /srv/factorio
   0.066 Binaries path: /srv/factorio/bin
   0.077 Running in headless mode
   0.125 Loading mod core 0.0.0 (data.lua)
   0.129 Loading mod base 0.12.35 (data.lua)
   0.221 Loading mod advanced-logistics-system 0.2.13 (data.lua)
   0.294 Loading mod Foreman 0.1.22 (data.lua)
   0.377 Loading mod Larger Inventory 1.0.0 (data.lua)
   0.451 Loading mod Logistics Railway 1.0.8 (data.lua)
   0.536 Loading mod Orbital Ion Cannon 1.2.1 (data.lua)
   0.621 Loading mod RailTanker 1.3.0 (data.lua)
   0.711 Loading mod upgrade-planner 1.1.10 (data.lua)
   0.820 Loading mod YARM 0.7.15 (data.lua)
   0.879 Loading mod Yuoki 0.2.45 (data.lua)
   0.987 Loading mod TheFatController 0.4.15 (data.lua)
   1.125 Loading mod yi_engines 0.1.8 (data.lua)
   1.254 Loading mod FARL 0.5.26 (data.lua)
   1.352 Loading mod yi_pfw 0.0.9 (data.lua)
   1.484 Loading mod Orbital Ion Cannon 1.2.1 (data-updates.lua)
   1.580 Loading mod rso-mod 1.5.6 (data-updates.lua)
   1.676 Loading mod FARL 0.5.26 (data-updates.lua)
   1.811 Loading mod advanced-logistics-system 0.2.13 (data-final-fixes.lua)
   1.925 Loading mod long-reach 0.0.4 (data-final-fixes.lua)
   2.054 Loading mod RailTanker 1.3.0 (data-final-fixes.lua)
   2.220 Loading mod FARL 0.5.26 (data-final-fixes.lua)
   2.361 Checksum for core: 3711505753
   2.361 Checksum for mod base: 2829195351
   2.361 Checksum for mod advanced-logistics-system: 3751780125
   2.361 Checksum for mod Foreman: 2914203519
   2.361 Checksum for mod Larger Inventory: 32563986
   2.361 Checksum for mod Logistics Railway: 1014020522
   2.361 Checksum for mod long-reach: 3674062281
   2.362 Checksum for mod Orbital Ion Cannon: 380781624
   2.362 Checksum for mod RailTanker: 4126319180
   2.362 Checksum for mod rso-mod: 2980975201
   2.362 Checksum for mod upgrade-planner: 795218219
   2.362 Checksum for mod YARM: 1978499928
   2.362 Checksum for mod Yuoki: 1403913410
   2.362 Checksum for mod TheFatController: 2969438843
   2.362 Checksum for mod yi_engines: 1741433325
   2.362 Checksum for mod FARL: 1177259896
   2.362 Checksum for mod yi_pfw: 1381579888
[New Thread 0x7ffff70d2700 (LWP 31907)]
[New Thread 0x7ffff68d1700 (LWP 31908)]
[New Thread 0x7ffff60d0700 (LWP 31909)]
   3.096 Factorio initialised
   3.097 Info Router.cpp:509: Router peerID(65535) shutting down.
   3.097 Info Router.cpp:536: Router state -> Disconnected
   3.097 Info MultiplayerManager.cpp:913: networkTick(0) mapTick(-1) changing state from(Ready) to(PreparedToHostGame)
   3.097 Info MultiplayerManager.cpp:913: networkTick(0) mapTick(-1) changing state from(PreparedToHostGame) to(CreatingGame)
[New Thread 0x7ffff58cf700 (LWP 31910)]
   3.098 Loading map /srv/factorio/saves/AerodoDotMe060316_v2.zip
   3.121 Info Scenario.cpp:124: Map version 0.12.35-0

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff58cf700 (LWP 31910)]
_int_malloc (av=0x7ffff0000020, bytes=96) at malloc.c:3302
3302    malloc.c: No such file or directory.
(gdb) bt
#0  _int_malloc (av=0x7ffff0000020, bytes=96) at malloc.c:3302
#1  0x00007ffff71556c0 in __GI___libc_malloc (bytes=96) at malloc.c:2891
#2  0x0000000000d61c18 in operator new(unsigned long) ()
#3  0x00000000004a9202 in loadCommand () at src/AI/Command.cpp:29
#4  0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#5  loadCommand () at src/AI/Command.cpp:29
#6  0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#7  loadCommand () at src/AI/Command.cpp:29
#8  0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#9  loadCommand () at src/AI/Command.cpp:29
#10 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#11 loadCommand () at src/AI/Command.cpp:29
#12 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#13 loadCommand () at src/AI/Command.cpp:29
#14 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#15 loadCommand () at src/AI/Command.cpp:29
#16 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#17 loadCommand () at src/AI/Command.cpp:29
#18 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#19 loadCommand () at src/AI/Command.cpp:29
#20 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#21 loadCommand () at src/AI/Command.cpp:29
#22 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#23 loadCommand () at src/AI/Command.cpp:29
#24 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#25 loadCommand () at src/AI/Command.cpp:29
#26 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#27 loadCommand () at src/AI/Command.cpp:29
#28 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#29 loadCommand () at src/AI/Command.cpp:29
#30 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#31 loadCommand () at src/AI/Command.cpp:29
#32 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#33 loadCommand () at src/AI/Command.cpp:29
#34 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#35 loadCommand () at src/AI/Command.cpp:29
#36 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#37 loadCommand () at src/AI/Command.cpp:29
#38 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#39 loadCommand () at src/AI/Command.cpp:29
#40 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#41 loadCommand () at src/AI/Command.cpp:29
#42 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#43 loadCommand () at src/AI/Command.cpp:29
#44 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#45 loadCommand () at src/AI/Command.cpp:29
#46 0x00000000004a92c7 in CompoundCommand () at src/AI/Command.cpp:245
#47 loadCommand () at src/AI/Command.cpp:29
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) c
Continuing.

Program received signal SIGSEGV, Segmentation fault.
_int_malloc (av=0x7ffff0000020, bytes=96) at malloc.c:3302
3302    in malloc.c
(gdb) c
Continuing.
[Thread 0x7ffff58cf700 (LWP 31910) exited]
[Thread 0x7ffff60d0700 (LWP 31909) exited]
[Thread 0x7ffff68d1700 (LWP 31908) exited]
[Thread 0x7ffff7fec880 (LWP 31901) exited]

Program terminated with signal SIGSEGV, Segmentation fault.
The program no longer exists.
(gdb)

[Rseding91]

How much RAM does the server have available? Can you upload the save file + mods? Is it reproducible?

[malk0lm]

Code: Select all

# free -h
             total       used       free     shared    buffers     cached
Mem:          993M       333M       660M        31M        23M       125M
-/+ buffers/cache:       184M       809M
Swap:         1.0G       136K       1.0G

Mod bundle is here: https://github.com/malk0lm/factorio-aer ... s/tag/v1.2 or https://aerodo.me

Actual mods being used are:

Code: Select all

   0.105 Running in headless mode
   0.149 Loading mod core 0.0.0 (data.lua)
   0.152 Loading mod base 0.12.35 (data.lua)
   0.282 Loading mod advanced-logistics-system 0.2.13 (data.lua)
   0.354 Loading mod Foreman 0.1.22 (data.lua)
   0.423 Loading mod Larger Inventory 1.0.0 (data.lua)
   0.486 Loading mod Logistics Railway 1.0.8 (data.lua)
   0.554 Loading mod Orbital Ion Cannon 1.2.1 (data.lua)
   0.631 Loading mod RailTanker 1.3.0 (data.lua)
   0.698 Loading mod upgrade-planner 1.1.10 (data.lua)
   0.790 Loading mod YARM 0.7.15 (data.lua)
   0.857 Loading mod Yuoki 0.2.45 (data.lua)
   0.953 Loading mod TheFatController 0.4.15 (data.lua)
   1.062 Loading mod yi_engines 0.1.8 (data.lua)
   1.154 Loading mod FARL 0.5.26 (data.lua)
   1.258 Loading mod yi_pfw 0.0.9 (data.lua)
   1.410 Loading mod Orbital Ion Cannon 1.2.1 (data-updates.lua)
   1.512 Loading mod rso-mod 1.5.6 (data-updates.lua)
   1.615 Loading mod FARL 0.5.26 (data-updates.lua)
   1.749 Loading mod advanced-logistics-system 0.2.13 (data-final-fixes.lua)
   1.902 Loading mod long-reach 0.0.4 (data-final-fixes.lua)
   2.039 Loading mod RailTanker 1.3.0 (data-final-fixes.lua)
   2.220 Loading mod FARL 0.5.26 (data-final-fixes.lua)
   2.357 Checksum for core: 3711505753
   2.358 Checksum for mod base: 2829195351
   2.358 Checksum for mod advanced-logistics-system: 3751780125
   2.358 Checksum for mod Foreman: 2914203519
   2.358 Checksum for mod Larger Inventory: 32563986
   2.358 Checksum for mod Logistics Railway: 1014020522
   2.358 Checksum for mod long-reach: 3674062281
   2.358 Checksum for mod Orbital Ion Cannon: 380781624
   2.358 Checksum for mod RailTanker: 4126319180
   2.358 Checksum for mod rso-mod: 2980975201
   2.358 Checksum for mod upgrade-planner: 795218219
   2.359 Checksum for mod YARM: 1978499928
   2.359 Checksum for mod Yuoki: 1403913410
   2.359 Checksum for mod TheFatController: 2969438843
   2.359 Checksum for mod yi_engines: 1741433325
   2.359 Checksum for mod FARL: 1177259896
   2.359 Checksum for mod yi_pfw: 1381579888

Savegame is here: https://github.com/malk0lm/factorio-aer ... 316_v2.zip

It reproduces everytime I try to start up the server.

[Oxyd]

So how did you get this save? What happened between last working save and the save you posted here? Can you find some reproducible steps to go from a working save to a save with this problem?

[malk0lm]

So I think this is the last working save: https://github.com/malk0lm/factorio-aer ... e1.zip.zip

What happened was that the many belts on the bus are maxing out CPU (I suspect rendering thread is overworked with position updates). So I shutdown the server and then loaded this save locally and replaced all the belts with underground belt. Then I uploaded the new save and started up the server again.

I'm not really sure what the deal is but I think thats all that we did.

[malk0lm]

Oh, the other thing we did was disable the robot army mod because that was too processor intensive as well.

[Oxyd]

Robot Army definitely sounds like a likely suspect. The save won't load because it contains compound commands nested so deep it overflows the call stack. The real question is how that happened.

Unfortunately, I still haven't been able to reproduce it. I can load your earlier save just fine, with or without Robot Army, and saving and reloading doesn't cause any crash either.

So I still have no clue what happened here.

[malk0lm]

What is a compound command?

[malk0lm]

Let me try to get the other person who worked on this originally with me on this thread to add some more information.

Standby.

[tobegreat]

We were currently using all the mods list plus robot army. Upon returning the next day, the game would load as in map/save file and when it said loading map for peers it said this.

dam.PNG (140.47 KiB) Viewed 4570 times

I said dam thats funny why could that be? so I thought maybe because factiorio upgraded to 12.5 i said maybe its me so i checked everything on my end. I deleted everything I had related to factiorio and reinstalled it and downloaded the mods again. Thinking it was me, when I tried to run the game again it still failed. Something on the server side saw that when i tried to connect to it. It disconnected me from connecting to it.
We had old game save files and removed robot army mod thinking that was the root of it. The game would still run fine then crash, within 2 to 5 minutes. Thinking it was the 12.5 Beta upgraded version we went back to 12.3 or 12.4 the version everyone had.

[malk0lm]

tobe,

Not the most useful bug report, but thanks nevertheless

So the only other things I can think of are:

• Take the original save and give it to someone on windows to run locally. Have that person change all the belt out with underground belt then run it on a linux server...?
• Also, the save first ran under a different user so I had to chown factorio:factorio to get it to work properly.