User:Philosopher/Nasal FGCommand crashes

Just a page to gather information about the crashes we've been seeing recently. It seems to be related to running Nasal bindings, especially with joysticks.

Bug tracker

• ticket #1397
• ticket #1240

Forum threads

• Frequent constant random crashes
• Flying Seneca II at night
• Re: Stop crashing, fgfs.exe

Comments so far

I'm fairly certain this is joystick related and it is new I downgraded to 2.12.1 and it has not crashed yet with the js plugged in It has previously crashed on all newer versions including the latest git but only when the joystick is plugged in. — chrisb

To be honest, and as previously mentioned, I find it hard to believe that there really is a bug that's just showing up in one particular subsystem (like the JS bindings) - now, I have never experienced this, but I also don't happen to use a Joystick :D I think it's been 2 releases ago that the JS stuff was revamped by Stuart, so it's possible that there's something going on in the C++ code and the way it's interacting with the Nasal engine. But so far, my impression really is that the naHash related issues are due to aggressive compiler optimizations - we have previously seen issues exactly due to this, and due to Andy's sophisticated pointer hacks, which also did show up in hash.c - to see if this could be the case here or not, I'd suggest to come up with a minimal test case that reliably triggers a segfault (no scenery, no complex aircraft - anything else disabled). Once we have that, we can either disable all optimizations, or explicitly mark hash.c (or libnasal.a) to not use any optimizations at all - if that helps, we need to look at the generated IR/ASM code to see what's going on, or simply use volatile/compiler attributes to prevent the compiler from touching some of Andy's data structures/code. I would be very surprised if we had a bug in the Nasal engine that only shows up when using Joysticks - after all, Nasal doesn't know anything about Joysticks at all. So such an issue should show up in a lot of other places, and not just joystick bindings. Currently, I find it much more likely to be either due to compiler optimizations or due to the way the C++ code is interfacing with the corresponding ghosts. If anybody around can come up with a very compact test case that triggers the whole thing, I may get a joystick and give it a try to see if I can reproduce the whole thing. — Hooray

Current status

It looks like something happens when calling the function, which is compiled each time from the property tree - something that didn't happen in 2.12. The crash is reproducible after enough iterations, with any kind of (valid) binding code, and regardless of other conditions. It appears that some sort of invalid symbol appears on the stack, which does not exist as a valid address.

Available backtraces

Complete

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000020
0x0000000100d85b04 in naiHash_newsym (hash=0x1070c9cb0, sym=0x20, val=0x7fff5fbfd978) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/hash.c:257
257 struct naStr *s = PTR(*sym).str;
(gdb) bt
#0 0x0000000100d85b04 in naiHash_newsym (hash=0x1070c9cb0, sym=0x20, val=0x7fff5fbfd978) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/hash.c:257
#1 0x0000000100d799e3 in setupArgs (ctx=0x105898400, f=0x105898400, args=0x0, nargs=0) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/code.c:304
#2 0x0000000100d7954e in naCall (ctx=0x105898400, func={num = 2.1730606295780482e-314, ref = {ptr = {obj = 0x1062917e0, str = 0x1062917e0, vec = 0x1062917e0, hash = 0x1062917e0, code = 0x1062917e0, func = 0x1062917e0, ccode = 0x1062917e0, ghost = 0x1062917e0}, reftag = 2147444617}}, argc=0, args=0x0, obj={num = 0, ref = {ptr = {obj = 0x0, str = 0x0, vec = 0x0, hash = 0x0, code = 0x0, func = 0x0, ccode = 0x0, ghost = 0x0}, reftag = 2147444617}}, locals={num = 2.1804274803696552e-314, ref = {ptr = {obj = 0x1070c9cb0, str = 0x1070c9cb0, vec = 0x1070c9cb0, hash = 0x1070c9cb0, code = 0x1070c9cb0, func = 0x1070c9cb0, ccode = 0x1070c9cb0, ghost = 0x1070c9cb0}, reftag = 2147444617}}) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/code.c:1010
#3 0x0000000100d7cdb4 in naCallMethodCtx (ctx=0x105898400, code={num = 2.1730606295780482e-314, ref = {ptr = {obj = 0x1062917e0, str = 0x1062917e0, vec = 0x1062917e0, hash = 0x1062917e0, code = 0x1062917e0, func = 0x1062917e0, ccode = 0x1062917e0, ghost = 0x1062917e0}, reftag = 2147444617}}, self={num = 0, ref = {ptr = {obj = 0x0, str = 0x0, vec = 0x0, hash = 0x0, code = 0x0, func = 0x0, ccode = 0x0, ghost = 0x0}, reftag = 2147444617}}, argc=0, args=0x0, locals={num = 2.1804274803696552e-314, ref = {ptr = {obj = 0x1070c9cb0, str = 0x1070c9cb0, vec = 0x1070c9cb0, hash = 0x1070c9cb0, code = 0x1070c9cb0, func = 0x1070c9cb0, ccode = 0x1070c9cb0, ghost = 0x1070c9cb0}, reftag = 2147444617}}) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/code.c:1084
#4 0x0000000100d7cec7 in naCallMethod (code={num = 2.1730606295780482e-314, ref = {ptr = {obj = 0x1062917e0, str = 0x1062917e0, vec = 0x1062917e0, hash = 0x1062917e0, code = 0x1062917e0, func = 0x1062917e0, ccode = 0x1062917e0, ghost = 0x1062917e0}, reftag = 2147444617}}, self={num = 0, ref = {ptr = {obj = 0x0, str = 0x0, vec = 0x0, hash = 0x0, code = 0x0, func = 0x0, ccode = 0x0, ghost = 0x0}, reftag = 2147444617}}, argc=0, args=0x0, locals={num = 2.1804274803696552e-314, ref = {ptr = {obj = 0x1070c9cb0, str = 0x1070c9cb0, vec = 0x1070c9cb0, hash = 0x1070c9cb0, code = 0x1070c9cb0, func = 0x1070c9cb0, ccode = 0x1070c9cb0, ghost = 0x1070c9cb0}, reftag = 2147444617}}) at /Users/philosopher/Documents/FlightGear/simgear/simgear/nasal/code.c:1095
#5 0x00000001005360ba in FGNasalSys::callMethod (this=0x110aa73f0, code={num = 2.1730606295780482e-314, ref = {ptr = {obj = 0x1062917e0, str = 0x1062917e0, vec = 0x1062917e0, hash = 0x1062917e0, code = 0x1062917e0, func = 0x1062917e0, ccode = 0x1062917e0, ghost = 0x1062917e0}, reftag = 2147444617}}, self={num = 0, ref = {ptr = {obj = 0x0, str = 0x0, vec = 0x0, hash = 0x0, code = 0x0, func = 0x0, ccode = 0x0, ghost = 0x0}, reftag = 2147444617}}, argc=0, args=0x0, locals={num = 2.1804274803696552e-314, ref = {ptr = {obj = 0x1070c9cb0, str = 0x1070c9cb0, vec = 0x1070c9cb0, hash = 0x1070c9cb0, code = 0x1070c9cb0, func = 0x1070c9cb0, ccode = 0x1070c9cb0, ghost = 0x1070c9cb0}, reftag = 2147444617}}) at /Users/philosopher/Documents/FlightGear/flightgear/src/Scripting/NasalSys.cxx:248
#6 0x0000000100536012 in FGNasalSys::call (this=0x110aa73f0, code={num = 2.1730606295780482e-314, ref = {ptr = {obj = 0x1062917e0, str = 0x1062917e0, vec = 0x1062917e0, hash = 0x1062917e0, code = 0x1062917e0, func = 0x1062917e0, ccode = 0x1062917e0, ghost = 0x1062917e0}, reftag = 2147444617}}, argc=0, args=0x0, locals={num = 2.1804274803696552e-314, ref = {ptr = {obj = 0x1070c9cb0, str = 0x1070c9cb0, vec = 0x1070c9cb0, hash = 0x1070c9cb0, code = 0x1070c9cb0, func = 0x1070c9cb0, ccode = 0x1070c9cb0, ghost = 0x1070c9cb0}, reftag = 2147444617}}) at /Users/philosopher/Documents/FlightGear/flightgear/src/Scripting/NasalSys.cxx:236
#7 0x000000010053a1c4 in FGNasalSys::handleCommand (this=0x110aa73f0, moduleName=0x10f7ee430 "__js0", fileName=0x149a93648 "/input/joysticks/js/axis/binding", src=0x115a5c9e0 "aileron.push()", arg=0x115a84860) at /Users/philosopher/Documents/FlightGear/flightgear/src/Scripting/NasalSys.cxx:1231
#8 0x000000010053a2a8 in FGNasalSys::handleCommand (this=0x110aa73f0, arg=0x115a84860) at /Users/philosopher/Documents/FlightGear/flightgear/src/Scripting/NasalSys.cxx:1241
#9 0x0000000100008949 in do_nasal (arg=0x115a84860) at /Users/philosopher/Documents/FlightGear/flightgear/src/Main/fg_commands.cxx:183
#10 0x0000000100011fde in SGCommandMgr::FunctionCommand::operator() (this=0x1069538f0, arg=0x115a84860) at commands.hxx:57
#11 0x0000000100dd0145 in SGBinding::innerFire (this=0x1305be9c0) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/SGBinding.cxx:92
#12 0x0000000100dd07b5 in SGBinding::fire (this=0x1305be9c0, setting=0.08504398912191391) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/SGBinding.cxx:133
#13 0x00000001002b02e8 in FGJoystickInput::updateJoystick (this=0x115adc4b0, index=0, joy=0x115adc4e0, dt=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/flightgear/src/Input/FGJoystickInput.cxx:347
#14 0x00000001002b08aa in FGJoystickInput::update (this=0x115adc4b0, dt=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/flightgear/src/Input/FGJoystickInput.cxx:386
#15 0x0000000100e1c5b7 in SGSubsystemGroup::Member::update (this=0x124af2600, delta_time_sec=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/subsystem_mgr.cxx:412
#16 0x0000000100e1b77a in SGSubsystemGroup::update (this=0x124a58790, delta_time_sec=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/subsystem_mgr.cxx:249
#17 0x0000000100e1c5b7 in SGSubsystemGroup::Member::update (this=0x124af26a0, delta_time_sec=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/subsystem_mgr.cxx:412
#18 0x0000000100e1b77a in SGSubsystemGroup::update (this=0x106822760, delta_time_sec=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/subsystem_mgr.cxx:249
#19 0x0000000100e1cfbb in SGSubsystemMgr::update (this=0x106822670, delta_time_sec=0.041666666666666664) at /Users/philosopher/Documents/FlightGear/simgear/simgear/structure/subsystem_mgr.cxx:515
#20 0x000000010004a184 in fgMainLoop () at /Users/philosopher/Documents/FlightGear/flightgear/src/Main/main.cxx:108
#21 0x000000010073e9d3 in fgOSMainLoop () at /Users/philosopher/Documents/FlightGear/flightgear/src/Viewer/fg_os_osgviewer.cxx:333
#22 0x0000000100049288 in fgMainInit (argc=3, argv=0x7fff5fbffa50) at /Users/philosopher/Documents/FlightGear/flightgear/src/Main/main.cxx:493
#23 0x0000000100007a3c in main (argc=3, argv=0x7fff5fbffa50) at /Users/philosopher/Documents/FlightGear/flightgear/src/Main/bootstrap.cxx:234

Incomplete

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0x0000000000000020
0x000000010091d821 in naiHash_newsym ()
(gdb) bt
#0  0x000000010091d821 in naiHash_newsym ()
#1  0x0000000100912b79 in setupArgs ()
#2  0x000000010091283f in naCall ()
#3  0x0000000100915664 in naCallMethodCtx ()
#4  0x0000000100915739 in naCallMethod ()
#5  0x000000010038ca0b in FGNasalSys::handleCommand ()
#6  0x000000010038ca93 in FGNasalSys::handleCommand ()
#7  0x00000001000086fc in do_nasal ()
#8  0x00000001000119fd in SGCommandMgr::FunctionCommand::operator() ()
#9  0x000000010094e48b in SGBinding::innerFire ()
#10 0x00000001001af7d7 in FGJoystickInput::updateJoystick ()
#11 0x00000001001afb19 in FGJoystickInput::update ()
#12 0x0000000100995b0e in SGSubsystemGroup::Member::update ()
#13 0x0000000100994c7f in SGSubsystemGroup::update ()
#14 0x0000000100995b0e in SGSubsystemGroup::Member::update ()
#15 0x0000000100994c7f in SGSubsystemGroup::update ()
#16 0x0000000100996807 in SGSubsystemMgr::update ()
#17 0x000000010003b326 in fgMainLoop ()
#18 0x00000001004d5098 in fgOSMainLoop ()
#19 0x000000010003ad2c in fgMainInit ()
#20 0x000000010000760c in main ()

    Core was generated by `/home/flight/FG/install/fgfs/bin/fgfs --fg-root=/home/flight/FG/install/fgfs/fg'.
    Program terminated with signal 11, Segmentation fault.
    #0  0x08bd8372 in naiHash_newsym ()
    (gdb) bt
    #0  0x08bd8372 in naiHash_newsym ()
    #1  0x08bcfbf9 in setupArgs.isra.3 ()
    #2  0x08bd37f9 in naCall ()
    #3  0x08bd3b91 in naCallMethodCtx ()
    #4  0x08bd3c8c in naCallMethod ()
    #5  0x086658d1 in callMethod (locals=..., self=..., code=..., args=0x0, argc=0,
        this=<optimized out>) at /home/flight/FG/fgfs/flightgear/src/Scripting/NasalSys.cxx:247
    #6  call (locals=..., args=0x0, argc=0, code=..., this=0xf692028)
        at /home/flight/FG/fgfs/flightgear/src/Scripting/NasalSys.cxx:235
    #7  FGNasalSys::handleCommand (this=0xf692028, moduleName=0xae98ef8 "__js0",
        fileName=0x154241dc "/input/joysticks/js/button[7]/binding",
        src=0xaf20978 "controls.adjPropeller(-0.25)", arg=0xae98df0)
        at /home/flight/FG/fgfs/flightgear/src/Scripting/NasalSys.cxx:1152
    #8  0x08660e53 in FGNasalSys::handleCommand (this=0xf692028, arg=0xae98df0)
        at /home/flight/FG/fgfs/flightgear/src/Scripting/NasalSys.cxx:1164
    #9  0x082799a8 in do_nasal (arg=0xae98df0)
        at /home/flight/FG/fgfs/flightgear/src/Main/fg_commands.cxx:183
    #10 0x08c09093 in SGBinding::innerFire() const ()
    #11 0x084385d6 in FGButton::update (this=0x10d37f48, modifiers=0, pressed=true, x=-1, y=-1)
        at /home/flight/FG/fgfs/flightgear/src/Input/FGButton.cxx:65
    #12 0x0844897e in FGJoystickInput::updateJoystick (this=0xae7b470, index=0, joy=0xae7b48c,
        dt=0.016666666666666666)
        at /home/flight/FG/fgfs/flightgear/src/Input/FGJoystickInput.cxx:360
    #13 0x08448acb in FGJoystickInput::update (this=0xae7b470, dt=0.016666666666666666)
        at /home/flight/FG/fgfs/flightgear/src/Input/FGJoystickInput.cxx:370
    #14 0x08c32039 in SGSubsystemGroup::Member::update(double) ()
    #15 0x08c322db in SGSubsystemGroup::update(double) ()
    #16 0x08c32039 in SGSubsystemGroup::Member::update(double) ()
    #17 0x08c322db in SGSubsystemGroup::update(double) ()
    #18 0x08c31e82 in SGSubsystemMgr::update(double) ()
    #19 0x0829f3ea in fgMainLoop () at /home/flight/FG/fgfs/flightgear/src/Main/main.cxx:108
    #20 0x087a52d9 in fgOSMainLoop ()
        at /home/flight/FG/fgfs/flightgear/src/Viewer/fg_os_osgviewer.cxx:333
    #21 0x0829ff97 in fgMainInit (argc=15, argv=0xbff7c144)
        at /home/flight/FG/fgfs/flightgear/src/Main/main.cxx:493
    #22 0x08251ce6 in main (argc=<error reading variable: Cannot access memory at address 0x0>,
        argv=<error reading variable: Cannot access memory at address 0x4>)
        at /home/flight/FG/fgfs/flightgear/src/Main/bootstrap.cxx:278