Howto:Reset/re-init Troubleshooting: Difference between revisions

m
→‎Nasal dialog: fix the fps/latency counters, which should really be using sprintf()
m (→‎Nasal dialog: fix the fps/latency counters, which should really be using sprintf())
 
(32 intermediate revisions by 2 users not shown)
Line 7: Line 7:
{{Portability Navbar}}
{{Portability Navbar}}
== Motivation ==
== Motivation ==
{{See also|Howto:Disable Nasal entirely}}
{{FGCquote
|1= The startup sequence has always proved to be rather tricky to get right
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/13175699/
  | title  = <nowiki>Re: [Flightgear-devel] Subsystem run-levels</nowiki>
  | author = <nowiki>Erik Hofman</nowiki>
  | date  = Apr 18th, 2006
  | added  = Apr 18th, 2006
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= the property tree as it is currently is in need of some rework because of the ownship (single desktop aircraft) approach. This is easier than it sounds - basically most of the property tree becomes part of the aircraft and only a few items are shared. This will also allow the switching of aircraft. The reason to consider this now, and maybe not implement it, is to ensure that the design will support this when it is time to implement it.
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34631875/
  | title  = <nowiki>Re: [Flightgear-devel] HLA developments</nowiki>
  | author = <nowiki>Richard Harrison</nowiki>
  | date  = Nov 19th, 2015
  | added  = Nov 19th, 2015
  | script_version = 0.23
  }}
}}
{{FGCquote
|1= Currently we don’t take good advantage of multi-core CPUs [...]
Defining a multi-threaded client-server architecture from the beginning would make a big difference today[...]
We’re also planning to use HLA/RTI, which will allow FlightGear to integrate with distributed simulation environments (important for industrial applications) and make better use of multi-core processors.
|2= {{cite web
  | url    = https://sourceforge.net/blog/november-2015-community-choice-project-of-the-month-flightgear/
  | title  = <nowiki>November 2015, “Community Choice” Project of the Month – FlightGear</nowiki>
  | author = <nowiki>Curtis Olson</nowiki>
  | date  = Nov, 2015
  }}
}}


{{FGCquote
{{FGCquote
Line 16: Line 52:
   | date  = Mar 25th, 2015
   | date  = Mar 25th, 2015
   }}
   }}
}}<references/>
}}
 
{{FGCquote
|1= Modularlisation, isolation, and standardisation of the FG codebase is very much WIP.  This is being pushed hard from 2 directions - James with the reinit() and Stuart with HLA.  Note my CppUnit comment on the mailing list - if this takes off, then this process will be massively accelerated.
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=273833#p273833
  | title  = <nowiki>Re: FGPython an propose for Python as an nasal alternative</nowiki>
  | author = <nowiki>bugman</nowiki>
  | date  = Jan 23rd, 2016
  | added  = Jan 23rd, 2016
  | script_version = 0.23
  }}
}}
 
 
{{FGCquote
|1= Instead of adding just-another-feature we need to strip it down to getting a fast and constand fps rendering engine. Everything else needs to run outside the main loop and has to interact with the core by, say HLA/RTI or whatever IPC we have.
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34793778/
  | title  = <nowiki>Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem.</nowiki>
  | author = <nowiki>Torsten Dreyer</nowiki>
  | date  = Jan 25th, 2016
  | added  = Jan 25th, 2016
  | script_version = 0.25
  }}
}}
 
{{FGCquote
|1= I'd just recommend to not do it but to focus on detangling the subsystems and creating a distributes simulation instead. That's my vision.
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34793948/
  | title  = <nowiki>Re: [Flightgear-devel] A FGPythonSys implementation: The embedded
Python interpreter as a FlightGear subsystem.</nowiki>
  | author = <nowiki>Torsten Dreyer</nowiki>
  | date  = Jan 25th, 2016
  | added  = Jan 25th, 2016
  | script_version = 0.25
  }}
}}
 
<references/>
 
== Objective ==
== Objective ==
[[File:Reset-reinit-control-panel.png|thumb|reset/reinit control panel for regression testing purposes implemented using Nasal & Canvas]]
[[File:Reset-reinit-control-panel.png|thumb|reset/reinit control panel for regression testing purposes implemented using Nasal & Canvas]]


FlightGear does not currently support saving/loading flights or reliably switching between aircraft at run-time (this is extensively discussed at [[FlightGear Sessions]]). [[Reset & re-init]] is an effort to refactor the FlightGear initialization process so that resetting and repositioning (switching aircraft) is supported, without having to exit or restart FlightGear. Currently, this is exposed via the Canvas-based [[Aircraft Center]], but is considered broken or fragile at best by most core developers.
FlightGear does not currently support saving/loading flights or reliably switching between aircraft at run-time (this is extensively discussed at [[FlightGear Sessions]]).  
 
[[Reset & re-init]] is an effort to refactor the FlightGear initialization process so that resetting and repositioning (switching aircraft) is supported, without having to exit or restart FlightGear. Currently, this is exposed via the Canvas-based [[Aircraft Center]], but is considered broken or fragile at best by most core developers.
 
The core developers are aiming to find out the dependencies of different subsystems<ref>http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg18066.html</ref>, and re-factor them so that more and more subsystems can be made optional (analogous to [[FlightGear Run Levels|run-levels]])<ref>http://forum.flightgear.org/</ref>, enabling them to be dynamically removed and re-added at run-time.


The core developers are aiming to find out the dependencies of different subsystems<ref>http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg18066.html</ref>, and re-factor them so that more and more subsystems can be made optional (analogous to [[FlightGear Run Levels|run-levels]])<ref>http://forum.flightgear.org/</ref>, enabling them to be dynamically removed and re-added at run-time. This will be particularly important to untangle implicit or hard-coded dependencies among the different subsystems, and will be one of the key tasks to move certain subsystems into dedicated [[High-Level Architecture]] (HLA) {{wikipedia|High-level architecture#Common HLA terminology|federates|noicon=1}}.
This will be particularly important to untangle implicit or hard-coded dependencies among the different subsystems<ref>http://sourceforge.net/p/flightgear/mailman/message/34793948/</ref>, and will be one of the key tasks to move certain subsystems into dedicated [[High-Level Architecture]] (HLA) {{wikipedia|High-level architecture#Common HLA terminology|federates|noicon=1}}.


One of the long-term goals is to provide a so called "[[FlightGear Headless|headless]]" mode so that certain features/subsystems (unrelated to graphics) can be better tested in isolation. An example would running FlightGear in an automated fashion on the [[FlightGear Build Server]], which could help increasingly automate the release process and related regression testing<ref>http://sourceforge.net/p/flightgear/mailman/message/34741767/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34627074/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34657550/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34701971/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34657898/</ref>.
One of the long-term goals is to provide a so called "[[FlightGear Headless|headless]]" mode so that certain features/subsystems (unrelated to graphics) can be better tested in isolation. An example would running FlightGear in an automated fashion on the [[FlightGear Build Server]], which could help increasingly automate the release process and related regression testing, which is another stated goals of several core developers<ref>http://sourceforge.net/p/flightgear/mailman/message/34741767/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34627074/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34657550/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34701971/</ref><ref>http://sourceforge.net/p/flightgear/mailman/message/34657898/</ref>.


The other goal is to increasingly modularize FlightGear by using HLA <ref>http://sourceforge.net/p/flightgear/mailman/message/34694887/</ref>and split off the simulation loops (see also [[FGViewer]]), as well as [[Supporting multiple renderers|supporting different renderers]] (such as [[Rembrandt]] and [[ALS]]), scenery engines (e.g., standard and [[osgEarth]]) and weather engines, in a fashion similar to how FlightGear already supports different FDM engines ([[JSBSim]] and [[YASim]]). HLA will make it possible for certain subsystems to be moved to dedicated cores by using separate threads or even processes, meaning that certain subsystems may even be able to be on a different computer, in a distributed setup.
The other goal is to increasingly modularize FlightGear by using HLA <ref>http://sourceforge.net/p/flightgear/mailman/message/34694887/</ref>and split off the simulation loops (see also [[FGViewer]]), as well as [[Supporting multiple renderers|supporting different renderers]] (such as [[Rembrandt]] and [[ALS]]), scenery engines (e.g., standard and [[osgEarth]]) and weather engines, in a fashion similar to how FlightGear already supports different FDM engines ([[JSBSim]] and [[YASim]]). HLA will make it possible for certain subsystems to be moved to dedicated cores by using separate threads or even processes, meaning that certain subsystems may even be able to be on a different computer, in a distributed setup.
Line 37: Line 119:
People running the code shown below should be prepared to trigger segfaults, and should ideally be able to provide gdb backtraces (if you are on Windows, please send [[CrashRpt|crash reports]].
People running the code shown below should be prepared to trigger segfaults, and should ideally be able to provide gdb backtraces (if you are on Windows, please send [[CrashRpt|crash reports]].


For testing purposes, it does make sense to run FlightGear using the minimal startup profile, with graphics/rendering disabled using [[Draw masks]].
For testing purposes, it does make sense to run FlightGear using the [[Minimal Startup Profile|minimal startup profile]], with graphics/rendering disabled using [[Draw masks]].


<references/>
<references/>


== Background ==
== Background ==
James has added initial code to work on dynamic subsystem creation, so that subsystems can be added or removed. Only some subsystems are supported so far, since many have non-default constructors or other complexities (see [[FlightGear Run Levels]] for further details). With this change, it's now possible to dynamically add and remove the traffic-manager at runtime using fgcommands, for example (pasted into the [[Nasal Console]]):
James has added initial code to work on dynamic subsystem creation, so that subsystems can be added or removed at runtime.  
 
Only some subsystems are supported so far, since many have non-default C++ constructors (e.g. systems designed as singletons) or other complexities (see [[FlightGear Run Levels]] for further details).  
 
With this change, it's now possible to dynamically add and remove the traffic-manager at runtime using fgcommands, for example (pasted into the [[Nasal Console]]):
<syntaxhighlight lang="nasal">
<syntaxhighlight lang="nasal">
fgcommand("add-subsystem", props.Node.new({
fgcommand("add-subsystem", props.Node.new({
Line 54: Line 140:


In addition, it will be much easier for developers to do regression testing and benchmarking once subsystems can be completely disabled, a long-standing feature request (see [[FlightGear Headless]]); it would also help to simplify, and to some extent even automate, release preparations.
In addition, it will be much easier for developers to do regression testing and benchmarking once subsystems can be completely disabled, a long-standing feature request (see [[FlightGear Headless]]); it would also help to simplify, and to some extent even automate, release preparations.
{{FGCquote
|1= thanks to all the reset/re-init work, the C++ dependencies are fairly straightforward to decouple these days, I just made the tile manager optional in under 10 minutes, i.e. all the scenery/sky/stars is gone now - which means that FG is now utiliing less than 140 MB of RAM here using this "startup mode".
So this about more than just "FGCanvas": It will make troubleshooting so much more straightforward, and we can also explore feature-scaling and run-time benchmarking once this works.
Currently, I'd even consider restructuring initialization to always boot up like this, because it's the safest possible subset of subsystems for now - i.e. roughly a dozen subsystems that basically stand no chance of "crashing" the sim. And from then on, additional initialization could be handled by Nasal obviously - and scripts could even scale up/down, depending on hardware support and features.
With 14 subsystems remaining, I am now getting 650 fps when showing the Canvas GUI (Aircraft Center) here.
As for C++ changes, I have come to the conclusion that this could be greatly simplified by making the whole aircraft shebang optional - i.e. aircraft as such, and all its related subsystems, e.g.:
*  view manager
*  FDM (obviously)
*  Autopilot, Property Rules
*  Route Manager
*  history
*  flight recorder
These are currently hard-coded as separate SGSubsystem instances in fg_init.cxx - even though this is conceptually a headache, simply because these should be all handled by a single SGSubsystemGroup and wrapped inside it.
|2= {{cite web
  | url    = http://forum.flightgear.org/viewtopic.php?p=214211#p214211
  | title  = <nowiki>Re: FGCanvas Experiments & Updates</nowiki>
  | author = <nowiki>Hooray</nowiki>
  | date  = Jul 6th, 2014
  | added  = Jul 6th, 2014
  | script_version = 0.23
  }}
}}


== Fgcommands ==
== Fgcommands ==
Line 212: Line 324:


== Nasal dialog ==
== Nasal dialog ==
{{Note|The following Nasal script can be executed via the Nasal Console or put in a separate file and executed via a menu item to easily test different aspects of reset/re-init. For the time being, you are likely to trigger segfaults/crashes or other undefined behavior (e.g., memory leaks), so it is recommended to run FlightGear in a GNU Debugger (GDB) session to obtain a backtrace. If you manage to cause a bug or crash, please file a bug report: {{Tickets}} }}
[[File:Updated-reset-reinit-control-panel-2016.2.png|thumb|updated screenshot showing the reset/re-init control panel for helping troubleshoot reset/re-init related segfaults]]


<syntaxhighlight lang="nasal" enclose="div">
{{Note|The following Nasal script can be executed via the Nasal Console or put in a separate file and executed via a menu item to easily test different aspects of reset/re-init. This is intended to be a stress test, by allowing people to easily switch aircraft, relocate repeatedly and/or stop/restart and suspend/resume different subsystems. For the time being, you are likely to trigger segfaults/crashes or other undefined behavior (e.g., memory leaks), so it is recommended to run FlightGear in a GNU Debugger (GDB) session to obtain a backtrace. If you manage to cause a bug or crash, please file a bug report: {{Tickets}} }}


canvas.MessageBox.warning(
<syntaxhighlight lang="javascript">canvas.MessageBox.warning(
     "Developer Feature",
     "Developer Feature",
     "This dialog is mainly intended for people familiar with FlightGear/core internals to help troubleshoot reset/re-init related bugs. You will probably want to run FlightGear inside a GNU Debugger (GDB) session when using this dialog",
     "This dialog is mainly intended for people familiar with FlightGear/core internals to help troubleshoot reset/re-init related bugs. You will probably want to run FlightGear inside a GNU Debugger (GDB) session when using this dialog",
Line 224: Line 336:
###################################################################
###################################################################
var (width, height) = (700, 480);
var (width, height) = (700, 480);
var title = 'Reset/re-init Panel';
# https://sourceforge.net/p/flightgear/mailman/message/35048478/
var title = 'Reset/re-init Panel (aka Segfault Paradise)';
   
   
var window = canvas.Window.new([width, height], "dialog")
var window = canvas.Window.new([width, height], "dialog")
Line 324: Line 437:
     row.addItem(label);
     row.addItem(label);


     foreach(var cmd; ["re-init", "remove-subsystem", "add-subsystem"]) {  
    var status = canvas.gui.widgets.Label.new(root, canvas.style, {wordWrap: 0});
    status.setText("nop");
    row.addItem(status);
 
    test.status =  status;
 
 
# this adds a row of buttons for the 3 currently supported fgcommands
# (support for suspend/resume is pending)
     foreach(var cmd; ["reinit", "remove-subsystem", "add-subsystem"]) {  
         var button = canvas.gui.widgets.Button.new(root, canvas.style, {})
         var button = canvas.gui.widgets.Button.new(root, canvas.style, {})
             .setText(cmd)
             .setText(cmd)
Line 332: Line 454:
     }
     }


     return label; # we want to update the label elsewhere
     return status; # we want to update the label elsewhere
}; # addTest
}; # addTest
   
   
##
##
# vector with hashes containing subsystems that will be added as buttons to the dialog
# vector with hashes containing subsystems that will be added as buttons to the dialog
# NOTE: this must match $FG_SRC/Main/subsystemFactory.cxx
# each hash will end up with an additional "label" field that is updated separately
# TODO: it would make sense to add a dedicated list-subsystems to get a list of all supported systems !
# Also, the hash can be extended to add tooltips and/or other meta information that can be
var Tests = [
# used for dealing with subsystems differently (think Nasal depending on events, sound/tilemgr being threaded etc).
    { name: "sound" },
#
    { name: "prop-interpolator" },
# this must match $FG_SRC/Main/subsystemFactory.cxx, so you need to extend this
    { name: "properties" },
# code if you are using this in conjunction with a topic branch like the FGPythonSys branch
    { name: "http" },
##
    { name: "flight" },
# NOTE: in the current form of the script, this is a vector with hashes
    { name: "environment" },
# that will be populated by briefly running the performance monitor
    { name: "ephemeris" },
# with a name field containing the name of the subsystem
    { name: "systems" },
# and a status field pointing to the Canvas text node for showing active/inactive flags
    { name: "instrumentation" },
var Tests = [];
    { name: "hud" },
 
    { name: "cockpit-displays" },
var performance_monitor = props.globals.getNode('/sim/performance-monitor');
    { name: "route-manager" },
 
    { name: "io" },
var subsystemListTimer = maketimer(1.5, func(){
    { name: "logger" },
     foreach(var s; performance_monitor.getNode("subsystems").getChildren('subsystem') ) {
     { name: "gui" },
     var name = s.getNode('name').getValue();
    { name: "Canvas" },
     print("Subsystem found:", name);
     { name: "CanvasGUI" },
     append(Tests, {name: name});
    { name: "ATC" },
}
     { name: "mp" },
 
    { name: "ai-model" },
print("subsystem list retrieved, disabling performance monitor again");
    { name: "submodel-mgr" },
performance_monitor.getNode('enabled').setBoolValue(0); # disable the performance monitor again
     { name: "traffic-manager" },
 
    { name: "controls" },
    { name: "http" },
    { name: "input" },
    { name: "replay" },
    { name: "history" },
    { name: "voice" },
    { name: "fgcom" },
    { name: "lighting" },
    { name: "aircraft-model" },
    { name: "model-manager" },
    { name: "view-manager" }
]; # vector with tests
   
   
debug.benchmark("button setup", func() {
debug.benchmark("button setup", func() {
Line 382: Line 492:
     # will add label fields to each test, so that labels can be dynamically updated
     # will add label fields to each test, so that labels can be dynamically updated
     # using a different callback
     # using a different callback
     test.label = addTest(root: scrollContent, layout: list, test: test);
     addTest(root: scrollContent, layout: list, test: test);
}
}
}); # button setup (benchmark)
}); # button setup (benchmark)
Line 389: Line 499:


var subsystemMonitor = func(){
var subsystemMonitor = func(){
    # update labels showing frame rate & frame spacing
     foreach(var d; dynamicLabels) {
     foreach(var d; dynamicLabels) {
    # FIXME: dynamic labels should probably be using sprintf() style format strings
       d.label.setText( d.cb() );
       d.label.setText( d.cb() );
     }  
     }  
Line 397: Line 510:
         var suffix = isRunning ? " (active)" : " (inactive)";  
         var suffix = isRunning ? " (active)" : " (inactive)";  
         #test.label.setBackground(color);
         #test.label.setBackground(color);
         var currentText = test.label._view._text.get("text");
         var currentText = test.status._view._text.get("text");
         var neededLabel = test.name ~ suffix;
         var neededLabel = suffix;
#FIXME: better use show()/hide() methods here which is more efficient than changing labels per frame ...
#FIXME: better use show()/hide() methods here which is more efficient than changing labels per frame ...
         if (currentText == neededLabel) continue;
         if (currentText == neededLabel) continue;
         test.label.setText(neededLabel);
         test.status.setText(neededLabel);
     }
     }
}
}
Line 417: Line 530:
fps.setText("45 fps");
fps.setText("45 fps");
statusbar.addItem(fps);
statusbar.addItem(fps);
append(dynamicLabels, {label:fps, cb:func getprop("/sim/rendering/frame-rate") });
append(dynamicLabels, {label:fps, cb:func() {return ""~ getprop("/sim/frame-rate");}, suffix:' fps' });


var ms=canvas.gui.widgets.Label.new(root, canvas.style, {wordWrap: 0});
var ms=canvas.gui.widgets.Label.new(root, canvas.style, {wordWrap: 0});
ms.setText("35 ms");
ms.setText("35 ms");
statusbar.addItem(ms);
statusbar.addItem(ms);
append(dynamicLabels, {label:ms, cb:func getprop("/sim/rendering/frame-latency") });
append(dynamicLabels, {label:ms, cb:func() {return ""~ getprop("/sim/frame-latency-max-ms");}, suffix:' ms' });




Line 433: Line 546:
     call(canvas.Window.del, [], me);
     call(canvas.Window.del, [], me);
};
};
}); # end of initial/setup timer
performance_monitor.getNode('enabled').setBoolValue(1);
subsystemListTimer.singleShot = 1; # timer will only be run once
gui.popupTip("Getting list of known subsystems from the performance monitor ...",1.5);
subsystemListTimer.start();


###################################################################
###################################################################
Line 438: Line 560:
     canvas.MessageBox.Ok |canvas.MessageBox.Cancel | canvas.MessageBox.DontShowAgain
     canvas.MessageBox.Ok |canvas.MessageBox.Cancel | canvas.MessageBox.DontShowAgain
);
);
</syntaxhighlight>
</syntaxhighlight>