Aircraft rating system: Difference between revisions

Replacing with rating system discussed on mailing list and implemented for some aircraft.
m (Robot: Cosmetic changes)
(Replacing with rating system discussed on mailing list and implemented for some aircraft.)
Line 2: Line 2:


=== Intro ===
=== Intro ===
Based on various mailing list discussions since 11/2007, it is getting clear that the current method of classifying aircraft development status is not satisfactory to adequately and fully describe an aircraft's development progress. Which can be pretty frustrating for developers, contributors and users, because of different expectations about an aircraft's "status".
With hundreds of aircraft available for FG, it is very difficult to find the true gems. The rating system below attempts to solve this by providing a quick objective way to rate aircraft. This was discussed in some detail on the -dev mailing list in December 2010.


This can be even witnessed in more recent discussions (03/2009), where users explicitly inquire about recommendations for feature-complete aircraft [http://www.flightgear.org/forums/viewtopic.php?f=2&t=3171], or users pointing out weaknesses in FDM configurations [http://flightgear.org/forums/viewtopic.php?f=4&t=3281].
The rating system scores an aircraft (0-5) in 4 individual criteria - FDM, Systems, Cockpit and External Model.  


This is mainly because of the fact that different people may have different requirements when it comes to aircraft status and are thus viewing the status from different angles, possibly neglecting progress in other areas.
These ratings can then be encoded in the aircraft -set.xml file from where they can be picked up by launchers, web pages etc.


In addition, users are frequently facing issues when it comes to determining compatibility requirements between aircraft tarballs and fgfs binary releases (see for example [http://flightgear.org/forums/viewtopic.php?f=6&t=2814]). This very issue is further complicated by the fact, that FlightGear in general doesn't expose any detailed version information that could be queried by arbitrary FlightGear resources (such as aircraft). Furthermore, it is questionable whether simply starting to expose version information in the property tree would be sufficient, i.e. for scenarios where individual features may be removed/added or modified in CVS/HEAD, which however wouldn't necessarily result in a modified version number, so that such modifications wouldn't be easily communicated to FlightGear resources that depend on such features and information (for a more in depth discussion of this particular issue, please see [http://wiki.flightgear.org/index.php/Simplifying_Aircraft_Deployment#Version_tracking.2FCompatibility_checking]).
The sum of the scores in the 4 criteria provides an overall score from 0 to 20, which can be mapped to a textual overall aircraft status.  


=== Scope ===
=== Flight Dynamics Model ===
This page is dedicated to providing a place to start collecting suggestions about how to describe aircraft development status more formally and reliably, to point out the strengths and weaknesses of individual aircraft implementations more comprehensively.


Given that people seem to be expecting different things when they read about an aircraft's development status being "alpha", "beta" or "production", it should be possible to address this issue, by starting to classify each aircraft's individual components (i.e. fdm, cockpit panel, 3d model etc) directly, rather than the overall aircraft itself which would result in less-developed parts of an aircraft (which can usually be assumed to be most of it) affecting an aircraft's development status negatively, even if it should feature some very well-developed parts.
*0: None, or using FDM from other aircraft
*1: JSBSim Aeromatic or YASim geometric model used without tuning. Flaps modeled.
*2: FDM tuned for cruise configuration.
*3: FDM tuned for rate of climb and cruise PoH performance numbers
*4: FDM matches PoH in 90% of configurations
*5: FDM matches PoH and most known test data.


=== Suggestions ===
=== Systems ===
This classification process should preferably take place by following certain conventions, so that it is likely that the majority of aircraft types, as well as their various features, are covered thoroughly.


This could for example be achieved by starting to evaluate aircraft based on abstract factors such as the following:
*0: No controllable systems: engine is always on, generic radio,
*1: Generic engine start/stop (}}s), correct size/number of fuel tanks,  generic (untuned) autopilot, working flaps/gear
*2: Working electrical system, fuel feed cockpit controls, stable autopilot
*3: Accurate startup procedure, tuned autopilot with cockpit controls matching real aircraft systems, generic failure modelling (Vne, +ve/-ve G, gear limits)
*4: Primary aircraft-specific systems modelled (aero-tow, radar, GPWS). User able to follow normal PoH checklists (e.g. startup, shutdown) in entirety
*5: Some aircraft-specific failure modes implemented (e.g. flame-out, inverted engine limitations). Some emergency procedures implemented (RAT, emergency gear release), able to follow some emergency PoH checklists in entirety.


* status/realism of FDM configs (performance characteristics)
=== Rating the Cockpit ===
* status/realism of exterior 3D models
* status/realism of cockpit panels (2D/3D)
* status/realism of autopilot implementation
* status/realism of aircraft-specific instruments
* status/realism of procedure modeling (i.e. startup)
* status/realism of systems modeling
* status/realism of failure modeling
* status/realism of integrated checklists/procedures
* status/realism of integrated help (i.e. in the form of help dialogs)
* status/realism of integrated scripted tutorials
* status/realism of available documentation
* status/realism of overall usability


Furthermore, it might also be worthwhile to consider evaluating aircraft based on additional criteria such as:
*0: No cockpit
*1: 2D panel, no cockpit.
*2: 2D panel in 3D cockpit, or incomplete 3D panel
*3: 3D panel and cockpit
*4: 3D panel and accurately modelled 3D cockpit, plain texturing. Hotspots for majority of controls.
*5: 3D panel and accurately modelled 3D cockpit with photo-realistic texturing.


* available support for multiplayer
=== External Model ===
* available support for AI traffic (i.e. 3D models featuring LOD support may be reused by the AIT system)
* available support for runtime customization (i.e. liveries)
* available support for eye candy/effects
* available support for saving/loading and continuing flights
* available support for saving/loading aircraft configurations


In addition, it is important to keep in mind that there are different requirements for aircraft to appear "realistic" - depending on whether they are mainly used in VFR settings or IFR settings, where the former application scenario puts an increased demand on visual effects, the latter requires IFR-specific features to be in place, for example proper IFR instrumentation, as well as correct navigation lights.
*0: None
*1: Simple 3D model, no animations
*2: 3D model with animated control surfaces (elevator, aileron, rudder, flaps)
*3: 3D model with animated control surfaces, gear detailing (retraction), prop
*4: 3D model with animated control surfaces, gear, prop, livery support.
*5: 3D model with animated control surfaces, gear, prop, livery support, Ambient Occlusion effect.


Therefore, it might be a good idea to not only evaluate individual components of an aircraft, but also its readiness for certain usage scenarios, such as for example VFR/IFR flight.
=== Overall Status ===


=== Desired Results ===
Sum the ratings of the 4 criteria, to produce an overall score from 0 to 20.


It can be expected that following this or similarly factually objective approaches to classifying aircraft development status, should not only help people understand more easily and more completely an individual aircraft's development status, but also highlight the strengths and weaknesses of an aircraft, so that aircraft authors are motivated because of the work they have actually done, rather than being discouraged because of the work they could not (yet) do (or might possibly never be able to do because of it being outside their usual scope of contributions).
This maps to overall status as follows:


Additionally, this would also help aircraft becoming more self-documented, so that new contributors could more easily see where and how their help could be used.
*18 or higher : advanced production (minimum 4 in each rating)
*16 to 17 : production (minimum 4 in each rating)
*12 to 15 : early production (minimum 3 in each rating)
*9 to 11 : beta
*8 or lower : alpha


== Implementation ==
=== Encoding the rating ===


Implementing support for more thorough status-tracking should be fairly straight forward, as the current approach of keeping aircraft status information by using a property tree node in an aircraft's *-set.xml file, could be easily extended to provide additional meta information, of the aforementioned nature - using the previously suggested type of information, this could end up looking like the following:
The ratings can be encoded in the -set.xml file for the aircraft, as follows.
 
  <nowiki>
<PropertyList>
<status>early production</status>
  <aircraft-status>
<rating>
  <fdm>beta</fdm>
  <FDM type="int">5</FDM>
  <exterior>early-production</exterior>
  <systems type="int">4</systems>
  <cockpit>alpha</cockpit>
  <model type="int">3</model>
  <instrumentation>alpha</instrumentation>
  <cockpit type="int">3</cockpit>
  <procedures>alpha</procedures>
</rating>
  <system-modeling>beta</system-modeling>
  </nowiki>
  <failure-modeling>alpha<failure-modeling>
  <checklist-support>alpha</checklist-support>
  <tutorial-support>alpha</tutorial-support>
  <documentation>alpha</documentation>
  <usability>beta</usability> 
  <eye-candy>alpha</eye-candy>
  </aircraft-status>
</PropertyList>
 
 
On the other hand, some of this information should probably not directly go into the toplevel *-set.xml file, but rather into files of the various components (i.e. the cockpit/panel status would be a good candidate to have its status being tracked directly in the panel file).
 
=== Extending status information ===
Given that not all conceivable status for the above mentioned criteria can be properly represented using the existing notion of "status" information being restricted to "alpha, beta, early-production...", it should be considered to support additional states, such as for example "none/unavailable", "draft" or "design".
 
While it would of course be much easier to simply provide a full-length description space of the status information for each individual entry, where each contributor could leave some information, this would make it more complicated to later on provide web-based frontends to facilitate searching/picking aircraft based on certain attributes.
 
On the other hand, it is not unlikely that even this level of more finely-grained status information should turn out insufficient in some cases, to describe the status properly, thus combining the two previously mentioned approaches of 1) providing formal status information and 2) providing an optional textual description of the progress would probably be the most powerful long-term solution.
 
This could for example be accomplished by allowing contributors to optionally specify a corresponding attribute to point to a plaintext (or possibly even HTML) file (residing in the aircraft's folder) that may contain further details, such an XML attribute, named "info-file" is shown in the following example:
 
  <PropertyList>
  <aircraft-status>
  <fdm '''info-file="fdm-status.txt"'''>beta</fdm>
  <exterior '''info-file="ext-status.txt"'''>early-production</exterior>
  <cockpit '''info-file="cockpit-status.txt"'''>alpha</cockpit>
  [...]
  <eye-candy '''info-file="effects-status.txt"'''>alpha</eye-candy>
  </aircraft-status>
</PropertyList>
 
These attributes could not only be easily parsed by an XML parser to provide web-based search facilities, they could even be used within FlightGear at runtime, to provide enriched status information by reading in the corresponding files and showing their contents in a dialog.
 
In the long run, it might even be an interesting option to provide optional support for three different attributes specifying filenames for these status-tags:
 
* bugs-filename=""
* fixme-filename=""
* todo-filename=""
 
That way, it would be possible for contributors to easily provide very detailed information about the development status of individual components or features, simply by dropping a corresponding file into the aircraft tarball, possibly using a standard-location/folder such as "meta-info" within the aircraft's folder.
 
Likewise, users noticing new issues could easily augment said files and send them to the maintainer to have them committed to HEAD, so that future users can first check out this information before possibly reporting a duplicate issue/feature request.
 
While it may seem unnecessary to provide this sort of information-collecting infrastructure to aircraft authors and their contributors, it would probably help aircraft development status become more self-documenting and aircraft-development itself become more self-contained, given that key information would be collected in one central place (the repository) and aircraft-specific issues/ideas could be easily reported and checked for by new contributors.
In the long run
 
=== A Flexible Approach ===
It is obvious that regardless of how many sensible rating-categories are now identified and used in FlightGear, people will sooner or later want to use modified or even completey new categories to rate the strengths and weaknesses of their aircraft.
 
Because of this, but also because hardcoding such tag names (i.e. property paths) would be unnecessarily tedious, given that FlightGear itself does not yet actively use this sort of information at the moment to provide aircraft picker/sorting facilities, one could simply by convention have ALL tags below one specific node (i.e. PropertyList/sim/aircraft-status/) -regardless- of their name be honored by potential search frontends.
 
=== Future Extendability ===
 
Because of the easy and flexibility of directly using XML/PropertyList files together with the FlightGear Property Tree, this approach to supporting more finely-grained status information tracking is extremely powerful and is promising enough to even support highly detailed status information for complex aircraft that may feature a number of distinct components, for example to appropriately describe various high performance aircraft or airliners, child nodes could be used:
 
<systems-modeling>
  <electrical></electrical>
  <hydraulic></hydraulic>
  <pneumatic></pneumatic> 
  ...
  </systems-modeling>
 
=== Fully searchable Aircraft - by Type ===
 
Furthermore, many users have frequently requested a possibility to search and pick aircraft based on certain aircraft-specific characteristics, such as for example the type of aircraft (i.e. single engine vs. multi engine, piston vs. jet etc.)
 
This could also be easily accomplished by having aircraft authors and contributors provide such meta information in the same way, for example:
 
<PropertyList>
  <!--ac-set.xml -->
  <type-of-aircraft include ="aircraft-type.xml"/>
</PropertyList>
 
 
NOTE: for the sake of illustration purposes, false values have
also been specified, even though they would be obsolete if all
all values are by default assumed to be false, which would
significantly reduce unnecessary overhead.
 
aircraft-type.xml:
  <PropertyList>
  <is-fixed-wing type="bool">true</is-fixed-wing>
  <is-helicopter type="bool">false</is-helicopter>
  <is-sailplane type="bool">false</is-sailplane>
  <is-amphibian  type="bool">false</is-amphibian>
 
  <is-military  type="bool">false</is-military>
  <is-civilian  type="bool">true</is-civilian>
 
  <is-airliner  type="bool">true</is-airliner>
  <is-multi-cew type="bool">true</is-multi-cew>
 
  <is-single-engine type="bool">false<is-single-engine>
  <is-multi-engine type="bool">true<is-multi-engine> 
 
  <has-piston-propulsion type="bool">false</has-piston-propulsion>
  <has-jet-propulsion type="bool">true</has-jet-propulsion>
  ...
  </PropertyList>
 
From a programming point of view, this doesn't require any work as it only requires contributors to provide said information-however from a usability point of view, it adds the possibility for GUI/WEB frontends to easily offer a way to search aircraft using this information. By restricting this information to be of Boolean type, it is very straightforward to either directly search the base package or to simply create a small database for very powerful queries-by just having users check/uncheck a corresponding checkbox, no matter whether this is on a webpage or directly in a FlightGear GUI launchr such as fgrun.
 
=== Challenges ===
For some components (like FDM configs, 2D panels, 3D cockpits, exterior models or liveries) it may be possible to specify multiple separately referenced files in one single *-set.xml file, in these cases it would seem more appropriate to also track their development status separately in these files, rather than in the top-level aircraft file, this would make sure that the development status is really tracked on a per-component basis. So, that sharing aircraft resources still works properly while retaining component status information.
 
== Ease of Deployment ==
 
In order to simplify the introduction of the aforementioned concept to store aircraft status information, it might make sense to consider requesting aircraft authors/maintainers by convention to specify said status information externally, in a separate PropertyList-encoded XML file, that is then only referenced in a corresponding include attribute (i.e. <aircraft-status include="ac-status.xml"/>.
 
While this doesn't make any conceptual difference, it helps to to keep this structured information in a separate place.
 
Enforcing this convention could simplify the required parsing process in certain situations and would thus not only help creating web-based search/sort frontends using such status information attributes, but would also enable the wealth of existing standalone FlightGear launchers (most of which do not link to SimGear, and may thus not easily process PropertyList-XML files) to be able to easily deal with this information without having to parse the whole *-set.xml file, rather it would become possible to just concentrate on parsing said "aircraft-status.xml" file for certain purposes.
 
== Overall Rating/Status ==
 
If an overall aircraft rating is desired, it should be possible to easily determine it, as a product of factors, based on the status information of its individual components, taking into account that the weight of said information varies, depending on the type of aircraft, and its requirements.
 
== Related Community Discussions ==
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg14021.html
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg14090.html
* http://www.mail-archive.com/flightgear-devel@lists.sourceforge.net/msg14098.html
* http://mail.flightgear.org/pipermail/flightgear-devel/2004-November/032295.html
* http://www.flightgear.org/forums/viewtopic.php?f=6&t=1531
* http://flightgear.org/forums/viewtopic.php?f=6&t=2214
* [http://www.flightgear.org/forums/viewtopic.php?f=17&t=7698 Aircraft frustration] (04/2010)
* [http://flightgear.org/forums/viewtopic.php?f=4&t=8441 Aircraft Metadata] (06/2010)
* [http://flightgear.org/forums/viewtopic.php?f=6&t=9034 fewer aircraft] (08/2010)
* [http://flightgear.org/forums/viewtopic.php?f=4&t=9057 Naming conventions] (08/2010)
* [http://flightgear.org/forums/viewtopic.php?f=4&t=9680 Discussing aircraft status] (10/2010)


[[Category:Base Package:Aircraft]]
[[Category:Base Package:Aircraft]]
[[Category:Aircraft TODO|*]]
[[Category:Aircraft TODO|*]]
[[Category:RFC]]
[[Category:RFC]]