FlightGear benchmark: Difference between revisions

Jump to navigation Jump to search

FlightGear benchmark (view source)

Revision as of 17:25, 21 June 2013

132 bytes added ,  21 June 2013
→‎Brainstorming
Line 22: Line 22:


== Brainstorming ==
== Brainstorming ==
=== Purpose ===
* maybe we really do need a benchmark .fgfsrc for proper comparison. [http://flightgear.org/forums/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156437]
* maybe we really do need a benchmark .fgfsrc for proper comparison. [http://flightgear.org/forums/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156437]
* Even a very simple benchmark could be useful for feature-scaling and regression testing purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience. We already have various building blocks in FG to do most of this, it's really just a matter of combining and integrating existing features to provide such a simple benchmark. From a troubleshooting perspective this could in fact also be useful, because we could ask users to open a certain dialog and run a certain benchmark and report the results here. [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177778&hilit=benchmark#p177778]
* We actually talked about that benchmarking idea a while ago, and I even implemented a proof of concept [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741]
* Several people mentioned that they would like to have some form of "benchmark" to run FlightGear on various different platforms to see how it performs.I think the idea is not that bad, and that this might actually help troubleshoot some issues. Also, I do think that such a benchmark could probably be implemented directly in FlightGear, just by using Nasal scripting and some custom XML files. This would be pretty much related to the idea of "feature scaling" which was discussed in the other thread. [http://flightgear.org/forums/viewtopic.php?f=6&t=7806#p75962]
* having a number of benchmarks available could probably provide useful metrics to get FlightGear to run. For example, even the very simple file that I posted can already be used for troubleshooting: if a user is not able to run this with more than 100 fps, he is unlikely to be able to run FlightGear with default settings.
=== Aircraft-set.xml based benchmarks ===
* The only problem is that FlightGear always makes the assumption that it is running some form of aircraft/vehicle, so any sort of "benchmark" needs to be provided as an aircraft. Also, one needs to override the global preferences.xml file because there is no way to use a different one.
* Well, while ago, we talked about creating benchmarks in the form of custom aircraft-set.xml files, which would already contain all startup settings (resolution, bpp, shaders etc) [http://flightgear.org/forums/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156440]
* Well, while ago, we talked about creating benchmarks in the form of custom aircraft-set.xml files, which would already contain all startup settings (resolution, bpp, shaders etc) [http://flightgear.org/forums/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156440]
* This would allow us to share "benchmarks" in the form of aircraft, so that people could easily launch them using fgrun (or whatever GUI frontend they have) - still, it'd be possible to export benchmark results to XML.
* This would allow us to share "benchmarks" in the form of aircraft, so that people could easily launch them using fgrun (or whatever GUI frontend they have) - still, it'd be possible to export benchmark results to XML.
* Several people mentioned that they would like to have some form of "benchmark" to run FlightGear on various different platforms to see how it performs.I think the idea is not that bad, and that this might actually help troubleshoot some issues. Also, I do think that such a benchmark could probably be implemented directly in FlightGear, just by using Nasal scripting and some custom XML files. This would be pretty much related to the idea of "feature scaling" which was discussed in the other thread. [http://flightgear.org/forums/viewtopic.php?f=6&t=7806#p75962]
* One would only need a way to create a default situation (i.e. like a custom preferences.xml file) and a way to dynamically toggle FlightGear features on/off and tweak them at runtime.
* One would only need a way to create a default situation (i.e. like a custom preferences.xml file) and a way to dynamically toggle FlightGear features on/off and tweak them at runtime.
=== Approaches ===
* This should be pretty straightforward to do, at least for those features (configuration properties) that are already using listeners or that are read every frame. This applies to most of the recent graphics additions (i.e. shaders), because these can be dynamically enabled, disabled and configured.
* This should be pretty straightforward to do, at least for those features (configuration properties) that are already using listeners or that are read every frame. This applies to most of the recent graphics additions (i.e. shaders), because these can be dynamically enabled, disabled and configured.
* So a FlightGear benchmark would then only have to be run with common default settings (e.g. window resolution, color depth, startup airport, aircraft and environment settings) while a Nasal script could then be used to dynamically tune these settings. Reading internal counters (namely the framerate counter for the time being) would then give us an instrument to see how significant certain settings are.
* So a FlightGear benchmark would then only have to be run with common default settings (e.g. window resolution, color depth, startup airport, aircraft and environment settings) while a Nasal script could then be used to dynamically tune these settings. Reading internal counters (namely the framerate counter for the time being) would then give us an instrument to see how significant certain settings are.
* In the beginning, the easiest way to have something like a benchmark in FlightGear would be to simply use static "situations" that are loaded from XML files, these would then override all local custom settings so that users can reliably compare their frame rates when running such "situations" on different machines.
* In the beginning, the easiest way to have something like a benchmark in FlightGear would be to simply use static "situations" that are loaded from XML files, these would then override all local custom settings so that users can reliably compare their frame rates when running such "situations" on different machines.
* The only problem is that FlightGear always makes the assumption that it is running some form of aircraft/vehicle, so any sort of "benchmark" needs to be provided as an aircraft. Also, one needs to override the global preferences.xml file because there is no way to use a different one.
* having a number of benchmarks available could probably provide useful metrics to get FlightGear to run. For example, even the very simple file that I posted can already be used for troubleshooting: if a user is not able to run this with more than 100 fps, he is unlikely to be able to run FlightGear with default settings.
*  imagine we would create a bunch of additional "benchmarks" like this, each of those testing individual features of FlightGear (shaders, effects, particles, shadows, AI aircraft and so on), all of these could be useful to allow users to see if their system (and configuration) is able to run FlightGear or if it needs to be modified (software/hardware configuration). [http://flightgear.org/forums/viewtopic.php?f=6&t=7806]
*  imagine we would create a bunch of additional "benchmarks" like this, each of those testing individual features of FlightGear (shaders, effects, particles, shadows, AI aircraft and so on), all of these could be useful to allow users to see if their system (and configuration) is able to run FlightGear or if it needs to be modified (software/hardware configuration). [http://flightgear.org/forums/viewtopic.php?f=6&t=7806]
=== Extending the replay/flight recorder subsystems ===
* Maybe we could use the flight recorder to record a flight, so that more people could try the same flight, recreating your settings ? That would basically be a simple benchmark [http://flightgear.org/forums/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625]
* Maybe we could use the flight recorder to record a flight, so that more people could try the same flight, recreating your settings ? That would basically be a simple benchmark [http://flightgear.org/forums/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625]
* Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult to create a simple benchmark framework. [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741]
* Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult to create a simple benchmark framework. [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741]
Line 39: Line 46:
* We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177733&hilit=benchmark#p177741]
* We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177733&hilit=benchmark#p177741]
* Still, the idea of creating a simple, easily recreatable benchmark flight, sounds good to me! What aircraft would be suitable? Is the UFO in the standard installation? I don't know, since I use the fgdata from git for years... Then I would make some flight over KSFO, which should be on every FG installation, and then? What would I do with the recorder tape? upload it somewhere? [http://flightgear.org/forums/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625]
* Still, the idea of creating a simple, easily recreatable benchmark flight, sounds good to me! What aircraft would be suitable? Is the UFO in the standard installation? I don't know, since I use the fgdata from git for years... Then I would make some flight over KSFO, which should be on every FG installation, and then? What would I do with the recorder tape? upload it somewhere? [http://flightgear.org/forums/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625]
* Even a very simple benchmark could be useful for feature-scaling and regression testing purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience. We already have various building blocks in FG to do most of this, it's really just a matter of combining and integrating existing features to provide such a simple benchmark. From a troubleshooting perspective this could in fact also be useful, because we could ask users to open a certain dialog and run a certain benchmark and report the results here. [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177778&hilit=benchmark#p177778]
* We actually talked about that benchmarking idea a while ago, and I even implemented a proof of concept [http://flightgear.org/forums/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741]


== Community Feedback ==
== Community Feedback ==
Retrieved from "https://wiki.flightgear.org/Special:MobileDiff/61058"

Navigation menu