4,400
edits
FlightGear benchmark: Difference between revisions
Jump to navigation
Jump to search
m
Update forum links
m (→Status: https://gitorious.org/testflight) |
m (Update forum links) |
||
| Line 1: | Line 1: | ||
{{Template:Mentored Volunteer Effort | {{Template:Mentored Volunteer Effort | ||
|mentors=[http://flightgear.org | |mentors=[http://forum.flightgear.org/memberlist.php?mode=viewprofile&u=1092 Hooray] (get in touch to learn more) | ||
|skills=[[PropertyList XML File]], [[Aircraft-set.xml]], [[Property Tree]], [[Nasal|Nasal scripting]], fgcommands, [[Howto:Making HTTP Requests from Nasal|Nasal/Web scripting]]}} | |skills=[[PropertyList XML File]], [[Aircraft-set.xml]], [[Property Tree]], [[Nasal|Nasal scripting]], fgcommands, [[Howto:Making HTTP Requests from Nasal|Nasal/Web scripting]]}} | ||
| Line 58: | Line 58: | ||
Obviously a flightgear specific benchmark could be much more suitable for flightgear than a generic gaming benchmark and much more helpful for people figuring out what settings are best for their systems, so what do you guys think of the idea? | Obviously a flightgear specific benchmark could be much more suitable for flightgear than a generic gaming benchmark and much more helpful for people figuring out what settings are best for their systems, so what do you guys think of the idea? | ||
I am totally new to flightgear and I haven't seen any indication that fg has a benchmarking system, so I thought that would be nice. I could provide/specify a list of settings and record some flights (if said feature exists), although I doubt I have the fg experience and authority to do so, so I hope you guys can sort this out- besides, a benchmark system should be pretty easy to implement compared to like say adding more realism/ better graphics/ revamping the engine, so I think that including a set of specific benchmarking tools and settings is plausible<ref>{{cite web |url=http://flightgear.org | I am totally new to flightgear and I haven't seen any indication that fg has a benchmarking system, so I thought that would be nice. I could provide/specify a list of settings and record some flights (if said feature exists), although I doubt I have the fg experience and authority to do so, so I hope you guys can sort this out- besides, a benchmark system should be pretty easy to implement compared to like say adding more realism/ better graphics/ revamping the engine, so I think that including a set of specific benchmarking tools and settings is plausible<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?f=6&t=19202&hilit=benchmark|title=Flightgear-specific benchmark|author=Ericolon|date=Thu Feb 21, 2013 4:35 am}}</ref>|Ericolon}} | ||
{{cquote|http://wiki.flightgear.org/Howto:Debugging_FlightGear_Crashes#Minimal_Startup_Profile | {{cquote|http://wiki.flightgear.org/Howto:Debugging_FlightGear_Crashes#Minimal_Startup_Profile | ||
| Line 84: | Line 84: | ||
== Brainstorming == | == Brainstorming == | ||
=== Purpose === | === Purpose === | ||
* maybe we really do need a benchmark .fgfsrc for proper comparison. [http://flightgear.org | * maybe we really do need a benchmark .fgfsrc for proper comparison. [http://forum.flightgear.org/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156437] | ||
* Even a very simple benchmark could be useful for feature-scaling and regression testing purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience. We already have various building blocks in FG to do most of this, it's really just a matter of combining and integrating existing features to provide such a simple benchmark. From a troubleshooting perspective this could in fact also be useful, because we could ask users to open a certain dialog and run a certain benchmark and report the results here. [http://flightgear.org | * Even a very simple benchmark could be useful for feature-scaling and regression testing purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience. We already have various building blocks in FG to do most of this, it's really just a matter of combining and integrating existing features to provide such a simple benchmark. From a troubleshooting perspective this could in fact also be useful, because we could ask users to open a certain dialog and run a certain benchmark and report the results here. [http://forum.flightgear.org/viewtopic.php?f=6&t=19202&p=177778&hilit=benchmark#p177778] | ||
* We actually talked about that benchmarking idea a while ago, and I even implemented a proof of concept [http://flightgear.org | * We actually talked about that benchmarking idea a while ago, and I even implemented a proof of concept [http://forum.flightgear.org/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741] | ||
* Several people mentioned that they would like to have some form of "benchmark" to run FlightGear on various different platforms to see how it performs.I think the idea is not that bad, and that this might actually help troubleshoot some issues. Also, I do think that such a benchmark could probably be implemented directly in FlightGear, just by using Nasal scripting and some custom XML files. This would be pretty much related to the idea of "feature scaling" which was discussed in the other thread. [http://flightgear.org | * Several people mentioned that they would like to have some form of "benchmark" to run FlightGear on various different platforms to see how it performs.I think the idea is not that bad, and that this might actually help troubleshoot some issues. Also, I do think that such a benchmark could probably be implemented directly in FlightGear, just by using Nasal scripting and some custom XML files. This would be pretty much related to the idea of "feature scaling" which was discussed in the other thread. [http://forum.flightgear.org/viewtopic.php?f=6&t=7806#p75962] | ||
* having a number of benchmarks available could probably provide useful metrics to get FlightGear to run. For example, even the very simple file that I posted can already be used for troubleshooting: if a user is not able to run this with more than 100 fps, he is unlikely to be able to run FlightGear with default settings. | * having a number of benchmarks available could probably provide useful metrics to get FlightGear to run. For example, even the very simple file that I posted can already be used for troubleshooting: if a user is not able to run this with more than 100 fps, he is unlikely to be able to run FlightGear with default settings. | ||
=== Regression Tests === | === Regression Tests === | ||
Troubleshooting bug reports is often extremely tedious, because we need to replicate lots of settings: | Troubleshooting bug reports is often extremely tedious, because we need to replicate lots of settings: | ||
* "What's the highest shader level, at which random buildings still work? Or the lowest at which they fail?" [http://flightgear.org | * "What's the highest shader level, at which random buildings still work? Or the lowest at which they fail?" [http://forum.flightgear.org/viewtopic.php?f=68&t=18924&hilit=recorder#p175577] | ||
* we should add a menu item to dump the current position and all rendering/environment settings to an XML file, so that we can more easily reproduce such things, just by loading a config from a file. [http://flightgear.org | * we should add a menu item to dump the current position and all rendering/environment settings to an XML file, so that we can more easily reproduce such things, just by loading a config from a file. [http://forum.flightgear.org/viewtopic.php?f=68&t=18924&hilit=recorder&start=15#p176017] | ||
* hat's a super idea! It wouldn't surprise me if some of these glitches are peculiar to specific hardware configurations, either, so perhaps that might be part of the report as well. I'll paste XML into forum posts all day if it helps the devs fix bugs. [http://flightgear.org | * hat's a super idea! It wouldn't surprise me if some of these glitches are peculiar to specific hardware configurations, either, so perhaps that might be part of the report as well. I'll paste XML into forum posts all day if it helps the devs fix bugs. [http://forum.flightgear.org/viewtopic.php?f=68&t=18924&hilit=recorder&start=15#p176023] | ||
* even just knowing that certain issues only occur with some GPUs would be VERY good to know. But obviously we would need a sane way to easily reproduce a certain configuration, including all startup settings, but also the runtime rendering settings. [http://flightgear.org | * even just knowing that certain issues only occur with some GPUs would be VERY good to know. But obviously we would need a sane way to easily reproduce a certain configuration, including all startup settings, but also the runtime rendering settings. [http://forum.flightgear.org/viewtopic.php?f=68&t=18924&hilit=recorder&start=30#p176025] | ||
* After all, having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion. We could add a dialog so that people could even describe the problem - so that the XML files would become self-contained and could be easily checked by different people without having to ask tons of tedious questions... Thinking about it, the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog), which should give us a way to reproduce settings fairly well. [http://flightgear.org | * After all, having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion. We could add a dialog so that people could even describe the problem - so that the XML files would become self-contained and could be easily checked by different people without having to ask tons of tedious questions... Thinking about it, the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog), which should give us a way to reproduce settings fairly well. [http://forum.flightgear.org/viewtopic.php?f=68&t=18924&hilit=recorder&start=30#p176025] | ||
=== Aircraft-set.xml based benchmarks === | === Aircraft-set.xml based benchmarks === | ||
* The only problem is that FlightGear always makes the assumption that it is running some form of aircraft/vehicle, so any sort of "benchmark" needs to be provided as an aircraft. Also, one needs to override the global preferences.xml file because there is no way to use a different one. | * The only problem is that FlightGear always makes the assumption that it is running some form of aircraft/vehicle, so any sort of "benchmark" needs to be provided as an aircraft. Also, one needs to override the global preferences.xml file because there is no way to use a different one. | ||
* Well, while ago, we talked about creating benchmarks in the form of custom aircraft-set.xml files, which would already contain all startup settings (resolution, bpp, shaders etc) [http://flightgear.org | * Well, while ago, we talked about creating benchmarks in the form of custom aircraft-set.xml files, which would already contain all startup settings (resolution, bpp, shaders etc) [http://forum.flightgear.org/viewtopic.php?f=5&t=16204&p=156440&hilit=benchmark#p156440] | ||
* This would allow us to share "benchmarks" in the form of aircraft, so that people could easily launch them using fgrun (or whatever GUI frontend they have) - still, it'd be possible to export benchmark results to XML. | * This would allow us to share "benchmarks" in the form of aircraft, so that people could easily launch them using fgrun (or whatever GUI frontend they have) - still, it'd be possible to export benchmark results to XML. | ||
* One would only need a way to create a default situation (i.e. like a custom preferences.xml file) and a way to dynamically toggle FlightGear features on/off and tweak them at runtime. | * One would only need a way to create a default situation (i.e. like a custom preferences.xml file) and a way to dynamically toggle FlightGear features on/off and tweak them at runtime. | ||
| Line 107: | Line 107: | ||
* So a FlightGear benchmark would then only have to be run with common default settings (e.g. window resolution, color depth, startup airport, aircraft and environment settings) while a Nasal script could then be used to dynamically tune these settings. Reading internal counters (namely the framerate counter for the time being) would then give us an instrument to see how significant certain settings are. | * So a FlightGear benchmark would then only have to be run with common default settings (e.g. window resolution, color depth, startup airport, aircraft and environment settings) while a Nasal script could then be used to dynamically tune these settings. Reading internal counters (namely the framerate counter for the time being) would then give us an instrument to see how significant certain settings are. | ||
* In the beginning, the easiest way to have something like a benchmark in FlightGear would be to simply use static "situations" that are loaded from XML files, these would then override all local custom settings so that users can reliably compare their frame rates when running such "situations" on different machines. | * In the beginning, the easiest way to have something like a benchmark in FlightGear would be to simply use static "situations" that are loaded from XML files, these would then override all local custom settings so that users can reliably compare their frame rates when running such "situations" on different machines. | ||
* imagine we would create a bunch of additional "benchmarks" like this, each of those testing individual features of FlightGear (shaders, effects, particles, shadows, AI aircraft and so on), all of these could be useful to allow users to see if their system (and configuration) is able to run FlightGear or if it needs to be modified (software/hardware configuration). [http://flightgear.org | * imagine we would create a bunch of additional "benchmarks" like this, each of those testing individual features of FlightGear (shaders, effects, particles, shadows, AI aircraft and so on), all of these could be useful to allow users to see if their system (and configuration) is able to run FlightGear or if it needs to be modified (software/hardware configuration). [http://forum.flightgear.org/viewtopic.php?f=6&t=7806] | ||
=== Extending the replay/flight recorder subsystems === | === Extending the replay/flight recorder subsystems === | ||
* We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings [http://flightgear.org | * We do have a so called "flight recorder/replay" system that can save flights. The whole system is property-driven, and it is possible to provide custom sets of properties that should be recorded. In other words, it would be possible to create a custom "flight recorder" configuration that doesn't just record aircraft settings, but also rendering related settings [http://forum.flightgear.org/viewtopic.php?f=6&t=19202&p=177733&hilit=benchmark#p177741] | ||
* Maybe we could use the flight recorder to record a flight, so that more people could try the same flight, recreating your settings ? That would basically be a simple benchmark [http://flightgear.org | * Maybe we could use the flight recorder to record a flight, so that more people could try the same flight, recreating your settings ? That would basically be a simple benchmark [http://forum.flightgear.org/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625] | ||
* Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult to create a simple benchmark framework. [http://flightgear.org | * Using a combination of prerecorded flights, the replay/flight recorder system and a Nasal script to change setting on the fly, it wouldn't necessarily be very difficult to create a simple benchmark framework. [http://forum.flightgear.org/viewtopic.php?f=6&t=19202&p=177741&hilit=benchmark#p177741] | ||
* having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion.[http://flightgear.org | * having an easy way to reproduce a certain configuration, could save us tons of time and question asking - so having such a feature would be really invaluable in my opinion.[http://forum.flightgear.org/viewtopic.php?p=176025#p176025] | ||
* the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog) and implement a benchmark, which should give us a way to reproduce settings fairly well. [http://flightgear.org | * the simplest option would seem to be using existing stuff. After all, this is just about recording and replaying properties. And that's exactly what the new flight recorder (replay tapes) system does. So we could simply abuse it a little to also provide a configuration to sample the various rendering properties (see rendering dialog) and implement a benchmark, which should give us a way to reproduce settings fairly well. [http://forum.flightgear.org/viewtopic.php?p=176025#p176025] | ||
* Still, the idea of creating a simple, easily recreatable benchmark flight, sounds good to me! What aircraft would be suitable? Is the UFO in the standard installation? I don't know, since I use the fgdata from git for years... Then I would make some flight over KSFO, which should be on every FG installation, and then? What would I do with the recorder tape? upload it somewhere? [http://flightgear.org | * Still, the idea of creating a simple, easily recreatable benchmark flight, sounds good to me! What aircraft would be suitable? Is the UFO in the standard installation? I don't know, since I use the fgdata from git for years... Then I would make some flight over KSFO, which should be on every FG installation, and then? What would I do with the recorder tape? upload it somewhere? [http://forum.flightgear.org/viewtopic.php?f=18&t=19738&p=181625&hilit=benchmark#p181625] | ||
== Community Feedback == | == Community Feedback == | ||