20,741
edits
Software testing: Difference between revisions
Jump to navigation
Jump to search
m
→Headless testing
| Line 93: | Line 93: | ||
=== Headless testing === | === Headless testing === | ||
{{Main article|FlightGear Headless}} | |||
For an FDM+systems test, we should run FG without a renderer (which is what the test_suite does), to benchmark the pure C++ performance of whatever system it is we care about (FDM or whatever). But this is not really worth doing anyway, since a few hours playing with ‘perf’ on Linux / Instruments on macOS will show you that 80% of our CPU time is spent in OSG + the GL drivers, and hence Amdhal’s law will always get you.<ref>https://sourceforge.net/p/flightgear/mailman/message/36977666/</ref> | |||
=== Graphics testing === | |||
The other thing is to setup some test-case scenes where you can quickly measure differences and compare via screenshots. The brain/eye/memory are very bad at this stuff, setup something you can load from the command line via a script to test old/new versions, if possible. <ref>https://sourceforge.net/p/flightgear/mailman/message/36959002/</ref> | |||
For example, a test would would be a .fgfsrc that makes: | |||
* c172p at some parking in a detailed airport; | |||
* camera looking at it in specific direction with specific FOV; | |||
* AW with specific METARs around (if not possible, BW with specific METAR); | |||
* fixed rendering settings (several variants may be needed for renderers and threading modes) <ref>https://sourceforge.net/p/flightgear/mailman/message/36975122/</ref> | |||
We can already load a [[Instant Replay|replay tape]] on startup, however, which is good for testing rendering (scenery, loading) performance, since the FDM is not involved, nor is the user-interface. | |||
But essentially any and all of the methods proposed can be done here with small amounts of shell-script + Nasal hacking I think, and any of them would be welcome additions. For lower-level tests run by developers, the unit-test framework is great (i.e ‘does the API call produce the right results in the system’), but a smoke-test that regular users can run would be ideal. | |||
A rendering performance would likely do: | |||
* select some particular rendering settings (clouds, draw distance, etc) | |||
* run a saved fgtape recording | |||
* record the mean/min/max FPS during this and save it in some text file / copy to the clipboard | |||
So yes, if anyone wants to work on the above, the code is all there, please jump in and start hacking - I don’t think it needs any more from the core code, but as ever, please just ask if it does.<ref>https://sourceforge.net/p/flightgear/mailman/message/36975213/</ref> | |||
For example, describing a /rendering/ test (to establish FPS) - the advantage of a replay tape is the actual position (and therefore what is rendered) will be 100% consistent acorss different computers. | |||
Keep in mind that the CPU use of the FDM+systems is typically < 10% of our total CPU use, even when running OSG single-threaded, so for a rendering performance test, whether the FD is run or not is probably noise compared to other things that do run (Nasal, Canvas for example) | |||
Also, Multi-monitor setup is not tested so commonly unfortunately. <ref>https://sourceforge.net/p/flightgear/mailman/message/36904782/</ref> | |||
== Fgdata == | == Fgdata == | ||