Feature Scaling
| This article is a stub. You can help the wiki by expanding it. |
| Note In its current form, this section/article is largely based on quotes collected from various related discussions/channels (devel list, forum etc) using the Instant-Cquotes script. Wiki users and other contributors are encouraged to help rewrite/edit contents accordingly to help get rid of unnecessary quoting (i.e. while the wiki is not intended to be a collection of quotes, quotes are sometimes the best/easiest way to bootstrap new articles, while also providing a good way to link back to related discussions in the archives).
While rewriting usually only entails changing first person speech to 3rd person. However, please try to retain references/links to the original discussion whenever possible. |
| Portability & Compatibility |
|---|
| Performance |
| Efforts |
Gallery
Screen shot showing the property browser running on Netbook in a patched version of FlightGear with resource tracking support using the cross platform SIGAR library
Screen shot showing the property browser running on Netbook in a patched version of FlightGear with resource tracking support using the cross platform SIGAR library
CPU information exposed to the property tree using the SIGAR library
ram usage metrics exposed to the property tree (Linux only for now)
system and GPU video memory exposed to the property tree(Nvidia-only for now).
Background
|
It does not seem to be a technical problem - to query a string from the property tree is easy, to write a routine which sets other properties based on what the string says is equally easy.
The problem is the meat of it - someone has to know what settings are likely to run on what graphics cards. And someone has to write the routine that does it. Then it's done. |
|
|
|
I wonder if it would be possible to create configuration files based on OpenGL capabilities to set certain options, mucht like a joystick configuration file but instead for video cards.
The glxinfo utility shows an awful lot of information which probably be used for tuning the default setup. |
|
|
|
I was more referring tot the scene rendering when first starting FlightGear.
And the option to turn on specific ALS options for certain OpenGL renderers. For instance mine is "GeForce 9600 GT/PCIe/SSE2" which can have almost anything set to on without much hit on frame-rate except for procedural 3d textured cities which hurts performance badly. But it does not allow to set the densities much higher than default. |
|
|
|
The only thing I want to do is detect the video card and based on the GL renderer name turn on ALS (or not), turn on water shader (or not), turn on the 3d city shader (or not), etc.
The default would not be touched but based on the detected video card certain options would get enabled, making the first impression much more pleasant (for first time users). |
|
|
|
For the more general question of what to enable by default, what we could really do with is a way to automatically adjust to the hardware. — Rebecca N. Palmer (Aug 13th, 2015). [Flightgear-devel] Defaults Re: Basic Weather.
(powered by Instant-Cquotes) |
|
|
|
I suggested automatically changing the LOD range according to available memory (which would probably be easier), but never got around to actually working on it.
— Rebecca N. Palmer (2015-02-05). Re: [Flightgear-devel] FlightGear on low (<=4GB) memory systems.
(powered by Instant-Cquotes) |
|
|
|
FlightGear's defaults don't scale very well at the moment. But to be honest, backward compatibility has never been a real priority ...
|
|
|
|
what's really needed is a form of dynamic feature scaling that can check the features supported by a platform and then find a working set of features which still provide reasonable performance and a good experience, all of this taking place during start-up.
|
|
|
|
it was stuart who mentioned this a while ago: This could be implemented by using a Nasal script to monitor a handful of properties and by adjusting some other parameters dynamically.
That shouldn't require any C++ modifications. — Hooray (Fri Mar 02). Re: [SUGGESTION] Auto-adjusting fog for FPS control.
(powered by Instant-Cquotes) |
|
|
|
We already have a number of examples doing a simple form of feature-scaling by monitoring certain parameters and then adjusting internal behavior accordingly. The local weather system already uses such heuristics for example. And "property rules" could be easily used to come up with additional rules.
|
|
|
|
anything that can be controlled at runtime via a property can theoretically be also adjusted ("feature-scaling") by watching critical properties like the fps counter or frame spacing latency. You would need to check the rendering settings dialog if your settings are listed there, and extract the corresponding properties, then register a timer that watches frame rate & frame spacing and detects lag (spikes in frame spacing) and throttle down features. That's basically 5-7 lines of trivial Nasal code, trivial even to people who never looked at Nasal before, once you want to apply some filtering, things get a bit more sophisticated, but still do-able.
|
|
|
|
Currently, we are spending resources without having any clue how much we have left - no matter if it's CPU or RAM/VRAM, our tools are pathetic and not well-developed to help with this sort of thing, as has been brought up numerous times by other before.
|
|
|
|
I did give this a try a while ago and ran into a problem.
Let's say, I set my minimum-fps setting to 20. The script will disable features. One at a time, until the fps are 20. However, you do want to have those features enabled again when your fps allows it. Problem is then that you end up in a circle. Once the fps is 30, 3D clouds are enabled, making the fps to drop below 20. Therefore, 3D clouds are disabled again, pushing the fps to 30 again. Which is enough to enabled 3D clouds, etc. etc. So the implentation isn't just a matter of auto enabling/disabling features based on framerates. Some other people might have better ideas/ways than I had though |
|
|
|
Yes - feature scaling isn't likely to work when the granularity of the feature's effect on FPS is so high, e.g. 3D clouds.
One thing you can do to help, if you haven't already, is to use a low pass filter on the FPS input. That will help smooth out any intermittant drops in FPS without disabling loads of features. |
|
|
|
In my opinion, scaling up is not going to be as difficult as scaling down - and for that reason, I would love FG to start up with an extremely minimal startup profile, and dynamically determine if there's hardware available to do more resource-hungry things, such as using shaders, multiple cores, many GBs of RAM etc. Meanwhile, we have reached a point, where it's become a real piece of work to start FG on old computers, or even on Intel-graphics based computers. |
|
|
|
Being able to start up FG using a "safe" subset of enabled features would be great for people with older/underpowered hardware - but it would also be great for us, i.e. from a troubleshooting standpoint, but also from a benchmarking perspective, because we could dynamically scale up settings-based on what's available.
|
|
|
|
At least starting with a minimal profile should be trivial to implement: Define a property /sim/use-minimal-profile at startup, have a Nasal script check whether that property is set, and if yes, have it execute a batch of setprop() statements forcing all config options to minimal. No need to bother users with modern hardware, well-defined starting point for users with older hardware.
|
|
|
|
the first step is providing built-in means to gather system statistics at run-time (analogous to a task/process monitor), to see how CPU/RAM/GPU/VRAM are utilized when certain features are active, so that we can better identify subsystems and features that are resource hogs, e.g. because they eat up RAM or frame-rate. The next step is to either introduce feature-scaling then, or make such features entirely optional.
With a focus on RAM, the basic idea is summarized at: Subsystem-level_Memory_Tracking_for_FlightGear But the same thing could be done for CPU/GPU utilization. |
|
|
|
exposing internal stats to the property tree would still seem useful, especially for people wanting to make bug reports, or even just for dynamic "feature-scaling" using Nasal and/or XML-Property Rules.
|
|
|
|
The neat thing about writing such stats directly to the property tree would be that we could use Nasal scripts to watch those values and change things on demand or issue warnings to the console, for example to implement feature scaling  .
|
|
|
|
having some kind of standardized "API" for feature-scaling would be good - on the other hand, I do believe that we first of all need a way to really "look under the hood" and see which features are expensive in terms of CPU/GPU, RAM and VRAM utilization - otherwise, it is far too easy to keep pointing out how poorly some aircraft perform, without giving aircraft developers the right tools to actually tell what's going on in terms of performance, i.e. regarding textures, 3D models, scripts etc
|
|
|
|
I do think that such a benchmark could probably be implemented directly in FlightGear, just by using Nasal scripting and some custom XML files.
|
|
|
|
SGTimer/maketimer() API would ideally also allow for a callback to be specified in case framerate/performance is too bad to run the timer at the interval specified. sort of like a notification callback that would get called if necessary - that would standardize the various feature-scaling attempts we have, i.e. down-grading features according to frame rate/latency. |
|
|
|
this is often called "feature scaling".If anybody is really interested in this, I think this could be rather easily done by using a Nasal script that dynamically enables, disables and tweaks various settings at runtime, I am willing to provide help if you have trouble getting started doing this in Nasal.
|
|
|
|
Even a very simple benchmark could be useful for feature-scaling purposes, and if it's implemented in a non-interactive fashion, it could even help with regression testing. Ideally, a benchmark would start out with the bare minimum settings and then dynamically change settings on the fly to determine their effect on frame rate and frame spacing, to come up with a list configuration settings that work properly, while ensuring a satisfying simulator experience.
|
|
|
|
it's commonly called "feature scaling", and has been discussed at lengths here. But for the time being, it would be as straightforward to implement as changing your FDM in-flight by adjusting engine/wing foil parameters - I am sure your FDM was never designed with such requirements in mind, and that also applies to most of the legacy FG code that's been around for over 15 years now.
Feature scaling involves not just reading a value/variable once, but either polling it (updating it permanently at frame rate) or acting upon events if some variable has changed, to reset/re-init or adjust the corresponding system - most people didn't/don't write their code that way, because it takes more time - see the XML discussion we had with Stiglr a few days ago: [2] |
|
|
|
"Feature scaling" is a long-standing feature request, and it simply involves making things better configurable, which also happens to be a key ingredient for runtime benchmarking, i.e. "in-game" profiling, which is another thing I am interested in supporting sooner or later. Not so much to have 1:1 benchmarks, but rather to do regression testing in an automated fashion, so that certain problems can no longer go unnoticed.
|
|
|
|
Some settings that come to mind include:
(and probably many others!) |
|
|
|
Many of these things can already be dynamically configured because they are driven using properties, most of these are made available in the "rendering options" dialog, so it would be straightforward to use a Nasal dialog to tweak FlightGear at runtime.
So it would be mostly a matter of using a Nasal script to optionally tune these settings automatically, maybe with some simple GUI dialog for configuration purposes. |
|
|
|
I think it also opens up a larger question of how we do memory management in FG, and whether we should be doing things such as more aggressively freeing up terrain tiles. At one level, removing entire terrain tiles from memory earlier if memory occupancy becomes a concern would be a better management strategy than just stopping generating new buildings.
|
|
|
|
That said, OSG has quite a lot of "clever" machinery to help with memory management, at least with all the rendering-related stuff. For "problems" like excessive ram usage with random buildings and random trees, there are well-defined "standard" solutions available in OSG - but for many others, there aren't any. So rendering-related issues can be obviously addressed by using what's provided by OSG. |
|
|
|
The other LOD setting I keep debating about, is the ‘lod-bias’ feature the OSG pager has. This can be used to bias the LOD metrics arbitrarily, and in particular based on frame-rate. It would need a PID controller to drive it I think (but we have the code for one of those!), but the idea would be to bias LOD to achieve a target frame-rate; effectively ‘stuff’ which uses the OSG pager system (random building/trees, scenery models) would be unloaded if frame-rate dropped.
— James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models.
(powered by Instant-Cquotes) |
|
|
|
I haven’t even made the experiments around this, but conceptually it feels nice to me - I’d much rather the system unload buildings/models to maintain a target 30fps. It would work even better with a ‘bare’ / ‘detailed’ BTG scheme of course, since we’d revert distant tiles to the bare version automatically.
— James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models.
(powered by Instant-Cquotes) |
|
|
|
What I don’t know is how badly the system would hunt for the target frame-rate; i.e load in some expensive PagedLOD containing random buildings -> FPS drops -> bias increases -> PagedLOD unloads -> repeat. That’s why I assume a PID controller is needed, plus some additional logic to reset the bias on re-position. I can also imagine making a 180-degree turn changing the factors hugely (looking at Manhattan vs looking out to sea at KJFK), and the PID controller working the bias hard to deal with that. /And/ as you said above, we know loading+unloading has its own costs.
— James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models.
(powered by Instant-Cquotes) |
|
|
|
Actually what I’d really be looking for here is not unloading, but simply skipping the culling+drawing - maybe that falls out from your expiry time change too.
— James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models.
(powered by Instant-Cquotes) |
|
|
|
X-plane seemingly has something on firstboot that sets graphic settings according to your computer. Having this integrated into FG can prevent some users complaining about bad imagery,although what they forgot is the Rendering setting. Rough algorithm suggestion -Set settings to Lowest/off -Gradually add effects THEN shaders until we get nearest to(but more than) 25fps. Interested? Help me plan the algorithm. I think this will be in Nasal. — Poomklao Teerawattanaprapha (Jan 14th, 2016). [Flightgear-devel] Feature suggestions:Automatic setting of
Rendering.
(powered by Instant-Cquotes) |
|
|
|
It's not as easy as it sounds here. First, users are different - some like 60 fps steady and don't care too much about visuals. Others want opulent visuals and are willing to go down to 10 fps. Second, effects drag the framerate when they're used, i.e. when you look at an object rendered with them. So you don't know what quality level of, say, the water effect the system can sustain at 25 fps unless you're looking at a water surface. Third, visibility influences framerate often more than quality settings - what runs find at 20 km visibility may not at 120 km. Fourth, we have different rendering frameworks to provide options - what do people want to burn their GPU cycles for? You can opt to do it for dynamical lights and shadows, then you get Rembrandt, you can opt to do it for atmosphere visuals, haes and procedural terrain texturing, then you get ALS. Fifth, I have no clue what distinction between effects and shaders you're trying to make here. We do have fixed pipeline effects which don't run any shaders, but they're sort of the lowest fallback option before declaring failure, so we can't really gradually add them. I guess in the end better explaining to users how the rendering settings can be chosen and what affects what or perhaps some more intuitive GUI tweaks may work better. My two cents... but feel free to try it - I've once toyed with an 'hold framerate' controller for how far out we render clouds, and I wasn't really happy with it. — Thorsten Renk (Jan 14th, 2016). Re: [Flightgear-devel] Feature suggestions:Automatic setting of
Rendering.
(powered by Instant-Cquotes) |
|
Efforts
|
|
I may also look at the Nasal side of things - because thanks to all the reset/re-init work, the C++ dependencies are fairly straightforward to decouple these days, I just made the tile manager optional in under 10 minutes, i.e. all the scenery/sky/stars is gone now - which means that FG is now utilizing less than 140 MB of RAM here using this "startup mode".
So this about more than just "FGCanvas": It will make troubleshooting so much more straightforward, and we can also explore feature-scaling and run-time benchmarking once this works. Currently, I'd even consider restructuring initialization to always boot up like this, because it's the safest possible subset of subsystems for now - i.e. roughly a dozen subsystems that basically stand no chance of "crashing" the sim. And from then on, additional initialization could be handled by Nasal obviously - and scripts could even scale up/down, depending on hardware support and features. |
|
|
|
In the near term, it would be better to extend the fgcommand-basd re-initialization mechanism that Zakalawe came up with ($FG_SRC/Main/subsystemFactor.?xx) to specifically support such initialization groups, analogous to "run-levels". That would then allow us to script initialization, without having any hard-coded stuff in fg_init.cxx that should be optional, and it would allow people to use the same mechanism to support benchmarking, feature-scaling or even regression testing.
|
|
|
|
A while back, I started preparing a patch to add the corresponding diagnostics (RAM usage tracking) to FG so that we can better identify problematic features, and either fix them or at least provide feature-scaling support if necessary. Back then, this was unfortunately ignored and disregarded - recently, this is getting some more attention again, so we'll probably revisit the issue during the 3.2 release cycle. The first step really is getting "live" memory usage details, and exposing this via the property tree - so that we can tell 1) how much RAM is there, 2) how much RAM is being used by fgfs, 3) and: are we swapping ?
|
|
|
|
agreed, I once talked with Stuart and TheTom about also adding details about the amount of physical RAM vs. swap usage, too - having these things available in FG, would make troubleshooting around here quite a bit easier, and would also allow us to do feature-scaling, I provided a patch for this for a Linux implementation back then, and we could probably extend this for all main OS (win, mac, linux): [3]
|
|
|
|
For Nasal space, we pretty much have a working solution that could help us identify scripts that are causing lots of new naRefs or temporaries. But C++ code is a different beast, and as can be seen in many discussions here - we have quite a few people with >= 8gb of RAM seeing crashes on 64 bit OS due to "out of RAM" errors.
|
|