PagedLOD

It has been suggested that this article or section be merged into Scenery LoD.

This article is a stub. You can help the wiki by expanding it.

See Scenery LoD for the main article about this subject.

PagedLOD is an enhanced LOD node supported by OpenSceneGraph. The conventional LOD node requires all of its immediate children be present in memory at-once (a requirement inherited from osg::Group). For scenegraphs with large numbers of LODs, this can become prohibitively costly in terms of memory consumption, even though only one LOD child of each LOD Node is being drawn at any one time.

The answer to this predicament is the PagedLOD node. PagedLOD is derived from LOD, and adds the ability to defer the loading of child nodes until they are actually required. A PagedLOD node may have a child immediately present, or it may not have the child present, instead storing a path/name where the child node can be loaded from on-demand [1].

We’re now using PagedLOD for sub-elemts of the tile, including objects, random veg+buildings, and maybe lights too. This is a big win since base tile loading is faster and we don’t pay the memory cost until the relevant stuff is in view. (And, it can be paged out again)

we use PageLoD nodes and the OSG paged loader to determine what stays in memory. Models are loaded in a separate thread, and then added to the scenegraph. In theory this should mean there is no impact due to model loading from disk. However, even though the model is loaded, mipmaps haven't been generated for all the textures at this point - this only happens when the texture is loaded into the graphics card memory. In theory, this mipmap creation should be handled by the graphics card itself, but it looks like there's a driver bug which means this is falling back to the CPU to perform, and causing additional work in the main thread. Additionally, there are OSG limits on the number of PagedLOD nodes that will be kept in memory at any one time.So, even after the model has been loaded, it may nevertheless end up paged to disk if it goes out of view. IIRC Matthias mentioned a number around 300. The workaround for the first problem is to manually generate the mipmaps for textures in the model loader thread, and indicate to the GPU that it doesn't need to generate mipmaps itself. This is something I plan to look into when I have the opportunity, though I'll need to ask for patience as it's outside my current expertise.

Status

we don’t have many cases of selecting differently detailed models, but all the nodes and machinery around paging and loading tiles and models is in classes whose name contains LOD, especially the class PagedLOD.

The key parameters there are setting view distances at which the content referenced by the PagedLOD is loaded, and determining when (due to culling / being too far away) models can be unloaded. Even when the PagedLOD only has a single ‘detail level’, all that machinery and parameters are exactly what drives our loading schedule, memory consumption and therefore quite a bit of our performance.

We use “PagedLOD” OSG constructs in a lot of places - AI models, scenery models and even the loading of STG files. Effectively it allows OSG to load something when it comes into range but otherwise leave it on disk. Obviously that loading from disk has a cost so if we have the memory, it’s worth keeping objects in memory even if they are temporarily out of sight.

While the slider goes all he way up to 10,000, we likely never had that number of PagedLOD and they were relatively low footprint.

WS3.0 uses PagedLOD for the scenery tiles themselves. There are a lot at different resolutions and they have textures. So if we’re not unloading them when they either go out of range or the views gets close enough for the level below to be loaded then memory consumption is going to get very high.

Assuming this is the problem that Durk and JP are seeing, then in the short term we should limit the slider to perhaps 4000.

Longer term we might need to find a way to separate the PagedLOD of the scenery from other elements.

we need some tagging / classification there, so we can use different thresholds or policies for scenery tiles compared to other PagedLOD use cases. Fortunately that should be some quite specific points where we create the PagedLOD instances so hopefully not too disruptive for most of the code.

The view distance also determines when the model referenced by a PagedLOD is loaded in the first place.)

The other elements that have a significant impact is the behaviour of the DatabasePager which loads the models in one or more separate threads. This is controlled by the following properties:

• /sim/rendering/database-pager/threads - the number of loading threads

(next only)

• /sim/rendering/max-paged-lod - the maximum number of PagedLOD models to

keep in memory

• /sim/rendering/plod-minimum-expiry-time-secs - the minimum time since

last "active" to keep a PagedLOD in memory (irrespective of /sim/rendering/max-paged-lod)

If the latter two properties are too large then memory occupancy balloons. If too low, then the DatabasePager can end up wasting time reloading models and you'll see delays in model loading.

WS3.0 relies very heavily on PagedLOD. Each 1x1 degree tile has ~340 of them if every subtile was loaded (which it shouldn't be!). So tuning the DatabasePager and getting the right balance is going to be more important going forward. The default values of 1800 PagedLOD nodes and an expiry time of 5 minutes are almost certainly too high and cause memory usage to grow until one runs out of memory.

One area where we do use LOD nodes very heavily is in the trees, clouds and instanced buildings (aka "random buildings"). The core code for this is now 13 years old, and at the time Tim Moore and I were tuning carefully to balance CPU and GPU load. We used a significant number of LOD notes (a quad-tree of depth 3 for each tile in the case of trees) to cull out as many of the trees as possible. Given modern graphics cards, that may not be as important as it was and it may be better to reduce the number of LoD nodes. Fortunately that is parameterized as an argument into the createForest call, so could be experimented with if anyone is interested - just hack the code and use the OSG on-screen statistics to look at some forested region of Canada or Russia!

It's worth getting some real data on this. Under the Debug menu there's a Dump Scenegraph which will dump the entirety of the scene graph to STDOUT/STDERR (I forget which). I haven't done so in a while, but it's a useful tool and a bit of grepping will give some insights as to just how many LOD and PagedLOD nodes we actually have. We've got lots of code that is run for each AI model, or each scenery tile so it adds up fast!

References LOD (Level of Detail) in OpenSceneGraph.

Background

one of the problems is that most of our objects have no concept of different LOD's. If we had objects with say 3 LOD, it would ease memory and graphics load. It should be possible to take an object and generate multiple LOD's for it automatically — Adrian Musceac (2013-09-20). Re: [Flightgear-devel] Upcoming Random Buildings changes. (powered by Instant-Cquotes)

I am about to drop most of the current code (in tilemgr, tilecache, tileentry, scenerypager, ...) in favor of (possibly subclassed) PagedLOD nodes for tiles. my last patch has made that a lot easier anyway. — till busch (2008-05-23). [Flightgear-devel] DatabasePager changes. (powered by Instant-Cquotes)

’ve checked with Mathias, and his alternate paging scheme (replaces the tile manager, using osgDB pager / pagedLODs for everything) is certainly a viable option here. It supports using a blue-marble style image as a base level of detail (think SGOceanTile, but with vertex colours at least showing land-cover), and he has support for coarse detail tiles too. ‘All’ that’s missing is generating some suitable tiles for this medium-level scheme, which I think is all work on TerraGear side. There are two features incompatible with this paging system - particles; due to some long standing design defects in that code, and integration with TerraSync. Both of these I would say are surmountable obstacles with some hacking, and life without particles is bearable if we want to run the two systems side-by-side for comparison. Personally, I am not sure runtime simplification is worthwhile for the increased complexity - I am not saying there are /no/ users it will benefit, but I am not sure it’s a sufficiently large set to be worthwhile. A pre-generated scheme as suggested above is a straightforward win for everybody, in comparison - both for users with lower-end systems, /and/ for those who want to push the draw distances. (And it reduces TerraSync bandwidth usage too, depending on the kind of flights you make) — James Turner (2014-02-24). Re: [Flightgear-devel] Terrain Simplifier. (powered by Instant-Cquotes)

Tweaking

’ve added some optional features to reduce memory consumption, and since ‘next’ is for testing, I’ve switched them both on by default in current Git. They are: /sim/tile-cache/enable Previous was effectively ‘true’, the default on next right now is ‘false’, so the tile cache is disabled. This leads to some visible popping when switching between tower and cockpit views, but otherwise I can’t see any problem with disabling the cache - and it saves an enormous amount of memory. Unless testing reveals any problem, I’d propose to make this default to ‘false’ for 3.2.0 too. /sim/rendering/max-paged-lod This exposes an OSG value, which we previously did not adjust. This is how many PagedLODs to keep around, even when they are inactive (outside the current view, in practice). This controls how quickly we unload Paged elements of tiles - trees & random buildings principally. The default value of 300 means we’re keeping many PagedLODs for tiles outside the view, so this has been lowered on next to 8. Some tuning of this value is welcomed, whether 0, 8, 16 or 32 make any appreciable difference to popping or memory consumption. I can imagine setting it to zero leading to more visible popping of trees & buildings when flying circuits around an airfield - that’s the motivation for the current value. — James Turner (2014-05-18). [Flightgear-devel] Memory consumption reduction. (powered by Instant-Cquotes)

If you comment out the new entries in prefrences.xml, the C++ code will use the old values (cache enabled, max-paged-lod=300) automatically. Of course, the mose important test is if this makes the 2.0 scenery usable on 32-bit machines. I can’t perform that test, so hopefully some other people here can. — James Turner (2014-05-18). [Flightgear-devel] Memory consumption reduction. (powered by Instant-Cquotes)

LOD Bias

for fly by view, the new scheme will cause way more unloading and re-loading, which is wasteful - equally your old ‘spin around to trigger loading’ trick is a programmer level hack that a normal user would be mystified by. Really what we’re saying is the DB Pager’s model of movement in the scene is wrong for fly-by-view, and we need to adjust it’s unloading parameters in that mode, for example setting a temporal expiry instead of a spatial one. If objects were to remain loaded for 10 or 20 seconds (or whatever value fly-by-view uses) /after/ they become invisible, fly by view should work perfectly with the pager. (And as far as I can recall, the Pager / PagedLOD nodes have settings for exactly that) — James Turner (2014-11-04). Re: [Flightgear-devel] Building of framerate doom. (powered by Instant-Cquotes)

The other LOD setting I keep debating about, is the ‘lod-bias’ feature the OSG pager has. This can be used to bias the LOD metrics arbitrarily, and in particular based on frame-rate. It would need a PID controller to drive it I think (but we have the code for one of those!), but the idea would be to bias LOD to achieve a target frame-rate; effectively ‘stuff’ which uses the OSG pager system (random building/trees, scenery models) would be unloaded if frame-rate dropped. — James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models. (powered by Instant-Cquotes)

I haven’t even made the experiments around this, but conceptually it feels nice to me - I’d much rather the system unload buildings/models to maintain a target 30fps. It would work even better with a ‘bare’ / ‘detailed’ BTG scheme of course, since we’d revert distant tiles to the bare version automatically. What I don’t know is how badly the system would hunt for the target frame-rate; i.e load in some expensive PagedLOD containing random buildings -> FPS drops -> bias increases -> PagedLOD unloads -> repeat. That’s why I assume a PID controller is needed, plus some additional logic to reset the bias on re-position. I can also imagine making a 180-degree turn changing the factors hugely (looking at Manhattan vs looking out to sea at KJFK), and the PID controller working the bias hard to deal with that. /And/ as you said above, we know loading+unloading has its own costs. Actually what I’d really be looking for here is not unloading, but simply skipping the culling+drawing - maybe that falls out from your expiry time change too. — James Turner (2014-11-19). Re: [Flightgear-devel] Improved LOD handling for AI Models. (powered by Instant-Cquotes)

FGViewer

I have in fgviewer a PagedLOD whole world database tree running. This is similar to osg's marble dataset but carefully designed around our tile structure. Using this I think we can really replace a lot of our fine structured scenery tiles with something more croase that is used for tiles more far away. Drawback with our current codebase: Our integration of the particle systems need to be rethought as this contains geometry with culling disabled which makes a pagedlod just never expire. Switching the particle systems off works pretty good so far. — Mathias Fröhlich (2012-07-21). Re: [Flightgear-devel] Rendering passes question. (powered by Instant-Cquotes)

AI Traffic

I was not aware that we limit the OSG default of 300 to only 16 paged LODs. Given our usages of PLODs, this number is way to low. With each AI model being such an object, a busy airport easily exceeds that limit. Increasing that number seems to make sense but it comes at the cost of increased memory usage and more time spent in scenegraph traversal. I'd be interested in the actual number of active PLOD objects, do you know if the on-screen-statistics displays that number? — Torsten Dreyer (2015-06-08). Re: [Flightgear-devel] max-paged-lod too low?. (powered by Instant-Cquotes)

64 seems OK for AI disabled but for AI enabled it's still too low. — Torsten Dreyer (2015-06-15). Re: [Flightgear-devel] max-paged-lod too low?. (powered by Instant-Cquotes)

aircraft disappearing from view but still being present in the pilot list may be due to OSG paging out the aircraft too aggressively. Try increasing /sim/rendering/max-paged-lod to 200 and see if that helps — Stuart Buchanan (2015-06-16). Re: [Flightgear-devel] The 707 on the multiplayer servers. (powered by Instant-Cquotes)

All AI models are now PagedLOD nodes, instead of plain LOD nodes, precisely so they can be unloaded. The rationale (my rationale!) here was that any popping when changing views here, was outweighed by many AI models far outside the view being kept in memory. In general we want to use the OSG paging system for loading and unloading, instead of our own custom logic which was previously happening. Now, of course the PagedLOD view/distance settings can be adjusted to change when loading / unloading occurs, but I would say the current behaviour is closer to what we want (OSG Pager makes decisions about loading) than the previous situation. — James Turner (2014-11-04). Re: [Flightgear-devel] Building of framerate doom. (powered by Instant-Cquotes)

Our AI Models are handled by so called PagedLOD nodes in OSG. These nodes get loaded and deleted on OSG's descretion. I have just added a user-settable parameter for a minimum expiry time to prevent models from getting unloaded as soon as they move out of view. This value now defaults to 180secs, enough to hold the models im memory during a standard rate turn. I think this PagedLOD loading/unloading was the main cause for some serious stutter with AI traffic enabled and rapid view movements (fly-by-view or sharp turns). — Torsten Dreyer (2014-11-19). [Flightgear-devel] Improved LOD handling for AI Models. (powered by Instant-Cquotes)

Random Buildings

t would really help memory use if we could defer tree/object/building generation until the tile is close to the viewer. Right now you generate lots of LOD nodes so we don't pay the rendering cost for distant tiles, but we're still paying a memory cost and loading time hit. The ideal approach is to use PagedLOD, i.e let the osgDB pager do the job it's intended for. So the base tile would have a PagedLOD which loads the building / trees / objects when the LOD threshold trips, with the usual queuing system and unloading. What this needs is to make a pseudo-file-name for to add to the loader, which causes a custom osgDB ReaderWriter to run. (Likely with a custom Options instance set specifying any parameter data needed for the tile - is there any? I can't recall) That ReaderWriter can then return the root osg::Node for the trees/buildings/objects as we already do. — James Turner (2013-09-18). Re: [Flightgear-devel] Upcoming Random Buildings changes. (powered by Instant-Cquotes)

Random objects are not generated when the tile is loaded. Instead they are deferred and loaded by a PagedLOD call when you get within ~ 15km of the tile centre. That avoids us having to generate them if we never get close to the tile, for example when we've got high visibility set. — Stuart Buchanan (2014-11-22). Re: [Flightgear-devel] Building of framerate doom. (powered by Instant-Cquotes)

With the PagedLOD approach the random objects will be generated only when the aircraft gets into range, so I'm hoping I can simply check against the bounding box of each static object. — Stuart Buchanan (2013-10-11). Re: [Flightgear-devel] Upcoming Random Buildings changes. (powered by Instant-Cquotes)

I've been working on using PagedLoD to deferred random object (buildings, trees, objects) so that they are not generated when the tile is loaded, but only when the viewer gets within a specific range. — Stuart Buchanan (2013-12-06). [Flightgear-devel] Deferred/Paged random object loading. (powered by Instant-Cquotes)

I've got a PagedLOD system working pretty well that defers loading of trees, random objects, random buildings and random lights for the entire tile until you get withing 30km of the tile. I haven't measured the memory usage impact with very large visibility ranges yet, but it should be significant — Stuart Buchanan (2013-11-26). Re: [Flightgear-devel] New scenery. (powered by Instant-Cquotes)

The ReaderWriters run in the osgDB pager thread, which is exactly where the current ReaderWriterSTG runs (which ultimately does the current tree/object/building placement, I think) - so the threading should not change at all. Indeed, the more I think on it, the more it feels like this should be a very small restructuring of the code, just requiring some slightly delicate PagedLOD plumbing to make it work. We're already doing the right work (building an osg::Node) and doing it in the right thread, we just need to change *when* we do it. BTW, I would also hope this means the fixed osg::LOD elements in the quadtrees could go away, or at least the layering reduced (to one layer of LODs instead of two) My gut feeling is the LOD-tree is preventing sufficiently large batching for both trees and random buildings. But of course the optimal batch size is very hardware dependant. — James Turner (2013-09-18). Re: [Flightgear-devel] Upcoming Random Buildings changes. (powered by Instant-Cquotes)

The bare tiles are indeed much simpler (and friendlier to GPUs) than everything we put on top, since there’s no animation nodes. If we want to go to really large draw distances, having a lower-detail tile strategy would still help (as I think we all know), but unlike things discussed so far in this thread, what you’re working on is (comparatively) low hanging fruit, not requiring any drastic tooling or whole-program changes. — James Turner (2013-11-26). Re: [Flightgear-devel] New scenery. (powered by Instant-Cquotes)