Canvas scenery overlays

This article is a stub. You can help the wiki by expanding it.

Canvas - the FlightGear 2D drawing API
Canvas - Main Page Canvas News - for the latest developments Shiva Alternatives - ongoing work for Core Profile support Canvas FAQs - frequently asked questions Canvas - Getting started Canvas Development (core) Canvas Troubleshooting (core) Canvas Sandbox (core) PUI2Canvas (PUI/XML Parser) Canvas GUI Canvas Widgets Canvas Maps Canvas Snippets
RFC
Canvas Instancing Howto:New Canvas Execution Model Canvas SVG Canvas Popout Windows Canvas Scenery Overlays
Frameworks
Canvas Map API (soon deprecated!) Canvas MapStructure Canvas MapStructure Layers MapStructure Snippets Canvas Radar Canvas Animation Framework (WIP) Canvas Plotting Framework (WIP) Canvas MFD Framework Canvas ND Framework Canvas PFD Framework Canvas MCDU Framework Canvas EFB Framework Canvas Wrappers Canvas Glass Cockpit Efforts
Tutorials
Howto: Canvas Troubleshooting Howto:Adding a canvas to a GUI dialog (PUI!) Howto:Creating a Canvas GUI dialog file (New) Howto:Dynamic Liveries via Canvas (new) Howto:Creating a Canvas GUI Widget Howto:Add a 2D canvas instrument to your aircraft Howto:Parsing 2D Instruments using the Canvas Howto:Creating fullscreen Canvas applications Howto:Use SVG inside a Canvas Howto:Extend the Canvas SVG module Howto:Serializing a Canvas to SVG Howto:Canvas applications as Nasal submodules Howto:Using raster images and nested canvases Canvas HUD Howto:Project 3D to 2D coordinates on desktop canvas
Reference
Canvas Element Group Image Map Path Text Canvas Nasal API MessageBox Layout System Canvas GUI API Canvas draw library Event Handling
Related
Canvas Properties Unifying the 2D rendering backend via canvas Howto:Use a Camera View in an Instrument Howto:Extending Canvas to support rendering 3D models Howto:Extending Canvas to support MathGL
To be ported
Navigation display Map FGCanvas Canvas Nasal/JavaScript Subset
Deprecated by Canvas
Complex Glass Cockpit Avionics
(click on headings to toggle)

When we have vector road data at runtime, we can do the following: render it in moving-map displays - either real ones (Garmin G1000 / G430) or the map dialog use it to drive building outline creation as Thomas was suggesting animate traffic on the roads, as point lights at night or even meshes in daytime (the important one...) generate tile overlay textures dynamically (and based on view distance) to show the roads on the terrain surface. (And the detail of this texture can include road markings / borders based on distance from the viewer / view angle / etc) — James Turner (2014-11-21). Re: [Flightgear-devel] Future city terrain strategy. (powered by Instant-Cquotes)

Using the Canvas system for adding geo-referenced textures to the terrain mesh has actually a plethora of potential applications and use-cases in FlightGear, and it could help unify/integrate a number of existing features, and make them much better accessible, i.e. by moving them to fgdata space:

• Osm2city.py
• Random Buildings
• AutoGen Scenery for FlightGear
• Photoscenery

As of mid 2016, this is revamped by increasing interest in coming up with texture-based landclass representations:

The idea is that we move away from vectorized landclasses and encode the information on landclasses in a raster-image (i.e. a meta-texture). This has a couple of advantages:

1. Generating an image seems to be much easier than vectorizing complex data structures
2. By avoiding any mesh subdivisions due to landclasses, the mesh vertex count can be cut (drastically?)
3. We get full control over how the transitions between landclasses look like
4. We can easily fit man-made landclasses into the rest of the scenery in a plausible way
5. Landclass editing by the end user can then be made with graphics software, no terragear runs are required

We should create the meta-texture at runtime from the vector data - i.e use CGAL + GDAL to ‘paint’ the vector data into textures as they are needed (note that the corresponding tgdb code already supports gdal optionally). This means the resolution of the meta-texture tiles can be adaptive, which gives a trivia solution to handling roads and similar - draw them into the meta texture (or an additional one) adaptively based on distance from viewer, etc.

Applications

In regard to the cray city texture stretching on the mountain, this is something that seems like a fundamental flaw in our texture coordinate calculations. We only calculate them in 2-D ( lon, lat ), so any cliff, steep slope stretch the texture out. The landclass shader mitigates this programatically, but with just plain textures, it looks awful. Ideally, city textures would be just gravel, etc, and could be done procedural, then random / osm building could be placed on top.

City textures are unlikely to work too well on slopes, i.e. they're usually made with the assumption that they're taken from a bird's eye-view (straight down), so applying them to a slope would require dealing with that assumption in a sane way, which would probably require some fairly fragile heuristics (basically changing the projection of the texture at runtime, which is unlikely to look that good, because the re-projection would need to extrapolate texels because it is lacking most relevant information, such as the color of a wall, because it only knows the color of the roof-top). Probably, it would be best to move away from static city textures and decompose the whole thing into its components and then re-add those procedurally, like you say. We already have most parts for this in place, they're just not very well integrated at the moment (think materials, effects, shaders, photo-scenery/texture draping, FBOs/canvas) For instance, if/when the Canvas system is extended to allow arbitrary Canvas textures to be registered as "materials" (in the Scenery/SimGear sense), that would make effects/shaders automatically available to Canvas textures, and the only other addition required is allowing a Canvas texture to be draped onto the terrain mesh, e.g. by using a customied subset of the existing photo-scenery patches. This may not be the most elegant, or the fastest, option - but it would provide for a truly generic toolbox to conduct various scenery/texturing related experiments without requiring major C++ changes apart from two entirely self-contained additions that would be useful even regardless of this particular use-case.

Background

If you want to pass substantial amounts of data, it makes sense to use a texture (with filtering disabled, probably) to pass the info to the effect/shader subsystem. Since we don’t have much chance of using the ‘correct’ solution (UBOs) in the near future.

So far, the recommendation has been to write dedicated C++ code to populate such a texture. Equally, photo-texturing requires C++ hooks to place overlay textures onto the terrain mesh and transform them properly.

This article documents how the Canvas system can be extended to provide support for experimenting with new scenery schemes. Specifically, by adding support for three new features:

• registering Canvas textures as materials (this allows effects/shaders to render into an empty buffer)
• allowing materials to be set per Canvas element (this makes arbitrary effects/shaders available to each element)
• adding a new geo-referenced placement for a Canvas texture to be draped over the terrain mesh

it should take roughly 20 lines of C++ code to allow an arbitrary Canvas texture to be registered as a "material", at that point it could use shaders and effects

for example, if we looked at the effects map, we could use that to provide an API that allows a Canvas to be registered as a material, e.g. using something like: myCanvas.registerMaterial("myAirportMaterial")

As long as that material/name is referenced in materials.xml that should work.

It seems, that would be the most straightforward/easiest way to hook up Canvas and effects - i.e. treating a Canvas texture as a material, at which point a cascade of Canvas-based FBOs can be set up in conjunction with effects/shaders for arbitrary post-processing.

For airport stuff, vector drawing capabilities like those available via Canvas (Canvas.Path/ShivaVG) could come in handy - so all that is needed is to "bridge" both worlds: to expose a Canvas to shader space so that you can use it, but also so that an arbitrary Canvas can be draped over terrain.

Problem

A number of folks have been playing with populating their home airport's area with buildings derived from OSM floorplan data, having many buildings in the correct place greatly improves realism over the current random buildings/sparse static models, especially when you know the area. However, now the buildings obviously don't match with ground textures or random trees.

This should be possible, it would have pretty much the benefits and drawbacks you mention (especially the VRAM consumption), and it would allow nice LoD and solve some other issues. Especially it avoid the nasty clipping issues we have with surface data in TerraGear, since you just paint into the texture, no need to clip all the linear data.

Motivation

There used to be a lack of coordination between the terrain generation pipeline and the rendering side. There's a lot of work done at load time because the .btg format was considered immutable.

FG is not Google Earth, but many uses of FG aren't that different from the use cases of a global terrain viewer; some people want a Space Shuttle simulator, others want to do nape-of-the-earth helicopter flying. IFR people don't care :) The point is that the rendering should be flexible enough to handle these different scenarios, especially given that so much work is going into building detailed terrain.

Another approach is used by MSFS X, which generates the terrain texture for a tile at a given LOD completely at run time (on the CPU!), including vector features, so that the appropriate texture resolution for sharp vector features is always available.

Once we have infrastructure in place to handle landclass merging, we could move away from X.0 scenery releases, and would only need to regenerate the entire planet if a new feature is added to the toolchain itself. Unfortunately, this is 'mostly' not a toolchain, or feature problem, it's an infrastructure problem

We're contemplaing moving away from vector landclasses and utilie shader compositing instead. The advantage is shown here: (see the linked image) (the text is drawn in a different landclass...) Unlike for vector seams, you can blur boundaries as you like (also using biased mixing functions,...). Also, the vertex count can be kept low because no subdivisions for landclasses need to be made.

instead of draping the airport could we just use (or specify a second) texture that has the airport info for a procedural shader? I think it's a very great concept; it has the possibility to solve a lot of the texture related landclass issues.

The idea for man-made landclasses would be to really put them 'on top' of natural surfaces, such that the original topology (and colors) are allowed to partially influence the visuals.

textures are sort of scale invariant entities - they just care that they have a texture and texture coordinates. Which means you can use the same renderer and feed it textures at whatever resolution you need as you get closer to an object, and you can pass a single pixel alpha = 0 texture when you don't need it. So for LOD purposes, textures look much more handy than vector geometry.

procedural techniques can add details like erosion, discoloration,... down at the milimeter level, so they don't have to be there.

this approach is a brilliant step forward from our current scheme (and 'photorealistic' scenery as well), And it is promising enough that I wouldn't mind loosing some features early on, to be added later. And being an Open Source project: many hands make light work, so release early and release often.

Status

This may or may not become an outlook at the structure of WS 4 It's a bit early to talk about memory consumption, framerate or random objects. There's a whole list of things which will need to change if we adopt this and a whole bunch of yet unsolved problems - which we need to study. You're not going to see any of this in FG for a while.

Approach

The encoding to be used is what Thorsten is trying to research at the moment. Thorsten is aiming for some experience with the rendering part so that he knows what's needed, useful and not relevant before stating potential requirements for the scenery creation.

Much of the complexity ( and problems ) in the terragear toolchain has to do with the landclass polygon clipping. Generating a raster image for a texture lookup is orders of magnitude easier

More proof of concept - we can select a texture from a palette based on the info encoded on the meta-texture and draw landclass boundaries hard, smooth or noise-distorted as desired: (see the linked image)

We may want to add complexity to the raster image though - specify what pattern to use for the transition for instance, store information about roughness and height-mapping

Canvas integration

Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments.

it does make sense to delegate as much power, and flexibility, to fgdata space (effects, shaders) as possible - the Canvas system would just seem like one of the most obvious candidates to be extended to support geo-referenced placements and let its textures to be used in conjunction with various effects and shaders, as both, the input buffer but also the output buffer of an effect, i.e. a rendering stage/pass.

a "Canvas" is just a fancy word for an OSG FBO/RTT context with a bunch of quads rendered into the texture, that is then placed somewhere in the scene - so compared to the screenshots, it's a dynamic texture rather than a static one. The main difference would be that you could do something like myCanvas.myText.setText("Blurry") to affect the texture, and call myCanvas.addPlacement({[lat0,lon0], [lat1,lon1]})

Basically a Canvas is just a FBO/RTT after all, and we do have the notion of arbitrary "placements" - thus, coming up with a new mode for geographic placements would not be very much work given the code/patches we have already.

This is not necessarily a suitable long-term strategy (see the PagedLOD implications),but it would provide for a very solid prototyping platform to come up with a proof-of-concept so that people can tinker with different approaches, without requiring tons of C++ changes (or familiarity with SG/TG).

Like psadro stated on the devel list, a 4096x4096 texture would be sufficient for most needs - but for markings etc specifically, it would be insufficient. However, it would be possible to apply markings using a different/additional overlay, i.e. that just applies to airports, taxiways and roads etc

James mentioned recently that said vector information would come in handy for many modern cockpit display - and as you can see in various MFDs/dialogs, we are already able to obtain /some/ vector information and turn that into a raster image.

The same raster image could be overlaid onto terrain tiles using geo-referencing, but it would also be possible to use such a texture as the input, or output buffer, for an arbitrary effects/shader pass - which is to say that the 4096x4096 sheet could be used for photo-imagery, and another texture could be merged with that using effects/shaders to apply markings etc selectively.

This approach provides a straightforward mechanism to allow others to explore different routes without having to dedicate months, or years, of C++ coding to see if something is a dead-end or not.

Realistically, we already have most components in either SG or FG (or some topic branches), it's just that these features are poorly integrated currently (think effects not being able to a Canvas and vice versa).

But as soon as the Canvas system provides support for geo-referenced terrain overlays, and has some integration with effects, our eye candy folks would basically be unstoppable, without having to know much about TerraGear or C++ at all.

we are already able to create a texture representation of arbitrary airport features like runways, taxiways etc - even in Nasal/Canvas space using a handful of APIs (navdb), which works using the Canvas::Map element to project lat/lon tuples of arbitrary Canvas::Path segments to the texture

This texture could be simplified/customied and fed to a shader/effect: Canvas Development#Effects .2F Shaders

And it could also be draped using geo-referencing, for which we have existing SimGear patches: Photoscenery#Using photoscenery This way, if/when Canvas+Shaders/effects are better integrated, many TerraGear-level things could actually be moved to fgdata space

Notice that the photoscenery patches are fairly old meanwhile, last I looked at them, I had something working that would render an arbitrary texture to a hard-coded tile using a lat/lon tuple - but meanwhile, all the PagedLOD refactorings are complicating things a bit - I guess that Stuart or poweroftwo would be in a better position to judge if/how that functionality can be generalied and integrated/exposed at the SimGear level to be useful here ... Also, given the shifting focus of the discussion, the Canvas route may be super-flexible, but it won't align nicely with all the effects/shader and PagedLOD infrastructure that you need, i.e. the Canvas system would literally need to provide its texture in a PagedLOD-compatible form which has other implications, especially from a multi-threading standpoint, and a texture would need to be able to use shaders and effects, for both, input and output. Thus, for prototyping and a proof-of-concept, the old patches may be suitable and should be straightforward to integrate with the Canvas system to come up with a geographic Canvas placement - but all the other fancy stuff you are talking about would sooner or later benefit from using osg::PagedLOD and osg::OverlayTexture - none of which is currrently used by the Canvas system.

One of the more straightforward ways to have our cake and eat it, would be providing a custom Canvas element type, implemented in C++, that renders generic vector data to a raster image (think ESRI-like stuff) - but then again, that's quickly approaching what osgEarth is already good at, like James mentioned.

However, that's the kind of stuff for which osgEarth is using GDAL/OGR already ... so it would not make sense to reinvent the wheel here - osgEarth-enabled builds will have gdal/ogr support included anyway

it could be pretty useful to allow a Canvas texture (aka the raster image created by a Canvas FBO) to be registered with the material cache (SGMaterialLib), i.e. in the form of a custom "material" - that would then automatically imply that all effects/shader infrastructure code be used "as is", without any additional C++ plumbing needed.

C++ wise that would mainly mean making a call to SGMaterialLib::insert() to register an arbitrary Canvas texture with the material library (as per $SG_SRC/scene/material/mat*.cxx)

Draping a texture onto the terrain mesh is a different beast though and will require integration with the PagedLOD changes, most of the original code was written long before that - so is out of date; fortunately, the photoscenery patches (wiki) are not exactly huge, i.e. it's pretty self-contained actually.

In its current form it will just look for <tilename>.dds files and drape them over the tile using 2 modified effects/shaders, and a patched obj.cxx that exposes the corresponding lat/lon tuples to shader space.

Canvas materials

For the existing noise texture generation, take a look at make3DNoiseImage in scene/material/TextureBuilder.cxx in the simgear tree. This particular texture is filled CPU-side, but if you're more comfortable expressing things in a shader, you/someone could rig up some FBO wiring to quickly render a shader-defined function into a texture.

the code that creates the texture is in scene/util/StateAttributeFactory.cxx, but you would have to do something similar to the noise texture code in TextureBuilder to access your texture from a texture. It would be nice if the Effects framework had a way to load arbitrary textures and make them available to effects.

These days you aren't limited to 8 bit texture components. There is a rich set of data formats for textures: 32 bit floats, 16 bit floats, exotic combinations of different length floats per RGB component... I don't know if there is a better way to create your texture offline than write C++ code in simgear. OSG will read a TIFF file with 32bits per component as a floating point texture... assuming you can create such a thing.

If you want to apply effects to other kinds of models, you would need to generalize MakeEffectVisitor "in both directions." StateSet objects can appear in any scene graph node and also in the Geometry nodes that sit below osg::Geode. A more general effects visitor needs to track the graphics state as it traverses the scene graph and also needs to examine the geometry. Effects sit at the Geode level.

Base package

diff --git a/Shaders/default.frag b/Shaders/default.frag
index 1c3fbaf..e9ede17 100644
--- a/Shaders/default.frag
+++ b/Shaders/default.frag
@@ -7,6 +7,7 @@ varying vec4 diffuse_term;
 varying vec3 normal;
 
 uniform sampler2D texture;
+uniform sampler2D overlay;
 
 ////fog "include" /////
 uniform int fogType;
@@ -50,8 +51,10 @@ void main()
     // is closer to what the OpenGL fixed function pipeline does.
     color = clamp(color, 0.0, 1.0);
     texel = texture2D(texture, gl_TexCoord[0].st);
+    vec4 texel2 = texture2D(overlay, gl_TexCoord[3].st);
+    texel = mix(texel, texel2, texel2.a);
     fragColor = color * texel + specular;
 
     fragColor.rgb = fog_Func(fragColor.rgb, fogType);
     gl_FragColor = fragColor;
-}
\ No newline at end of file
+}
diff --git a/Shaders/default.vert b/Shaders/default.vert
index 1bc2405..94f5c44 100644
--- a/Shaders/default.vert
+++ b/Shaders/default.vert
@@ -30,6 +30,8 @@ uniform int colorMode;
 void main()
 {
     gl_Position = ftransform();
+    gl_TexCoord[0] = gl_TextureMatrix[0] * gl_MultiTexCoord0;    
+    gl_TexCoord[3] = gl_TextureMatrix[3] * gl_MultiTexCoord3;
     gl_TexCoord[0] = gl_TextureMatrix[0] * gl_MultiTexCoord0;
     normal = gl_NormalMatrix * gl_Normal;
     vec4 ambient_color, diffuse_color;

SimGear

Hooray has been reviewing the changes that would "break" the old photoscenery patch (as it can currently be found on the wiki) - and fortunately, it's just a single "recent" refactoring/commit, made by psadro_gm (in early 2015), one where he split up obj.cxx and moved everything related to the patch to $SG_SRC/scene/tgdb/SGTileGeometryBin.hxx: SimGear commit 148640be3450e34416c77111103f637c451362c8

It seems, this is the main/only thing that no longer applies cleanly - and it's responsible for setting up the overlay coordinates and pass them to the shaders.

So, that's basically what we'd need to review/discuss to re-add the photoscenery functionality so that arbitrary scenery overlays (no matter the concrete use-case) can be added - however, one of the comments in that file mentioned that the method is deprecated, i.e only used by WS 1.0, so I guess psadro would be in the best position to judge if/how to re-introduce that functionality, so that overlays also work with WS 2.0+

Maybe psadro could briefly review the changes, and the patch, and tell us how to best make this work with the more recent WS 2.0 scheme ?

FlightGear

The FlightGear side of things will need a way to register a new Canvas placement so that orthographic/terrain placements can be implemented. This will basically require implementing the Canvas::Placement interface, i.e. by looking at $SG_SRC/canvas/CanvasObjectPlacement.?xx

In particular, this new placement type will need a tuple of lat/lon coordinates, the texture dimensions to use, as well as other placement information affecting the transformation/projection

These attributes would be exposed in the form of placement specific properties in the property tree.

Nasal console

Work in progress This article or section will be worked on in the upcoming hours or days. See history for the latest developments.

Let's assume, we'd like to render a splash screen onto the terrain mesh underneath the aircraft:

Canvas dialog rendering a splash screen, that is to be placed in the scenery (terrain mesh) using a geo-referenced quad

var (width,height) = (320,320);
var title = 'Canvas Overlay Test';

var window = canvas.Window.new([width,height],"dialog").set('title',title);

var myCanvas = window.createCanvas();
myCanvas.set("background", "#ffaac0");

var root = myCanvas.createGroup();

var path = "Textures/Splash1.png";
var child = root.createChild("image")
    .setFile(path)
    .setTranslation(10, 10)
    .setSize(300, 300);

# get the position of the aircraft
var myPosition = geo.aircraft_position();

# var geoPlacement = { type: "geo-referenced", lat0:, lon1:, lat1:, lon2: };

References

References