Read canvas image by HTTP

Canvas/HTTP Server

Started in	10/2016
Description	Serving Canvas textures via httpd
Contributor(s)	ThomasS (since 10/2016)
Status	Available as of 2018.3.1 (still might cause stability problems)

Objective

See also Howto:Multi-instance Canvas use.

There are situations, e.g., for home cockpit builders , where it is useful to display instruments like a PFD, ND, EICAS or any MFD externally from the FlightGear 3D main window in a separate window or on a separate monitor, computer or a mobile device .

Many of these avionics/graphics are created by FlightGear's 2D drawing Canvas system internally.

In addition there, are a number of other use-cases where being able to obtain a Canvas from fgfs using a network protocol like http may be desirable (e.g. imagine getting a tilemap based on actual scenery from FlightGear ) .

This article provides a patch to FlightGear for downloading any canvas image from a running FlightGear process by HTTP by serializing it to a raster image and serving that via the built-in mongoose based httpd server .

This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.

An additional option for displaying canvas images on remote computer is FGQCanvas, which doesn't require to patch the core FlightGear code.

Problem

a modern biz jet will need to be able to display certain MFDs - fgpanel is not too useful for that (unless you are primarily dealing with steam gauges) - the most useful search terms for the forum/wiki will be Phi and FGCanvas - the latter of which is a special startup mode of FlightGear itself that can render Canvas based MFDs in a separate instance.

If you don't need fancy MFDs, you could probably use fgfs standalone and/or fgpanel. Phi has the lowest barrier to entry probably for people familiar with HTML and JavaScript, i.e. you don't even need to know much about fgfs. The Canvas stuff will really only be needed if you want to render things like the 747 PFD/ND or CDU in a distributed fashion, i.e. using multiple independent displays.

you should be aware of glass cockpit related efforts, especially Canvas - most airliners and jets will sooner or later benefit from being ported to Canvas, e.g. to use Gijs' NavDisplay framework, or at least Philosopher's MapStructure framework for mapping purposes.

Thus, if this is also about the actual display itself, people should be aware of related canvas efforts, especially FGCanvas: FGCanvas

The my mid-term plan involves supporting a standalone mode for all Canvas-based glass instruments, including the ND, but also other instruments like the PFD, EICAS, CDU or EFB. This may sound like a lot of work, but it's mainlyy a matter of introducing a a few helper classes and ensuring that people actually adopt and use those.

In the long-term, we really want to support distributed FlightGear setups like those at FSWeekend/LinuxTag, where multiple computers may be used to run a single simulator session - including properly synchronized glass instruments like the PFD/ND etc. This would also help improve the multiplayer experience, especially dual-pilot setups etc. Regarding a pure panel with switches, kobs and buttons, we'd prefer something like that to be aircraft-agnostic, i.e. just consist of property-mapped controls that can be individually assigned, so that this could be reused for other MFDs, not just the ND - but also the PFD or EFB.

There's the long-term plan to eventually port FGPanel back into FG and come up with some sort of "FGCanvas" mode, where Canvas-based displays could be run in a separate "standalone" mode and interface to the main/master FG instance.

I don't think that the Canvas system currently builds against just OpenGL ES - but it should be possible to identify problematic use-cases and make the Canvas support OpenGL ES by setting some OSG traits accordingly - at some point, i.e. 12+ months from now In general, it pays off to unify things - otherwise, there's lots of duplication and re-invention involved.

Technically, we could definitely run an OpenGL ES-based Canvas on a mobile device, we just need people interested in pursuing this and playing with it - as well as report any issues/showstoppers Such an integrated solution would also make it possible to show any canvas texture (instrument, dialog/window, MFD etc) on external devices:

FGCanvas

Implementation

This solution was created roughly by:

• adding a DrawCallback to the canvas camera
• attach an image to the canvas camera
• duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera

Alternatives

There are possible alternative solutions for rendering canvases on a remote system, that remove the rendering load from the main flightgear process. These might be based on Nasal, Javascript, Python, etc.

One currently available solution is FGQCanvas, created by James.

Gallery

• 

  Screenshot showing a Canvas camera rendered to an image streamed to FireFox

• 

  Canvas Map dialog streamed to FireFox

• 

  FlightGear PUI XML dialog rendered via Pui2canvas and streamed to FireFox

• 

  Screenshot showing a FlightGear PUI GUI dialog with two independent NavDisplay instances streamed to FireFox

• 

  Screenshot showing Canvas MFDs streamed to Firefox

• 

  Screenshot showing a single Canvas texture (NavDisplay) streamed to firefox via httpd at 2hz

Status (1/2019)

This feature was merged into the official 2018.3.1 release. It works so far. However, there are latencies and the solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms. For the time being, this should still be considered a "proof-of-concept" that may go through more iterations of optmizations. Integrating it into 2018.3.1 makes it possible to explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback. Special attention might be required with situations where the canvas and/or the connection are shut down, but also to support Reset & re-init without causing a race condition.

Note  In its current form, the streaming capability is available but might overburden the flightgear process, depending on the number of canvases displayed. Its recommended not to use the streaming feature but to refresh the canvas from the browser. If any instability of FlightGear occurs, the remote canvas feature should be disabled for checking whether it is the reason.

Optimizations

• Sample rate ~1-3 hz
• JPEG_QUALITY <= 6
• image size ~ 512x512
• RTSP/ffmpeg streaming
• Turn the whole thing into a configurable placement for capturing/streaming purposes
• Howto:Serializing a Canvas to SVG (streaming a Canvas/MFD as a SVG/XML document)

Using

Start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL

http://localhost:5701/screenshot?canvasindex=4&type=png

in your browser. The first request will add the property rendertoimage to the canvas with index 4 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 4 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout). For the 777-200 canvas index 4 is the ND, 5 the MFD. PFD seems not to use Canvas.

Sample Showcase CanvasView.html

The following showcase HTML snippet CanvasView.html shows the main instruments of Soitanens 737 (and its branches) with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load. The script contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs.

<!DOCTYPE html>
<!-- See Flightgear wiki for information-->
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Canvas View</title>
</head>
<body>
<select id="sizebox" onChange="setSize();">
    <option value="256" selected="selected">256</option>
    <option value="512">512</option>
    <option value="768">768</option>
</select>
<table>
    <row>
        <td>
            <img id="CptPFD">
        </td>
        <td>
            <img id="CptND">
        </td>
    </row>
    <row>
        <td>
            <img id="UpperEICAS">
        </td>
        <td>
            <img id="CptCDU">
        </td>
    </row>
</table>

<script type="text/javascript">
var refreshinterval=500;
var instruments = ["CptPFD","CptND","UpperEICAS","CptCDU"];
var canvasindex = [];
canvasindex["CptPFD"] = 3;
canvasindex["CptND"] = 5;
canvasindex["UpperEICAS"] = 4;
canvasindex["CptCDU"] = 2;

function refreshImage(imageid) {
    //var image = document.getElementById(imageid);
    var image = document.querySelector("#"+imageid);    

    var url = "http://localhost:5701/screenshot?type=png&canvasindex="+canvasindex[imageid];
    image.src = url;
    setSize(imageid);
    setTimeout(function() {
         refreshImage(imageid);
     }, refreshinterval);
   
}

function setSize(imageid){
    var e = document.getElementById("sizebox");
    var size = e.options[e.selectedIndex].value;
    var image = document.querySelector("#"+imageid);
    if (image != null){
        image.style.width=size+"px";
        image.style.height="auto";
    }
}

for (var i = 0; i < instruments.length; i++) {
    refreshImage(instruments[i]);
}
//refreshImage("CptND");
//refreshImage("CptPFD");


</script>
</body>
</html>

Discussion

The forum thread for discussing this feature is https://forum.flightgear.org/viewtopic.php?f=71&t=30642&start=15.

Related

• https://sourceforge.net/projects/remote-canvas-for-flightgear
• forum search for ffmpeg+stream FlightGear Forum
• github/openscenegraph/osg/examples/osgmovie/osgmovie.cpp

• http://trac.openscenegraph.org/projects/osg//wiki/Support/Tutorials/LoadingProgress
• http://trac.openscenegraph.org/projects/osg//wiki/Support/KnowledgeBase/SerializationSupport
• http://bensoftware.com/blog/comparison-of-streaming-formats/
• http://www.osgart.org/index.php/NodeCallback_Scene_Setup

References

References