Read canvas image by HTTP

From FlightGear wiki
Revision as of 07:16, 21 January 2019 by ThomasS (talk | contribs) (→‎Using)
Jump to navigation Jump to search
This article describes content/features that may not yet be available in the latest stable version of FlightGear (2020.3).
You may need to install some extra components, use the latest development (Git) version or even rebuild FlightGear from source, possibly from a custom topic branch using special build settings: .

This feature is scheduled for FlightGear (unknown). 10}% completed

If you'd like to learn more about getting your own ideas into FlightGear, check out Implementing new features for FlightGear.


Canvas/HTTP Server
Canvas-ND-served-via-httpd.png
Started in 10/2016
Description Serving Canvas texturues via httpd
Contributor(s) ThomasS (since 10/2016)
Status Under active development as of 10/2016

Objective

ND-Dialog-with-DisplayMode-support.png

There are situations, e.g., for home cockpit builders [1], where it is useful to display instruments like a PFD, ND, EICAS or any MFD externally from the FlightGear 3D main window in a separate window or on a separate monitor, computer or a mobile device [2] [3] [4].

Many of these avionics/graphics are created by FlightGear's 2D drawing Canvas system internally.


In addition there, are a number of other use-cases where being able to obtain a Canvas from fgfs using a network protocol like http may be desirable (e.g. imagine getting a tilemap based on actual scenery from FlightGear [5]) [6][7].


This article provides a patch to FlightGear for downloading any canvas image from a running FlightGear process by HTTP by serializing it to a raster image and serving that via the built-in mongoose based httpd server [8] [9].

This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.

An additional option for displaying canvas images on remote computer is FGQCanvas, which doesn't require to patch the core FlightGear code.

Problem

a modern biz jet will need to be able to display certain MFDs - fgpanel is not too useful for that (unless you are primarily dealing with steam gauges) - the most useful search terms for the forum/wiki will be Phi and FGCanvas - the latter of which is a special startup mode of FlightGear itself that can render Canvas based MFDs in a separate instance.

If you don't need fancy MFDs, you could probably use fgfs standalone and/or fgpanel. Phi has the lowest barrier to entry probably for people familiar with HTML and JavaScript, i.e. you don't even need to know much about fgfs. The Canvas stuff will really only be needed if you want to render things like the 747 PFD/ND or CDU in a distributed fashion, i.e. using multiple independent displays.[10]

you should be aware of glass cockpit related efforts, especially Canvas - most airliners and jets will sooner or later benefit from being ported to Canvas, e.g. to use Gijs' NavDisplay framework, or at least Philosopher's MapStructure framework for mapping purposes.

Thus, if this is also about the actual display itself, people should be aware of related canvas efforts, especially FGCanvas: FGCanvas

The my mid-term plan involves supporting a standalone mode for all Canvas-based glass instruments, including the ND, but also other instruments like the PFD, EICAS, CDU or EFB. This may sound like a lot of work, but it's mainlyy a matter of introducing a a few helper classes and ensuring that people actually adopt and use those.

In the long-term, we really want to support distributed FlightGear setups like those at FSWeekend/LinuxTag, where multiple computers may be used to run a single simulator session - including properly synchronized glass instruments like the PFD/ND etc. This would also help improve the multiplayer experience, especially dual-pilot setups etc. Regarding a pure panel with switches, kobs and buttons, we'd prefer something like that to be aircraft-agnostic, i.e. just consist of property-mapped controls that can be individually assigned, so that this could be reused for other MFDs, not just the ND - but also the PFD or EFB.[11]

There's the long-term plan to eventually port FGPanel back into FG and come up with some sort of "FGCanvas" mode, where Canvas-based displays could be run in a separate "standalone" mode and interface to the main/master FG instance.

I don't think that the Canvas system currently builds against just OpenGL ES - but it should be possible to identify problematic use-cases and make the Canvas support OpenGL ES by setting some OSG traits accordingly - at some point, i.e. 12+ months from now In general, it pays off to unify things - otherwise, there's lots of duplication and re-invention involved.

Technically, we could definitely run an OpenGL ES-based Canvas on a mobile device, we just need people interested in pursuing this and playing with it - as well as report any issues/showstoppers Such an integrated solution would also make it possible to show any canvas texture (instrument, dialog/window, MFD etc) on external devices:

Navigation display centered MAP mode.png

FGCanvas[12]


Implementation

What I did now roughly is:

  • adding a DrawCallback to the canvas camera
  • attach an image to the canvas camera
  • duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera

This works so far and I can grab ND images from my browser by http like a screenshot. And as you mentioned before, there are latencies and it isn't efficient. Just creating the PNG image from OSG takes up to 100ms. Using an uncomressed bitmap format like tiff results in 3MB image sies. Assuming a frame rate of 5 will be sufficient for displaying smooth instruments (which I doubt) this results in 15 image creations per second (for Captains PFD, ND and Eicas).

However, it is working and I will use this approach for my first setup and for some long running tests for checking for memory leaks and other problems. Probably there will be some fine tuning required. And in the meantime I keep thinking about an external canvas drawing solution. Maybe I'll pursue the Nasal/Javascript approach, though my personal favorite is a Nasal/generic approach which allows implementing drawing in any environment like JavaScript/Python/Java aso.[13]

Gallery

Status (1/2019)

This feature was merged into the official 2018.3.1 release. It works so far. However, there are latencies and the solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms. For the time being, this should still be considered a "proof-of-concept" that may go through more iterations of optmizations. Integrating it into 2018.3.1 makes it possible to explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback. Special attention might be required with situations where the canvas and/or the connection are shut down, but also to support Reset & re-init without causing a race condition.

Note  In its current form, the streaming capability [16] is available but might overburden the flightgear process, depending on the number of canvases displayed. Its recommended not to use the streaming feature but to refresh the canvas from the browser. If any instability of FlightGear occurs, the remote canvas feature should be disabled for checking whether it is the reason.

Patches

Base Package

CanvasView.html

Note  For the time being, the following HTML snippet is specific to Soitanen's 737, and contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs}. Also, you need to edit httpd-settings.xml to add a corresponding canvasimage handler there. For a Canvas to become available for streaming, set the rendertoimage flag accordingly and assign a name to the top-level canvas.
<!DOCTYPE html>
<!-- See Flightgear wiki for information-->
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Canvas View</title>
</head>
<body>
<select id="sizebox" onChange="setSize();">
    <option value="256" selected="selected">256</option>
    <option value="512">512</option>
    <option value="768">768</option>
</select>
<table>
    <row>
        <td>
            <img id="CptPFD">
        </td>
        <td>
            <img id="CptND">
        </td>
    </row>
    <row>
        <td>
            <img id="UpperEICAS">
        </td>
        <td>
            <img id="CptCDU">
        </td>
    </row>
</table>

<script type="text/javascript">
var refreshinterval=500;
var instruments = ["CptPFD","CptND","UpperEICAS","CptCDU"];
var canvasindex = [];
canvasindex["CptPFD"] = 3;
canvasindex["CptND"] = 5;
canvasindex["UpperEICAS"] = 4;
canvasindex["CptCDU"] = 2;

function refreshImage(imageid) {
    //var image = document.getElementById(imageid);
    var image = document.querySelector("#"+imageid);    

    var url = "http://localhost:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://localhost:5701/canvasimage?type=png&canvasindex=5";
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex=5";
    //alert(url);
    image.src = url;
    setSize(imageid);
    setTimeout(function() {
         refreshImage(imageid);
     }, refreshinterval);
   
}

function setSize(imageid){
    var e = document.getElementById("sizebox");
    var size = e.options[e.selectedIndex].value;
    var image = document.querySelector("#"+imageid);
    if (image != null){
        image.style.width=size+"px";
        image.style.height="auto";
    }
}

for (var i = 0; i < instruments.length; i++) {
    refreshImage(instruments[i]);
}
//refreshImage("CptND");
//refreshImage("CptPFD");


</script>
</body>
</html>

Optimizations

  • Sample rate ~1-3 hz
  • JPEG_QUALITY <= 6
  • image size ~ 512x512
  • RTSP/ffmpeg streaming
  • Turn the whole thing into a configurable placement for capturing/streaming purposes
  • Howto:Serializing a Canvas to SVG (streaming a Canvas/MFD as a SVG/XML document)

Using

Add the line

<canvasimage>/canvasimage</canvasimage>

to the uri handler defined in file FG_ROOT/httpd-settings.xml

  <uri-handler>
                <screenshot>/screenshot</screenshot>
                <property>/props/</property>
                <json>/json/</json>
                <pkg>/pkg/</pkg>
                <flighthistory>/flighthistory/</flighthistory>
                <run>/run.cgi</run>
                <navdb>/navdb</navdb>
                <canvasimage>/canvasimage</canvasimage>
        </uri-handler>

and start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL

localhost:5701/canvasimage?type=png&canvasindex=0

in your browser. The first request will add the property rendertoimage to the canvas with index 0 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 0 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout).

Sample Showcase

The following showcase script CanvasView.html shows the main instruments of Soitanens 737 with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load.

Related

References

References