Read canvas image by HTTP: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
Line 171: Line 171:
Canvas-ND-served-via-httpd.png|Screenshot showing a FlightGear [[PUI]] GUI dialog with two independent [[NavDisplay]] instances streamed to FireFox
Canvas-ND-served-via-httpd.png|Screenshot showing a FlightGear [[PUI]] GUI dialog with two independent [[NavDisplay]] instances streamed to FireFox
Screenshot-streaming.png|Screenshot showing [[Canvas]] MFDs streamed to Firefox<ref>https://forum.flightgear.org/viewtopic.php?f=71&t=30642&p=297413#p297413</ref>
Screenshot-streaming.png|Screenshot showing [[Canvas]] MFDs streamed to Firefox<ref>https://forum.flightgear.org/viewtopic.php?f=71&t=30642&p=297413#p297413</ref>
Canvas-ND-live-streaming-via-httpd.png|thumb|Screenshot showing a single Canvas texture ([[NavDisplay]]) streamed to firefox via httpd <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=30642&p=297716#p297716</ref>
Canvas-ND-live-streaming-via-httpd.png|Screenshot showing a single Canvas texture ([[NavDisplay]]) streamed to firefox via httpd <ref>https://forum.flightgear.org/viewtopic.php?f=71&t=30642&p=297716#p297716</ref>
</gallery>
</gallery>



Revision as of 21:28, 29 October 2016

This article describes content/features that may not yet be available in the latest stable version of FlightGear (2020.3).
You may need to install some extra components, use the latest development (Git) version or even rebuild FlightGear from source, possibly from a custom topic branch using special build settings: .

This feature is scheduled for FlightGear (unknown). 10}% completed

If you'd like to learn more about getting your own ideas into FlightGear, check out Implementing new features for FlightGear.


Canvas/HTTP Server
Canvas-ND-served-via-httpd.png
Started in 10/2016
Description Serving Canvas texturues via httpd
Contributor(s) ThomasS (since 10/2016)
Status Under active development as of 10/2016

Objective

ND-Dialog-with-DisplayMode-support.png

There are situations, e.g., for home cockpit builders [1], where it is useful to display instruments like a PFD, ND, EICAS or any MFD externally from the FlightGear 3D main window in a separate window or on a separate monitor, computer or a mobile device [2] [3] [4].

Many of these avionics/graphics are created by FlightGear's 2D drawing Canvas system internally.


In addition there, are a number of other use-cases where being able to obtain a Canvas from fgfs using a network protocol like http may be desirable (e.g. imagine getting a tilemap based on actual scenery from FlightGear [5]) [6][7].


This article provides a patch to FlightGear for downloading any canvas image from a running FlightGear process by HTTP by serializing it to a raster image and serving that via the built-in mongoose based httpd server [8] [9].

This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.

Problem

a modern biz jet will need to be able to display certain MFDs - fgpanel is not too useful for that (unless you are primarily dealing with steam gauges) - the most useful search terms for the forum/wiki will be Phi and FGCanvas - the latter of which is a special startup mode of FlightGear itself that can render Canvas based MFDs in a separate instance.

If you don't need fancy MFDs, you could probably use fgfs standalone and/or fgpanel. Phi has the lowest barrier to entry probably for people familiar with HTML and JavaScript, i.e. you don't even need to know much about fgfs. The Canvas stuff will really only be needed if you want to render things like the 747 PFD/ND or CDU in a distributed fashion, i.e. using multiple independent displays.[10]

you should be aware of glass cockpit related efforts, especially Canvas - most airliners and jets will sooner or later benefit from being ported to Canvas, e.g. to use Gijs' NavDisplay framework, or at least Philosopher's MapStructure framework for mapping purposes.

Thus, if this is also about the actual display itself, people should be aware of related canvas efforts, especially FGCanvas: FGCanvas

The my mid-term plan involves supporting a standalone mode for all Canvas-based glass instruments, including the ND, but also other instruments like the PFD, EICAS, CDU or EFB. This may sound like a lot of work, but it's mainlyy a matter of introducing a a few helper classes and ensuring that people actually adopt and use those.

In the long-term, we really want to support distributed FlightGear setups like those at FSWeekend/LinuxTag, where multiple computers may be used to run a single simulator session - including properly synchronized glass instruments like the PFD/ND etc. This would also help improve the multiplayer experience, especially dual-pilot setups etc. Regarding a pure panel with switches, kobs and buttons, we'd prefer something like that to be aircraft-agnostic, i.e. just consist of property-mapped controls that can be individually assigned, so that this could be reused for other MFDs, not just the ND - but also the PFD or EFB.[11]

There's the long-term plan to eventually port FGPanel back into FG and come up with some sort of "FGCanvas" mode, where Canvas-based displays could be run in a separate "standalone" mode and interface to the main/master FG instance.

I don't think that the Canvas system currently builds against just OpenGL ES - but it should be possible to identify problematic use-cases and make the Canvas support OpenGL ES by setting some OSG traits accordingly - at some point, i.e. 12+ months from now In general, it pays off to unify things - otherwise, there's lots of duplication and re-invention involved.

Technically, we could definitely run an OpenGL ES-based Canvas on a mobile device, we just need people interested in pursuing this and playing with it - as well as report any issues/showstoppers Such an integrated solution would also make it possible to show any canvas texture (instrument, dialog/window, MFD etc) on external devices:

Navigation display centered MAP mode.png

FGCanvas[12]


Implementation

What I did now roughly is:

  • adding a DrawCallback to the canvas camera
  • attach an image to the canvas camera
  • duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera

This works so far and I can grab ND images from my browser by http like a screenshot. And as you mentioned before, there are latencies and it isn't efficient. Just creating the PNG image from OSG takes up to 100ms. Using an uncomressed bitmap format like tiff results in 3MB image sies. Assuming a frame rate of 5 will be sufficient for displaying smooth instruments (which I doubt) this results in 15 image creations per second (for Captains PFD, ND and Eicas).

However, it is working and I will use this approach for my first setup and for some long running tests for checking for memory leaks and other problems. Probably there will be some fine tuning required. And in the meantime I keep thinking about an external canvas drawing solution. Maybe I'll pursue the Nasal/Javascript approach, though my personal favorite is a Nasal/generic approach which allows implementing drawing in any environment like JavaScript/Python/Java aso.[13]

Gallery

Status (10/2016)

Note  In its current form, the streaming capability that is part of the original code has been disabled, i.e. is not available currently

This feature was created based on version 2016.3.1 (the screenshots were actually taken by applying the patches to 2016.4). It works so far. However, there are latencies and this solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms. For the time being, this should be considered - i.e. a "proof-of-concept" that may go through several iterations before this gets reviewed/integrated, but sharing this patch now makes it possible to also explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback.

Patches

SimGear

--- /simgear/simgear/canvas/Canvas.hxx	2016-05-17 02:40:01.297701130 +0200
+++ canvas/Canvas.hxx	2016-10-19 16:46:47.000000000 +0200
@@ -44,6 +44,14 @@
   class CanvasMgr;
   class MouseEvent;
 
+  class CanvasImageReadyListener {
+  public:
+    virtual void imageReady(osg::ref_ptr<osg::Image>) = 0;
+    virtual ~CanvasImageReadyListener()
+    {
+    }
+  };
+
   /**
    * Canvas to draw onto (to an off-screen render target).
    */
@@ -162,6 +170,9 @@
 
       void update(double delta_time_sec);
 
+      int subscribe(CanvasImageReadyListener * subscriber);
+      int unsubscribe(CanvasImageReadyListener * subscriber);
+
       bool addEventListener(const std::string& type, const EventListener& cb);
       bool dispatchEvent(const EventPtr& event);
--- /simgear/simgear/canvas/Canvas.cxx	2016-05-17 02:40:01.297701130 +0200
+++ Canvas.cxx	2016-10-20 08:04:14.000000000 +0200
@@ -37,6 +37,66 @@
 {
 namespace canvas
 {
+  class CanvasImageCallback : public osg::Camera::DrawCallback {
+  public:
+  osg::Image *_rawImage;
+
+  CanvasImageCallback(osg::Image *rawImage)
+  : _min_delta_tick(1.0 / 8.0) {
+    _previousFrameTick = osg::Timer::instance()->tick();
+    _rawImage = rawImage;
+  }
+
+  virtual void operator()(osg::RenderInfo& renderInfo/*const osg::Camera& camera*/) const {
+    osg::Timer_t n = osg::Timer::instance()->tick();
+    double dt = osg::Timer::instance()->delta_s(_previousFrameTick, n);
+    if (dt < _min_delta_tick)
+      return;
+    _previousFrameTick = n;
+
+    bool hasSubscribers = false;
+    {
+      OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+      hasSubscribers = !_subscribers.empty();
+    }
+    if (hasSubscribers) {
+      //Make sure image can be overwritten by next frame while it is still returned to the client
+      osg::Image* image = new osg::Image(*_rawImage, osg::CopyOp::DEEP_COPY_ALL);
+      {
+        OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+        while (!_subscribers.empty()) {
+          try {
+            SG_LOG(SG_GENERAL,SG_INFO,"CanvasImageCallback image ready for subscriber");
+            CanvasImageReadyListener *subs = _subscribers.back();
+            if (subs){
+              subs->imageReady(image);
+            }else{
+              SG_LOG(SG_GENERAL,SG_WARN,"CanvasImageCallback subscriber null");
+            }
+          } catch (...) { }
+          _subscribers.pop_back();
+        }
+      }
+    }
+  }
+
+  void subscribe(CanvasImageReadyListener * subscriber) {
+    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+    _subscribers.push_back(subscriber);
+  }
+
+  void unsubscribe(CanvasImageReadyListener * subscriber) {
+    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+    _subscribers.remove(subscriber);
+  }
+
+  private:
+    mutable list<CanvasImageReadyListener*> _subscribers;
+    mutable OpenThreads::Mutex _lock;
+    mutable double _previousFrameTick;
+    double _min_delta_tick;
+  };
+
 
   //----------------------------------------------------------------------------
   Canvas::CullCallback::CullCallback(const CanvasWeakPtr& canvas):
@@ -251,6 +311,22 @@
 
       osg::Camera* camera = _texture.getCamera();
 
+      const char *canvasname = _node->getStringValue("name");
+      int rendertoimage = _node->getBoolValue("rendertoimage");
+      if (camera && rendertoimage && canvasname) {
+        SG_LOG(SG_GENERAL,SG_INFO,"rendertoimage of canvas "<<  rendertoimage);
+
+        CanvasImageCallback *_screenshotCallback = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+        if (!_screenshotCallback) {
+          osg::Image* shot = new osg::Image();
+          shot->allocateImage(getSizeX(), getSizeY(), 24, GL_RGB, GL_UNSIGNED_BYTE);
+          camera->attach(osg::Camera::COLOR_BUFFER, shot);
+          camera->setFinalDrawCallback(new CanvasImageCallback(shot));
+          SG_LOG(SG_GENERAL,SG_INFO,"attached");
+        }
+      }
+
+
       // TODO Allow custom render order? For now just keep in order with
       //      property tree.
       camera->setRenderOrder(osg::Camera::PRE_RENDER, _node->getIndex());
@@ -350,6 +426,36 @@
     }
   }
 
+  int Canvas::subscribe(CanvasImageReadyListener * subscriber) {
+    if (!_node->getBoolValue("rendertoimage")) {
+      SG_LOG(SG_GENERAL,SG_INFO,"Setting rendertoimage");
+      _node->addChild("rendertoimage", 0)->setBoolValue(1);
+      setStatusFlags(STATUS_DIRTY, true);
+    }
+
+    osg::Camera* camera = _texture.getCamera();
+    SG_LOG(SG_GENERAL,SG_INFO,"Canvas: subscribe to camera "<< camera);
+    CanvasImageCallback *_screenshotCallback = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+    if (_screenshotCallback) {
+      SG_LOG(SG_GENERAL,SG_INFO,"Canvas: really subscribe to camera ");
+      _screenshotCallback->subscribe(subscriber);
+      // TODO: check: Is this the correct way to ensure the canvas will be available?
+      enableRendering(true);
+    }
+    return 0;
+  }
+
+  int Canvas::unsubscribe(CanvasImageReadyListener * subscriber) {
+    osg::Camera* camera = _texture.getCamera();
+    SG_LOG(SG_GENERAL,SG_INFO,"CanvasImage: unsubscribe from camera "<< camera);
+    CanvasImageCallback *cb = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+    if (cb) {
+      SG_LOG(SG_GENERAL,SG_INFO,"CanvasImage: unsubscribe ");
+      cb->unsubscribe(subscriber);
+    }
+    return 0;
+  }
+
   //----------------------------------------------------------------------------
   bool Canvas::addEventListener( const std::string& type,
                                  const EventListener& cb )

FlightGear

diff --git a/src/Canvas/canvas_mgr.cxx b/src/Canvas/canvas_mgr.cxx
index 6646b77..5316543 100644
--- a/src/Canvas/canvas_mgr.cxx
+++ b/src/Canvas/canvas_mgr.cxx
@@ -64,7 +64,7 @@ CanvasMgr::CanvasMgr():
     fgGetNode("/sim/signals/model-reinit", true)
   )
 {
-
+ SG_LOG(SG_GENERAL, SG_ALERT, "CanvasMgr() constructor invoked");
 }
 
 //----------------------------------------------------------------------------
diff --git a/src/Network/http/CMakeLists.txt b/src/Network/http/CMakeLists.txt
index 9e913d0..1a099cb 100644
--- a/src/Network/http/CMakeLists.txt
+++ b/src/Network/http/CMakeLists.txt
@@ -3,6 +3,7 @@ include(FlightGearComponent)
 set(SOURCES
 	httpd.cxx
 	ScreenshotUriHandler.cxx
+	CanvasImageUriHandler.cxx
 	PropertyUriHandler.cxx
 	JsonUriHandler.cxx
     FlightHistoryUriHandler.cxx
@@ -19,6 +20,7 @@ set(HEADERS
 	urihandler.hxx
 	httpd.hxx
 	ScreenshotUriHandler.hxx
+	CanvasImageUriHandler.hxx
 	PropertyUriHandler.hxx
 	JsonUriHandler.hxx
     FlightHistoryUriHandler.hxx
diff --git a/src/Network/http/CanvasImageUriHandler.cxx b/src/Network/http/CanvasImageUriHandler.cxx
new file mode 100644
index 0000000..7cd5a0e
--- /dev/null
+++ b/src/Network/http/CanvasImageUriHandler.cxx
@@ -0,0 +1,335 @@
+// CanvasImageUriHandler.cxx -- Provide canvasimages via http
+//
+// Started by Curtis Olson, started June 2001.
+// osg support written by James Turner
+// Ported to new httpd infrastructure by Torsten Dreyer
+// Derived from ScreenshotUrihandler originally by Torsten Dreyer)
+//
+// This program is free software; you can redistribute it and/or
+// modify it under the terms of the GNU General Public License as
+// published by the Free Software Foundation; either version 2 of the
+// License, or (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+#include "CanvasImageUriHandler.hxx"
+
+#ifdef _WIN32
+#include <windows.h>
+#endif
+#include <osgDB/Registry>
+#include <osgDB/ReaderWriter>
+#include <osgUtil/SceneView>
+#include <osgViewer/Viewer>
+
+#include <Canvas/canvas_mgr.hxx>
+#include <simgear/canvas/Canvas.hxx>
+
+#include <simgear/threads/SGQueue.hxx>
+#include <simgear/structure/Singleton.hxx>
+#include <Main/globals.hxx>
+#include <Viewer/renderer.hxx>
+
+#include <queue>
+#include <boost/lexical_cast.hpp>
+
+using std::string;
+using std::vector;
+using std::list;
+
+namespace sc = simgear::canvas;
+
+namespace flightgear {
+    namespace http {
+
+        ///////////////////////////////////////////////////////////////////////////
+
+        class StringReadyListener {
+        public:
+            virtual void stringReady(const std::string &) = 0;
+
+            virtual ~StringReadyListener() {
+            }
+        };
+
+        struct ImageCompressionTask {
+            StringReadyListener * stringReadyListener;
+            string format;
+            osg::ref_ptr<osg::Image> image;
+
+            ImageCompressionTask() {
+                stringReadyListener = NULL;
+            }
+
+            ImageCompressionTask(const ImageCompressionTask & other) {
+                stringReadyListener = other.stringReadyListener;
+                format = other.format;
+                image = other.image;
+            }
+
+            ImageCompressionTask & operator=(const ImageCompressionTask & other) {
+                stringReadyListener = other.stringReadyListener;
+                format = other.format;
+                image = other.image;
+                return *this;
+            }
+
+        };
+        //TODO reuse from screenshoturihandler
+
+        class ImageCompressorCI : public OpenThreads::Thread {
+        public:
+
+            ImageCompressorCI() {
+            }
+            virtual void run();
+            void addTask(ImageCompressionTask & task);
+        private:
+            typedef SGBlockingQueue<ImageCompressionTask> TaskList;
+            TaskList _tasks;
+        };
+
+        typedef simgear::Singleton<ImageCompressorCI> ImageCompressorCISingleton;
+
+        void ImageCompressorCI::run() {
+            osg::ref_ptr<osgDB::ReaderWriter::Options> options = new osgDB::ReaderWriter::Options("JPEG_QUALITY 80 PNG_COMPRESSION 9");
+
+            SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI is running");
+            for (;;) {
+                ImageCompressionTask task = _tasks.pop();
+                SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI has an image");
+
+                if (NULL != task.stringReadyListener) {
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI checking for writer for " << task.format);
+                    osgDB::ReaderWriter* writer = osgDB::Registry::instance()->getReaderWriterForExtension(task.format);
+                    if (!writer)
+                        continue;
+
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI compressing to " << task.format);
+                    std::stringstream outputStream;
+                    osgDB::ReaderWriter::WriteResult wr;
+                    wr = writer->writeImage(*task.image, outputStream, options);
+
+                    if (wr.success()) {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI compressed to  " << task.format);
+                        task.stringReadyListener->stringReady(outputStream.str());
+                    }
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI done for this image" << task.format);
+
+                }
+            }
+            SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI exiting");
+        }
+
+        void ImageCompressorCI::addTask(ImageCompressionTask & task) {
+            _tasks.push(task);
+        }
+
+
+        ///////////////////////////////////////////////////////////////////////////
+
+        class CanvasImageRequest : public ConnectionData, public simgear::canvas::CanvasImageReadyListener, StringReadyListener {
+        public:
+            ImageCompressionTask *currenttask=NULL;
+            sc::CanvasPtr canvas;
+
+            CanvasImageRequest(const string & window, const string & type, int canvasindex, bool stream)
+            : _type(type), _stream(stream) {
+                SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImageRequest: \n");
+                
+                if (NULL == osgDB::Registry::instance()->getReaderWriterForExtension(_type))
+                    throw sg_format_exception("Unsupported image type: " + type, type);
+
+                CanvasMgr* canvas_mgr = static_cast<CanvasMgr*> (globals->get_subsystem("Canvas"));
+                if (!canvas_mgr) {
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:CanvasMgr not found\n");
+                } else {
+                    canvas = canvas_mgr->getCanvas(canvasindex);
+                    if (!canvas) {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:Canvas not found\n");
+                    } else {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:Canvas found\n");
+                        //SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImageRequest: found camera %d. width=%d, height=%d\n", camera, canvas->getSizeX(), canvas->getSizeY());
+
+                        SGConstPropertyNode_ptr canvasnode = canvas->getProps();
+                        if (canvasnode) {
+                            const char *canvasname = canvasnode->getStringValue("name");
+                            if (canvasname) {
+                                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImageRequest: node=%s(%s)\n");//, canvasnode->getDisplayName().c_str(), canvasname);
+                            }
+                        }
+
+                        canvas->subscribe(this);
+                    }
+                }
+            }
+
+            // Assumption: when unsubscribe returns,there might just be a compressor thread running,
+            // causing a crash when the deconstructor finishes. Rare, but might happen. Just wait to be sure.
+            virtual ~CanvasImageRequest() {
+                if (currenttask){
+                    SG_LOG(SG_NETWORK, SG_INFO, "canvasimage task running");
+#ifdef _WIN32
+                    Sleep(15000);
+#else
+                     sleep(15);
+#endif
+                }
+
+                if (canvas){
+                    canvas->unsubscribe(this);
+                }
+                //_canvasimageCallback->unsubscribe(this);
+            }
+
+            virtual void imageReady(osg::ref_ptr<osg::Image> rawImage) {
+                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage:imageReady");
+                // called from a rendering thread, not from the main loop
+                ImageCompressionTask task;
+                currenttask = &task;
+                task.image = rawImage;
+                task.format = _type;
+                task.stringReadyListener = this;
+                ImageCompressorCISingleton::instance()->addTask(task);
+            }
+
+            void requestCanvasImage() {
+            //    _canvasimageCallback->subscribe(this);
+            }
+
+            mutable OpenThreads::Mutex _lock;
+
+            virtual void stringReady(const string & s) {
+                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage:stringReady");
+                
+                // called from the compressor thread
+                OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+                _compressedData = s;
+                // allow destructor
+                currenttask = NULL;
+            }
+
+            string getCanvasImage() {
+                string reply;
+                {
+                    // called from the main loop
+                    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+                    reply = _compressedData;
+                    _compressedData.clear();
+                }
+                return reply;
+            }
+
+            bool isStream() const {
+                return _stream;
+            }
+
+            const string & getType() const {
+                return _type;
+            }
+
+        private:
+            string _type;
+            bool _stream;
+            string _compressedData;
+            //CanvasImageCallback * _canvasimageCallback;
+        };
+
+        CanvasImageUriHandler::CanvasImageUriHandler(const char * uri)
+        : URIHandler(uri) {
+        }
+
+        CanvasImageUriHandler::~CanvasImageUriHandler() {
+            ImageCompressorCISingleton::instance()->cancel();
+            //ImageCompressorSingleton::instance()->join();
+        }
+
+        const static string KEY("CanvasImageUriHandler::CanvasImageRequest");
+#define BOUNDARY "--fgfs-canvasimage-boundary"
+
+        bool CanvasImageUriHandler::handleGetRequest(const HTTPRequest & request, HTTPResponse & response, Connection * connection) {
+            if (!ImageCompressorCISingleton::instance()->isRunning())
+                ImageCompressorCISingleton::instance()->start();
+
+            string type = request.RequestVariables.get("type");
+            if (type.empty()) type = "jpg";
+
+            string canvasindex = request.RequestVariables.get("canvasindex");
+            if (canvasindex.empty()) canvasindex = "0";
+            
+            //  string camera = request.RequestVariables.get("camera");
+            string window = request.RequestVariables.get("window");
+
+            bool stream = (false == request.RequestVariables.get("stream").empty());
+
+            SGSharedPtr<CanvasImageRequest> canvasimageRequest;
+            try {
+                SG_LOG(SG_NETWORK, SG_INFO, "new CanvasImageRequest(" << window << "," << type << "," << stream << ")");
+                canvasimageRequest = new CanvasImageRequest(window, type, atoi(canvasindex.c_str()),stream);
+            } catch (sg_format_exception & ex) {
+                SG_LOG(SG_NETWORK, SG_INFO, ex.getFormattedMessage());
+                response.Header["Content-Type"] = "text/plain";
+                response.StatusCode = 410;
+                response.Content = ex.getFormattedMessage();
+                return true;
+            } catch (sg_error & ex) {
+                SG_LOG(SG_NETWORK, SG_INFO, ex.getFormattedMessage());
+                response.Header["Content-Type"] = "text/plain";
+                response.StatusCode = 500;
+                response.Content = ex.getFormattedMessage();
+                return true;
+            }
+
+            if (false == stream) {
+                response.Header["Content-Type"] = string("image/").append(type);
+                response.Header["Content-Disposition"] = string("inline; filename=\"fgfs-canvasimage.").append(type).append("\"");
+            } else {
+                response.Header["Content-Type"] = string("multipart/x-mixed-replace; boundary=" BOUNDARY);
+
+            }
+
+            connection->put(KEY, canvasimageRequest);
+            return false; // call me again thru poll
+        }
+
+        bool CanvasImageUriHandler::poll(Connection * connection) {
+
+            SGSharedPtr<ConnectionData> data = connection->get(KEY);
+            CanvasImageRequest * canvasimageRequest = dynamic_cast<CanvasImageRequest*> (data.get());
+            if (NULL == canvasimageRequest) return true; // Should not happen, kill the connection
+
+            const string & canvasimage = canvasimageRequest->getCanvasImage();
+            if (canvasimage.empty()) {
+                SG_LOG(SG_NETWORK, SG_INFO, "No canvasimage available.");
+                return false; // not ready yet, call again.
+            }
+
+            SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage is ready, size=" << canvasimage.size());
+
+            if (canvasimageRequest->isStream()) {
+                string s(BOUNDARY "\r\nContent-Type: image/");
+                s.append(canvasimageRequest->getType()).append("\r\nContent-Length:");
+                s += boost::lexical_cast<string>(canvasimage.size());
+                s += "\r\n\r\n";
+                connection->write(s.c_str(), s.length());
+            }
+
+            connection->write(canvasimage.data(), canvasimage.size());
+
+            /* unknown purpose
+            if (canvasimageRequest->isStream()) {
+                canvasimageRequest->requestCanvasImage();
+                // continue until user closes connection
+                return false;
+            }
+            */
+
+            // single canvasimage, send terminating chunk
+            connection->remove(KEY);
+            connection->write("", 0);
+            return true; // done.
+        }
+
+    } // namespace http
+} // namespace flightgear
+
diff --git a/src/Network/http/CanvasImageUriHandler.hxx b/src/Network/http/CanvasImageUriHandler.hxx
new file mode 100644
index 0000000..f50be37
--- /dev/null
+++ b/src/Network/http/CanvasImageUriHandler.hxx
@@ -0,0 +1,40 @@
+// CanvasImageUriHandler.hxx -- Provide canvasimages via http
+//
+// Written by Torsten Dreyer, started April 2014.
+//
+// Copyright (C) 2014  Torsten Dreyer
+// derived from ScreenshotUrihandler by ThomasS
+// This program is free software; you can redistribute it and/or
+// modify it under the terms of the GNU General Public License as
+// published by the Free Software Foundation; either version 2 of the
+// License, or (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+#ifndef __FG_CANVASIMAGE_URI_HANDLER_HXX
+#define __FG_CANVASIMAGE_URI_HANDLER_HXX
+
+#include "urihandler.hxx"
+
+namespace flightgear {
+namespace http {
+
+class CanvasImageUriHandler : public URIHandler {
+public:
+  CanvasImageUriHandler( const char * uri = "/canvasimage/" );
+  ~CanvasImageUriHandler();
+  virtual bool handleGetRequest( const HTTPRequest & request, HTTPResponse & response, Connection * connection );
+  virtual bool poll( Connection * connection );
+};
+
+} // namespace http
+} // namespace flightgear
+
+#endif //#define __FG_CANVASIMAGE_URI_HANDLER_HXX
diff --git a/src/Network/http/httpd.cxx b/src/Network/http/httpd.cxx
index 358b247..3bfff3a 100644
--- a/src/Network/http/httpd.cxx
+++ b/src/Network/http/httpd.cxx
@@ -22,6 +22,7 @@
 #include "HTTPRequest.hxx"
 #include "PropertyChangeWebsocket.hxx"
 #include "ScreenshotUriHandler.hxx"
+#include "CanvasImageUriHandler.hxx"
 #include "PropertyUriHandler.hxx"
 #include "JsonUriHandler.hxx"
 #include "FlightHistoryUriHandler.hxx"
@@ -443,6 +444,11 @@ void MongooseHttpd::init()
       _uriHandler.push_back(new flightgear::http::ScreenshotUriHandler(uri));
     }
 
+if ((uri = n->getStringValue("canvasimage"))[0] != 0) {
+      SG_LOG(SG_NETWORK, SG_INFO, "httpd: adding canvasimage uri handler at " << uri);
+      _uriHandler.push_back(new flightgear::http::CanvasImageUriHandler(uri));
+    }
+
     if ((uri = n->getStringValue("property"))[0] != 0) {
       SG_LOG(SG_NETWORK, SG_INFO, "httpd: adding property uri handler at " << uri);
       _uriHandler.push_back(new flightgear::http::PropertyUriHandler(uri));

Base Package

CanvasView.html

Note  For the time being, the following HTML snippet is specific to Soitanen's 737, and contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs}. Also, you need to edit httpd-settings.xml to add a corresponding canvasimage handler there. For a Canvas to become available for streaming, set the rendertoimage flag accordingly and assign a name to the top-level canvas.
<!DOCTYPE html>
<!-- See Flightgear wiki for information-->
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Canvas View</title>
</head>
<body>
<select id="sizebox" onChange="setSize();">
    <option value="256" selected="selected">256</option>
    <option value="512">512</option>
    <option value="768">768</option>
</select>
<table>
    <row>
        <td>
            <img id="CptPFD">
        </td>
        <td>
            <img id="CptND">
        </td>
    </row>
    <row>
        <td>
            <img id="UpperEICAS">
        </td>
        <td>
            <img id="CptCDU">
        </td>
    </row>
</table>

<script type="text/javascript">
var refreshinterval=500;
var instruments = ["CptPFD","CptND","UpperEICAS","CptCDU"];
var canvasindex = [];
canvasindex["CptPFD"] = 3;
canvasindex["CptND"] = 5;
canvasindex["UpperEICAS"] = 4;
canvasindex["CptCDU"] = 2;

function refreshImage(imageid) {
    //var image = document.getElementById(imageid);
    var image = document.querySelector("#"+imageid);    

    var url = "http://localhost:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://localhost:5701/canvasimage?type=png&canvasindex=5";
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex=5";
    //alert(url);
    image.src = url;
    setSize(imageid);
    setTimeout(function() {
         refreshImage(imageid);
     }, refreshinterval);
   
}

function setSize(imageid){
    var e = document.getElementById("sizebox");
    var size = e.options[e.selectedIndex].value;
    var image = document.querySelector("#"+imageid);
    if (image != null){
        image.style.width=size+"px";
        image.style.height="auto";
    }
}

for (var i = 0; i < instruments.length; i++) {
    refreshImage(instruments[i]);
}
//refreshImage("CptND");
//refreshImage("CptPFD");


</script>
</body>
</html>

Building

  • Download and extract the latest attached tar archive (https://sourceforge.net/projects/remote-canvas-for-flightgear). It contains the patches listed above,
  • Copy the simgear and flightgear subfolders over your existing source tree.
  • Run the platform specific make for building FG.
  • Copy the resulting fgfs executable to your FlightGears installation bin directory after saving the existing.

Optimizations

  • Sample rate ~1-3 hz
  • JPEG_QUALITY <= 6
  • image size ~ 512x512
  • RTSP/ffmpeg streaming
  • Turn the whole thing into a configurable placement for capturing/streaming purposes

Using

Add the line

<canvasimage>/canvasimage</canvasimage>

to the uri handler defined in file FG_ROOT/httpd-settings.xml and start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL

localhost:5701/canvasimage?type=png&canvasindex=0

in your browser. The first request will add the property rendertoimage to the canvas with index 0 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 0 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout).

The showcase script CanvasView.html shows the main instruments of Soitanens 737 with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load.


Related

References

References