Read canvas image by HTTP: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
No edit summary
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Non-stable}}


{{infobox subsystem
{{infobox subsystem
Line 6: Line 4:
|name = Canvas/HTTP Server
|name = Canvas/HTTP Server
|started= 10/2016  
|started= 10/2016  
|description = Serving Canvas texturues via httpd  
|description = Serving Canvas textures via httpd  
|status = Under active development as of 10/2016
|status = Available as of 2018.3.1 (still might cause stability problems)
|developers =  ThomasS (since 10/2016)  
|developers =  ThomasS (since 10/2016)  
<!--
<!--
Line 96: Line 94:


This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.
This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.
An additional option for displaying canvas images on remote computer is [[FGQCanvas]], which doesn't require to patch the core FlightGear code.


== Problem ==
== Problem ==
Line 147: Line 147:


== Implementation ==
== Implementation ==
What I did now roughly is:
This solution was created roughly by:
*adding a DrawCallback to the canvas camera
*adding a DrawCallback to the canvas camera
*attach an image to the canvas camera
*attach an image to the canvas camera
*duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera
*duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera


This works so far and I can grab ND images from my browser by http like a screenshot. And as you mentioned before, there are latencies and it isn't efficient. Just creating the PNG image from OSG takes up to 100ms. Using an uncomressed bitmap format like tiff results in 3MB image sies. Assuming a frame rate of 5 will be sufficient for displaying smooth instruments (which I doubt) this results in 15 image creations per second (for Captains PFD, ND and Eicas).
== Alternatives ==
 
There are possible alternative solutions for rendering canvases on a remote system, that remove the rendering load from the main flightgear process. These might be based on Nasal, Javascript, Python, etc.<ref>{{cite web
However, it is working and I will use this approach for my first setup and for some long running tests for checking for memory leaks and other problems. Probably there will be some fine tuning required.
And in the meantime I keep thinking about an external canvas drawing solution. Maybe I'll pursue the Nasal/Javascript approach, though my personal favorite is a Nasal/generic approach which allows implementing drawing in any environment like JavaScript/Python/Java aso.<ref>{{cite web
   |url    =  https://forum.flightgear.org/viewtopic.php?p=296627#p296627  
   |url    =  https://forum.flightgear.org/viewtopic.php?p=296627#p296627  
   |title  =  <nowiki> Re: Canvas remote drawing </nowiki>  
   |title  =  <nowiki> Re: Canvas remote drawing </nowiki>  
Line 163: Line 161:
   |script_version = 0.40  
   |script_version = 0.40  
   }}</ref>
   }}</ref>
One currently available solution is [[FGQCanvas]], created by James.


== Gallery ==
== Gallery ==
Line 174: Line 174:
</gallery>
</gallery>


== Status (10/2016) ==
== Status (1/2019) ==
{{Note|In its current form, the streaming capability that is part of the original code has been disabled, i.e. is not available currently <ref>https://forum.flightgear.org/viewtopic.php?f=71&p=297716#p297716</ref>}}
This feature was merged into the official 2018.3.1 release. It works so far. However, there are latencies and the solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms.
This feature was created based on version 2016.3.1 (the screenshots were actually taken by applying the patches to 2016.4). It works so far. However, there are latencies and this solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms.
For the time being, this should still be considered a "proof-of-concept" that may go through more iterations of optmizations. Integrating it into 2018.3.1 makes it possible to explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback. Special attention might be required with situations where the canvas and/or the connection are shut down, but also to support [[Reset & re-init]] without causing a race  condition.
For the time being, this should be considered - i.e. a "proof-of-concept" that may go through several iterations before this gets reviewed/integrated, but sharing this patch now makes it possible to also explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback.


== Patches ==
{{Note|In its current form, the streaming capability <ref>https://forum.flightgear.org/viewtopic.php?f=71&p=297716#p297716</ref> is available but might overburden the flightgear process, depending on the number of canvases displayed. Its recommended not to use the streaming feature but to refresh the canvas from the browser. If any instability of FlightGear occurs, the remote canvas feature should be disabled for checking whether it is the reason.
=== SimGear ===
  }}
<syntaxhighlight lang="diff">
--- /simgear/simgear/canvas/Canvas.hxx 2016-05-17 02:40:01.297701130 +0200
+++ canvas/Canvas.hxx 2016-10-19 16:46:47.000000000 +0200
@@ -44,6 +44,14 @@
  class CanvasMgr;
  class MouseEvent;
+  class CanvasImageReadyListener {
+  public:
+    virtual void imageReady(osg::ref_ptr<osg::Image>) = 0;
+    virtual ~CanvasImageReadyListener()
+    {
+    }
+  };
+
  /**
    * Canvas to draw onto (to an off-screen render target).
    */
@@ -162,6 +170,9 @@
   
      void update(double delta_time_sec);
+      int subscribe(CanvasImageReadyListener * subscriber);
+      int unsubscribe(CanvasImageReadyListener * subscriber);
+
      bool addEventListener(const std::string& type, const EventListener& cb);
      bool dispatchEvent(const EventPtr& event);
</syntaxhighlight>


<syntaxhighlight lang="diff">
== Optimizations ==
--- /simgear/simgear/canvas/Canvas.cxx 2016-05-17 02:40:01.297701130 +0200
* Sample rate ~1-3 hz
+++ Canvas.cxx 2016-10-20 08:04:14.000000000 +0200
* JPEG_QUALITY <= 6
@@ -37,6 +37,66 @@
* image size ~ 512x512
{
* RTSP/ffmpeg streaming
namespace canvas
* Turn the whole thing into a configurable placement for capturing/streaming purposes
{
* [[Howto:Serializing a Canvas to SVG]] (streaming a Canvas/MFD as a SVG/XML document)
+  class CanvasImageCallback : public osg::Camera::DrawCallback {
+  public:
+  osg::Image *_rawImage;
+
+  CanvasImageCallback(osg::Image *rawImage)
+  : _min_delta_tick(1.0 / 8.0) {
+    _previousFrameTick = osg::Timer::instance()->tick();
+    _rawImage = rawImage;
+  }
+
+  virtual void operator()(osg::RenderInfo& renderInfo/*const osg::Camera& camera*/) const {
+    osg::Timer_t n = osg::Timer::instance()->tick();
+    double dt = osg::Timer::instance()->delta_s(_previousFrameTick, n);
+    if (dt < _min_delta_tick)
+      return;
+    _previousFrameTick = n;
+
+    bool hasSubscribers = false;
+    {
+      OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+      hasSubscribers = !_subscribers.empty();
+    }
+    if (hasSubscribers) {
+      //Make sure image can be overwritten by next frame while it is still returned to the client
+      osg::Image* image = new osg::Image(*_rawImage, osg::CopyOp::DEEP_COPY_ALL);
+      {
+        OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+        while (!_subscribers.empty()) {
+          try {
+            SG_LOG(SG_GENERAL,SG_INFO,"CanvasImageCallback image ready for subscriber");
+            CanvasImageReadyListener *subs = _subscribers.back();
+            if (subs){
+              subs->imageReady(image);
+            }else{
+              SG_LOG(SG_GENERAL,SG_WARN,"CanvasImageCallback subscriber null");
+            }
+          } catch (...) { }
+          _subscribers.pop_back();
+        }
+      }
+    }
+  }
+
+  void subscribe(CanvasImageReadyListener * subscriber) {
+    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+    _subscribers.push_back(subscriber);
+  }
+
+  void unsubscribe(CanvasImageReadyListener * subscriber) {
+    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+    _subscribers.remove(subscriber);
+  }
+
+  private:
+    mutable list<CanvasImageReadyListener*> _subscribers;
+    mutable OpenThreads::Mutex _lock;
+    mutable double _previousFrameTick;
+    double _min_delta_tick;
+  };
+
  //----------------------------------------------------------------------------
  Canvas::CullCallback::CullCallback(const CanvasWeakPtr& canvas):
@@ -251,6 +311,22 @@
      osg::Camera* camera = _texture.getCamera();
+      const char *canvasname = _node->getStringValue("name");
+      int rendertoimage = _node->getBoolValue("rendertoimage");
+      if (camera && rendertoimage && canvasname) {
+        SG_LOG(SG_GENERAL,SG_INFO,"rendertoimage of canvas "<<  rendertoimage);
+
+        CanvasImageCallback *_screenshotCallback = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+        if (!_screenshotCallback) {
+          osg::Image* shot = new osg::Image();
+          shot->allocateImage(getSizeX(), getSizeY(), 24, GL_RGB, GL_UNSIGNED_BYTE);
+          camera->attach(osg::Camera::COLOR_BUFFER, shot);
+          camera->setFinalDrawCallback(new CanvasImageCallback(shot));
+          SG_LOG(SG_GENERAL,SG_INFO,"attached");
+        }
+      }
+
+
      // TODO Allow custom render order? For now just keep in order with
      //      property tree.
      camera->setRenderOrder(osg::Camera::PRE_RENDER, _node->getIndex());
@@ -350,6 +426,36 @@
    }
  }
+  int Canvas::subscribe(CanvasImageReadyListener * subscriber) {
+    if (!_node->getBoolValue("rendertoimage")) {
+      SG_LOG(SG_GENERAL,SG_INFO,"Setting rendertoimage");
+      _node->addChild("rendertoimage", 0)->setBoolValue(1);
+      setStatusFlags(STATUS_DIRTY, true);
+    }
+
+    osg::Camera* camera = _texture.getCamera();
+    SG_LOG(SG_GENERAL,SG_INFO,"Canvas: subscribe to camera "<< camera);
+    CanvasImageCallback *_screenshotCallback = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+    if (_screenshotCallback) {
+      SG_LOG(SG_GENERAL,SG_INFO,"Canvas: really subscribe to camera ");
+      _screenshotCallback->subscribe(subscriber);
+      // TODO: check: Is this the correct way to ensure the canvas will be available?
+      enableRendering(true);
+    }
+    return 0;
+  }
+
+  int Canvas::unsubscribe(CanvasImageReadyListener * subscriber) {
+    osg::Camera* camera = _texture.getCamera();
+    SG_LOG(SG_GENERAL,SG_INFO,"CanvasImage: unsubscribe from camera "<< camera);
+    CanvasImageCallback *cb = dynamic_cast<CanvasImageCallback*> (camera->getFinalDrawCallback());
+    if (cb) {
+      SG_LOG(SG_GENERAL,SG_INFO,"CanvasImage: unsubscribe ");
+      cb->unsubscribe(subscriber);
+    }
+    return 0;
+  }
+
  //----------------------------------------------------------------------------
  bool Canvas::addEventListener( const std::string& type,
                                  const EventListener& cb )


== Using ==
Start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL
<syntaxhighlight>
http://localhost:5701/screenshot?canvasindex=4&type=png
</syntaxhighlight>
</syntaxhighlight>
in your browser. The first request will add the property rendertoimage to the canvas with index 4 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 4 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout). For the 777-200 canvas index 4 is the ND, 5 the MFD. PFD seems not to use Canvas.


=== FlightGear ===
=== Sample Showcase CanvasView.html ===
<syntaxhighlight lang="diff">
The following showcase HTML snippet CanvasView.html shows the main instruments of Soitanens 737 (and its branches) with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load. The script contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs.
diff --git a/src/Canvas/canvas_mgr.cxx b/src/Canvas/canvas_mgr.cxx
index 6646b77..5316543 100644
--- a/src/Canvas/canvas_mgr.cxx
+++ b/src/Canvas/canvas_mgr.cxx
@@ -64,7 +64,7 @@ CanvasMgr::CanvasMgr():
    fgGetNode("/sim/signals/model-reinit", true)
  )
{
-
+ SG_LOG(SG_GENERAL, SG_ALERT, "CanvasMgr() constructor invoked");
}
   
//----------------------------------------------------------------------------
diff --git a/src/Network/http/CMakeLists.txt b/src/Network/http/CMakeLists.txt
index 9e913d0..1a099cb 100644
--- a/src/Network/http/CMakeLists.txt
+++ b/src/Network/http/CMakeLists.txt
@@ -3,6 +3,7 @@ include(FlightGearComponent)
set(SOURCES
httpd.cxx
ScreenshotUriHandler.cxx
+ CanvasImageUriHandler.cxx
PropertyUriHandler.cxx
JsonUriHandler.cxx
    FlightHistoryUriHandler.cxx
@@ -19,6 +20,7 @@ set(HEADERS
urihandler.hxx
httpd.hxx
ScreenshotUriHandler.hxx
+ CanvasImageUriHandler.hxx
PropertyUriHandler.hxx
JsonUriHandler.hxx
    FlightHistoryUriHandler.hxx
diff --git a/src/Network/http/CanvasImageUriHandler.cxx b/src/Network/http/CanvasImageUriHandler.cxx
new file mode 100644
index 0000000..7cd5a0e
--- /dev/null
+++ b/src/Network/http/CanvasImageUriHandler.cxx
@@ -0,0 +1,335 @@
+// CanvasImageUriHandler.cxx -- Provide canvasimages via http
+//
+// Started by Curtis Olson, started June 2001.
+// osg support written by James Turner
+// Ported to new httpd infrastructure by Torsten Dreyer
+// Derived from ScreenshotUrihandler originally by Torsten Dreyer)
+//
+// This program is free software; you can redistribute it and/or
+// modify it under the terms of the GNU General Public License as
+// published by the Free Software Foundation; either version 2 of the
+// License, or (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+#include "CanvasImageUriHandler.hxx"
+
+#ifdef _WIN32
+#include <windows.h>
+#endif
+#include <osgDB/Registry>
+#include <osgDB/ReaderWriter>
+#include <osgUtil/SceneView>
+#include <osgViewer/Viewer>
+
+#include <Canvas/canvas_mgr.hxx>
+#include <simgear/canvas/Canvas.hxx>
+
+#include <simgear/threads/SGQueue.hxx>
+#include <simgear/structure/Singleton.hxx>
+#include <Main/globals.hxx>
+#include <Viewer/renderer.hxx>
+
+#include <queue>
+#include <boost/lexical_cast.hpp>
+
+using std::string;
+using std::vector;
+using std::list;
+
+namespace sc = simgear::canvas;
+
+namespace flightgear {
+    namespace http {
+
+        ///////////////////////////////////////////////////////////////////////////
+
+        class StringReadyListener {
+        public:
+            virtual void stringReady(const std::string &) = 0;
+
+            virtual ~StringReadyListener() {
+            }
+        };
+
+        struct ImageCompressionTask {
+            StringReadyListener * stringReadyListener;
+            string format;
+            osg::ref_ptr<osg::Image> image;
+
+            ImageCompressionTask() {
+                stringReadyListener = NULL;
+            }
+
+            ImageCompressionTask(const ImageCompressionTask & other) {
+                stringReadyListener = other.stringReadyListener;
+                format = other.format;
+                image = other.image;
+            }
+
+            ImageCompressionTask & operator=(const ImageCompressionTask & other) {
+                stringReadyListener = other.stringReadyListener;
+                format = other.format;
+                image = other.image;
+                return *this;
+            }
+
+        };
+        //TODO reuse from screenshoturihandler
+
+        class ImageCompressorCI : public OpenThreads::Thread {
+        public:
+
+            ImageCompressorCI() {
+            }
+            virtual void run();
+            void addTask(ImageCompressionTask & task);
+        private:
+            typedef SGBlockingQueue<ImageCompressionTask> TaskList;
+            TaskList _tasks;
+        };
+
+        typedef simgear::Singleton<ImageCompressorCI> ImageCompressorCISingleton;
+
+        void ImageCompressorCI::run() {
+            osg::ref_ptr<osgDB::ReaderWriter::Options> options = new osgDB::ReaderWriter::Options("JPEG_QUALITY 80 PNG_COMPRESSION 9");
+
+            SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI is running");
+            for (;;) {
+                ImageCompressionTask task = _tasks.pop();
+                SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI has an image");
+
+                if (NULL != task.stringReadyListener) {
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI checking for writer for " << task.format);
+                    osgDB::ReaderWriter* writer = osgDB::Registry::instance()->getReaderWriterForExtension(task.format);
+                    if (!writer)
+                        continue;
+
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI compressing to " << task.format);
+                    std::stringstream outputStream;
+                    osgDB::ReaderWriter::WriteResult wr;
+                    wr = writer->writeImage(*task.image, outputStream, options);
+
+                    if (wr.success()) {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI compressed to  " << task.format);
+                        task.stringReadyListener->stringReady(outputStream.str());
+                    }
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI done for this image" << task.format);
+
+                }
+            }
+            SG_LOG(SG_NETWORK, SG_DEBUG, "ImageCompressorCI exiting");
+        }
+
+        void ImageCompressorCI::addTask(ImageCompressionTask & task) {
+            _tasks.push(task);
+        }
+
+
+        ///////////////////////////////////////////////////////////////////////////
+
+        class CanvasImageRequest : public ConnectionData, public simgear::canvas::CanvasImageReadyListener, StringReadyListener {
+        public:
+            ImageCompressionTask *currenttask=NULL;
+            sc::CanvasPtr canvas;
+
+            CanvasImageRequest(const string & window, const string & type, int canvasindex, bool stream)
+            : _type(type), _stream(stream) {
+                SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImageRequest: \n");
+               
+                if (NULL == osgDB::Registry::instance()->getReaderWriterForExtension(_type))
+                    throw sg_format_exception("Unsupported image type: " + type, type);
+
+                CanvasMgr* canvas_mgr = static_cast<CanvasMgr*> (globals->get_subsystem("Canvas"));
+                if (!canvas_mgr) {
+                    SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:CanvasMgr not found\n");
+                } else {
+                    canvas = canvas_mgr->getCanvas(canvasindex);
+                    if (!canvas) {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:Canvas not found\n");
+                    } else {
+                        SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImage:Canvas found\n");
+                        //SG_LOG(SG_NETWORK, SG_DEBUG, "CanvasImageRequest: found camera %d. width=%d, height=%d\n", camera, canvas->getSizeX(), canvas->getSizeY());
+
+                        SGConstPropertyNode_ptr canvasnode = canvas->getProps();
+                        if (canvasnode) {
+                            const char *canvasname = canvasnode->getStringValue("name");
+                            if (canvasname) {
+                                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImageRequest: node=%s(%s)\n");//, canvasnode->getDisplayName().c_str(), canvasname);
+                            }
+                        }
+
+                        canvas->subscribe(this);
+                    }
+                }
+            }
+
+            // Assumption: when unsubscribe returns,there might just be a compressor thread running,
+            // causing a crash when the deconstructor finishes. Rare, but might happen. Just wait to be sure.
+            virtual ~CanvasImageRequest() {
+                if (currenttask){
+                    SG_LOG(SG_NETWORK, SG_INFO, "canvasimage task running");
+#ifdef _WIN32
+                    Sleep(15000);
+#else
+                    sleep(15);
+#endif
+                }
+
+                if (canvas){
+                    canvas->unsubscribe(this);
+                }
+                //_canvasimageCallback->unsubscribe(this);
+            }
+
+            virtual void imageReady(osg::ref_ptr<osg::Image> rawImage) {
+                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage:imageReady");
+                // called from a rendering thread, not from the main loop
+                ImageCompressionTask task;
+                currenttask = &task;
+                task.image = rawImage;
+                task.format = _type;
+                task.stringReadyListener = this;
+                ImageCompressorCISingleton::instance()->addTask(task);
+            }
+
+            void requestCanvasImage() {
+            //    _canvasimageCallback->subscribe(this);
+            }
+
+            mutable OpenThreads::Mutex _lock;
+
+            virtual void stringReady(const string & s) {
+                SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage:stringReady");
+               
+                // called from the compressor thread
+                OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+                _compressedData = s;
+                // allow destructor
+                currenttask = NULL;
+            }
+
+            string getCanvasImage() {
+                string reply;
+                {
+                    // called from the main loop
+                    OpenThreads::ScopedLock<OpenThreads::Mutex> lock(_lock);
+                    reply = _compressedData;
+                    _compressedData.clear();
+                }
+                return reply;
+            }
+
+            bool isStream() const {
+                return _stream;
+            }
+
+            const string & getType() const {
+                return _type;
+            }
+
+        private:
+            string _type;
+            bool _stream;
+            string _compressedData;
+            //CanvasImageCallback * _canvasimageCallback;
+        };
+
+        CanvasImageUriHandler::CanvasImageUriHandler(const char * uri)
+        : URIHandler(uri) {
+        }
+
+        CanvasImageUriHandler::~CanvasImageUriHandler() {
+            ImageCompressorCISingleton::instance()->cancel();
+            //ImageCompressorSingleton::instance()->join();
+        }
+
+        const static string KEY("CanvasImageUriHandler::CanvasImageRequest");
+#define BOUNDARY "--fgfs-canvasimage-boundary"
+
+        bool CanvasImageUriHandler::handleGetRequest(const HTTPRequest & request, HTTPResponse & response, Connection * connection) {
+            if (!ImageCompressorCISingleton::instance()->isRunning())
+                ImageCompressorCISingleton::instance()->start();
+
+            string type = request.RequestVariables.get("type");
+            if (type.empty()) type = "jpg";
+
+            string canvasindex = request.RequestVariables.get("canvasindex");
+            if (canvasindex.empty()) canvasindex = "0";
+           
+            //  string camera = request.RequestVariables.get("camera");
+            string window = request.RequestVariables.get("window");
+
+            bool stream = (false == request.RequestVariables.get("stream").empty());
+
+            SGSharedPtr<CanvasImageRequest> canvasimageRequest;
+            try {
+                SG_LOG(SG_NETWORK, SG_INFO, "new CanvasImageRequest(" << window << "," << type << "," << stream << ")");
+                canvasimageRequest = new CanvasImageRequest(window, type, atoi(canvasindex.c_str()),stream);
+            } catch (sg_format_exception & ex) {
+                SG_LOG(SG_NETWORK, SG_INFO, ex.getFormattedMessage());
+                response.Header["Content-Type"] = "text/plain";
+                response.StatusCode = 410;
+                response.Content = ex.getFormattedMessage();
+                return true;
+            } catch (sg_error & ex) {
+                SG_LOG(SG_NETWORK, SG_INFO, ex.getFormattedMessage());
+                response.Header["Content-Type"] = "text/plain";
+                response.StatusCode = 500;
+                response.Content = ex.getFormattedMessage();
+                return true;
+            }
+
+            if (false == stream) {
+                response.Header["Content-Type"] = string("image/").append(type);
+                response.Header["Content-Disposition"] = string("inline; filename=\"fgfs-canvasimage.").append(type).append("\"");
+            } else {
+                response.Header["Content-Type"] = string("multipart/x-mixed-replace; boundary=" BOUNDARY);
+
+            }
+
+            connection->put(KEY, canvasimageRequest);
+            return false; // call me again thru poll
+        }
+
+        bool CanvasImageUriHandler::poll(Connection * connection) {
+
+            SGSharedPtr<ConnectionData> data = connection->get(KEY);
+            CanvasImageRequest * canvasimageRequest = dynamic_cast<CanvasImageRequest*> (data.get());
+            if (NULL == canvasimageRequest) return true; // Should not happen, kill the connection
+
+            const string & canvasimage = canvasimageRequest->getCanvasImage();
+            if (canvasimage.empty()) {
+                SG_LOG(SG_NETWORK, SG_INFO, "No canvasimage available.");
+                return false; // not ready yet, call again.
+            }
+
+            SG_LOG(SG_NETWORK, SG_INFO, "CanvasImage is ready, size=" << canvasimage.size());
+
+            if (canvasimageRequest->isStream()) {
+                string s(BOUNDARY "\r\nContent-Type: image/");
+                s.append(canvasimageRequest->getType()).append("\r\nContent-Length:");
+                s += boost::lexical_cast<string>(canvasimage.size());
+                s += "\r\n\r\n";
+                connection->write(s.c_str(), s.length());
+            }
+
+            connection->write(canvasimage.data(), canvasimage.size());
+
+            /* unknown purpose
+            if (canvasimageRequest->isStream()) {
+                canvasimageRequest->requestCanvasImage();
+                // continue until user closes connection
+                return false;
+            }
+            */
+
+            // single canvasimage, send terminating chunk
+            connection->remove(KEY);
+            connection->write("", 0);
+            return true; // done.
+        }
+
+    } // namespace http
+} // namespace flightgear
+
diff --git a/src/Network/http/CanvasImageUriHandler.hxx b/src/Network/http/CanvasImageUriHandler.hxx
new file mode 100644
index 0000000..f50be37
--- /dev/null
+++ b/src/Network/http/CanvasImageUriHandler.hxx
@@ -0,0 +1,40 @@
+// CanvasImageUriHandler.hxx -- Provide canvasimages via http
+//
+// Written by Torsten Dreyer, started April 2014.
+//
+// Copyright (C) 2014  Torsten Dreyer
+// derived from ScreenshotUrihandler by ThomasS
+// This program is free software; you can redistribute it and/or
+// modify it under the terms of the GNU General Public License as
+// published by the Free Software Foundation; either version 2 of the
+// License, or (at your option) any later version.
+//
+// This program is distributed in the hope that it will be useful, but
+// WITHOUT ANY WARRANTY; without even the implied warranty of
+// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+// General Public License for more details.
+//
+// You should have received a copy of the GNU General Public License
+// along with this program; if not, write to the Free Software
+// Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
+
+#ifndef __FG_CANVASIMAGE_URI_HANDLER_HXX
+#define __FG_CANVASIMAGE_URI_HANDLER_HXX
+
+#include "urihandler.hxx"
+
+namespace flightgear {
+namespace http {
+
+class CanvasImageUriHandler : public URIHandler {
+public:
+  CanvasImageUriHandler( const char * uri = "/canvasimage/" );
+  ~CanvasImageUriHandler();
+  virtual bool handleGetRequest( const HTTPRequest & request, HTTPResponse & response, Connection * connection );
+  virtual bool poll( Connection * connection );
+};
+
+} // namespace http
+} // namespace flightgear
+
+#endif //#define __FG_CANVASIMAGE_URI_HANDLER_HXX
diff --git a/src/Network/http/httpd.cxx b/src/Network/http/httpd.cxx
index 358b247..3bfff3a 100644
--- a/src/Network/http/httpd.cxx
+++ b/src/Network/http/httpd.cxx
@@ -22,6 +22,7 @@
#include "HTTPRequest.hxx"
#include "PropertyChangeWebsocket.hxx"
#include "ScreenshotUriHandler.hxx"
+#include "CanvasImageUriHandler.hxx"
#include "PropertyUriHandler.hxx"
#include "JsonUriHandler.hxx"
#include "FlightHistoryUriHandler.hxx"
@@ -443,6 +444,11 @@ void MongooseHttpd::init()
      _uriHandler.push_back(new flightgear::http::ScreenshotUriHandler(uri));
    }
+if ((uri = n->getStringValue("canvasimage"))[0] != 0) {
+      SG_LOG(SG_NETWORK, SG_INFO, "httpd: adding canvasimage uri handler at " << uri);
+      _uriHandler.push_back(new flightgear::http::CanvasImageUriHandler(uri));
+    }
+
    if ((uri = n->getStringValue("property"))[0] != 0) {
      SG_LOG(SG_NETWORK, SG_INFO, "httpd: adding property uri handler at " << uri);
      _uriHandler.push_back(new flightgear::http::PropertyUriHandler(uri));
 
</syntaxhighlight>


=== Base Package ===
CanvasView.html


{{Note|For the time being, the following HTML snippet is specific to Soitanen's 737, and contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs}. Also, you need to edit httpd-settings.xml to add a corresponding canvasimage handler there. For a Canvas to become available for streaming, set the <code>rendertoimage</code> flag accordingly and assign a name to the top-level canvas.}}
<syntaxhighlight lang="html">
<syntaxhighlight lang="html">
<!DOCTYPE html>
<!DOCTYPE html>
Line 849: Line 246:
     var image = document.querySelector("#"+imageid);     
     var image = document.querySelector("#"+imageid);     


     var url = "http://localhost:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
     var url = "http://localhost:5701/screenshot?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex="+canvasindex[imageid];
    //var url = "http://localhost:5701/canvasimage?type=png&canvasindex=5";
    //var url = "http://192.168.98.165:5701/canvasimage?type=png&canvasindex=5";
    //alert(url);
     image.src = url;
     image.src = url;
     setSize(imageid);
     setSize(imageid);
Line 884: Line 277:
</syntaxhighlight>
</syntaxhighlight>


== Building ==
== Discussion ==
 
The forum thread for discussing this feature is https://forum.flightgear.org/viewtopic.php?f=71&t=30642&start=15.
* Download and extract the latest attached tar archive (https://sourceforge.net/projects/remote-canvas-for-flightgear). It contains the patches listed above,
* Copy the simgear and flightgear subfolders over your existing source tree.
* Run the platform specific make for building FG.
* Copy the resulting fgfs executable to your FlightGears installation bin directory '''after''' saving the existing.
 
== Optimizations ==
* Sample rate ~1-3 hz
* JPEG_QUALITY <= 6
* image size ~ 512x512
* RTSP/ffmpeg streaming
* Turn the whole thing into a configurable placement for capturing/streaming purposes
 
== Using ==
Add the line
<syntaxhighlight>
<canvasimage>/canvasimage</canvasimage>
</syntaxhighlight >
to the uri handler defined in file FG_ROOT/httpd-settings.xml and start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL
<syntaxhighlight>
localhost:5701/canvasimage?type=png&canvasindex=0
</syntaxhighlight>
in your browser. The first request will add the property rendertoimage to the canvas with index 0 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 0 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout).
 
The showcase script CanvasView.html shows the main instruments of Soitanens 737 with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load.
 


== Related ==
== Related ==
* https://sourceforge.net/projects/remote-canvas-for-flightgear
* {{Search|mode=forum|keywords=ffmpeg+stream}}
* {{Search|mode=forum|keywords=ffmpeg+stream}}
* {{OSG example|example=osgmovie}}
* {{OSG example|example=osgmovie}}
Line 924: Line 293:


[[Category:Cockpit building]]
[[Category:Cockpit building]]
[[Category:Canvas]]

Latest revision as of 06:53, 23 January 2019

Canvas/HTTP Server
Canvas-ND-served-via-httpd.png
Started in 10/2016
Description Serving Canvas textures via httpd
Contributor(s) ThomasS (since 10/2016)
Status Available as of 2018.3.1 (still might cause stability problems)

Objective

ND-Dialog-with-DisplayMode-support.png

There are situations, e.g., for home cockpit builders [1], where it is useful to display instruments like a PFD, ND, EICAS or any MFD externally from the FlightGear 3D main window in a separate window or on a separate monitor, computer or a mobile device [2] [3] [4].

Many of these avionics/graphics are created by FlightGear's 2D drawing Canvas system internally.


In addition there, are a number of other use-cases where being able to obtain a Canvas from fgfs using a network protocol like http may be desirable (e.g. imagine getting a tilemap based on actual scenery from FlightGear [5]) [6][7].


This article provides a patch to FlightGear for downloading any canvas image from a running FlightGear process by HTTP by serializing it to a raster image and serving that via the built-in mongoose based httpd server [8] [9].

This could be considered the groundwork needed for more sophisticated use-cases such as e.g. actually streaming a live video of a certain MFD to a browser.

An additional option for displaying canvas images on remote computer is FGQCanvas, which doesn't require to patch the core FlightGear code.

Problem

a modern biz jet will need to be able to display certain MFDs - fgpanel is not too useful for that (unless you are primarily dealing with steam gauges) - the most useful search terms for the forum/wiki will be Phi and FGCanvas - the latter of which is a special startup mode of FlightGear itself that can render Canvas based MFDs in a separate instance.

If you don't need fancy MFDs, you could probably use fgfs standalone and/or fgpanel. Phi has the lowest barrier to entry probably for people familiar with HTML and JavaScript, i.e. you don't even need to know much about fgfs. The Canvas stuff will really only be needed if you want to render things like the 747 PFD/ND or CDU in a distributed fashion, i.e. using multiple independent displays.[10]

you should be aware of glass cockpit related efforts, especially Canvas - most airliners and jets will sooner or later benefit from being ported to Canvas, e.g. to use Gijs' NavDisplay framework, or at least Philosopher's MapStructure framework for mapping purposes.

Thus, if this is also about the actual display itself, people should be aware of related canvas efforts, especially FGCanvas: FGCanvas

The my mid-term plan involves supporting a standalone mode for all Canvas-based glass instruments, including the ND, but also other instruments like the PFD, EICAS, CDU or EFB. This may sound like a lot of work, but it's mainlyy a matter of introducing a a few helper classes and ensuring that people actually adopt and use those.

In the long-term, we really want to support distributed FlightGear setups like those at FSWeekend/LinuxTag, where multiple computers may be used to run a single simulator session - including properly synchronized glass instruments like the PFD/ND etc. This would also help improve the multiplayer experience, especially dual-pilot setups etc. Regarding a pure panel with switches, kobs and buttons, we'd prefer something like that to be aircraft-agnostic, i.e. just consist of property-mapped controls that can be individually assigned, so that this could be reused for other MFDs, not just the ND - but also the PFD or EFB.[11]

There's the long-term plan to eventually port FGPanel back into FG and come up with some sort of "FGCanvas" mode, where Canvas-based displays could be run in a separate "standalone" mode and interface to the main/master FG instance.

I don't think that the Canvas system currently builds against just OpenGL ES - but it should be possible to identify problematic use-cases and make the Canvas support OpenGL ES by setting some OSG traits accordingly - at some point, i.e. 12+ months from now In general, it pays off to unify things - otherwise, there's lots of duplication and re-invention involved.

Technically, we could definitely run an OpenGL ES-based Canvas on a mobile device, we just need people interested in pursuing this and playing with it - as well as report any issues/showstoppers Such an integrated solution would also make it possible to show any canvas texture (instrument, dialog/window, MFD etc) on external devices:

Navigation display centered MAP mode.png

FGCanvas[12]


Implementation

This solution was created roughly by:

  • adding a DrawCallback to the canvas camera
  • attach an image to the canvas camera
  • duplicate Torstens http ScreenshotUriHandler to a CanvasImageUriHandler that subscribes to the callback in the canvas camera

Alternatives

There are possible alternative solutions for rendering canvases on a remote system, that remove the rendering load from the main flightgear process. These might be based on Nasal, Javascript, Python, etc.[13]

One currently available solution is FGQCanvas, created by James.

Gallery

Status (1/2019)

This feature was merged into the official 2018.3.1 release. It works so far. However, there are latencies and the solution isn't efficient. Just creating the PNG image from OSG takes up to 100ms. For the time being, this should still be considered a "proof-of-concept" that may go through more iterations of optmizations. Integrating it into 2018.3.1 makes it possible to explore a number of opportunities to optimize the whole thing for different use-cases and solicit community feedback. Special attention might be required with situations where the canvas and/or the connection are shut down, but also to support Reset & re-init without causing a race condition.

Note  In its current form, the streaming capability [16] is available but might overburden the flightgear process, depending on the number of canvases displayed. Its recommended not to use the streaming feature but to refresh the canvas from the browser. If any instability of FlightGear occurs, the remote canvas feature should be disabled for checking whether it is the reason.

Optimizations

  • Sample rate ~1-3 hz
  • JPEG_QUALITY <= 6
  • image size ~ 512x512
  • RTSP/ffmpeg streaming
  • Turn the whole thing into a configurable placement for capturing/streaming purposes
  • Howto:Serializing a Canvas to SVG (streaming a Canvas/MFD as a SVG/XML document)

Using

Start fgfs with option "--httpd=5701" (the port number is free, but 5701 is used by the showcase HTML script). Enter the URL

http://localhost:5701/screenshot?canvasindex=4&type=png

in your browser. The first request will add the property rendertoimage to the canvas with index 4 but return no image (this is no intended behaviour but the result of insufficient synchronization between the rendering thread and the HTTP thread). The request needs to be send again and the image of canvas 4 is returned. Its up to the user to specify a valid canvas index. An invalid canvas index or any form of invalid URL will not result in an error message but simply return no result (until it runs into some browser timeout). For the 777-200 canvas index 4 is the ND, 5 the MFD. PFD seems not to use Canvas.

Sample Showcase CanvasView.html

The following showcase HTML snippet CanvasView.html shows the main instruments of Soitanens 737 (and its branches) with a refresh rate of 2 per second. This will not be enough for a smooth display and might be increased by reducing the timeoutinterval down from 500ms. But keep an eye on the overall performance of your running fgfs instance. Providing the images causes a significant system load. The script contains hard-coded assumptions such as the name/index of the canvas textures to download from fgfs.


<!DOCTYPE html>
<!-- See Flightgear wiki for information-->
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Canvas View</title>
</head>
<body>
<select id="sizebox" onChange="setSize();">
    <option value="256" selected="selected">256</option>
    <option value="512">512</option>
    <option value="768">768</option>
</select>
<table>
    <row>
        <td>
            <img id="CptPFD">
        </td>
        <td>
            <img id="CptND">
        </td>
    </row>
    <row>
        <td>
            <img id="UpperEICAS">
        </td>
        <td>
            <img id="CptCDU">
        </td>
    </row>
</table>

<script type="text/javascript">
var refreshinterval=500;
var instruments = ["CptPFD","CptND","UpperEICAS","CptCDU"];
var canvasindex = [];
canvasindex["CptPFD"] = 3;
canvasindex["CptND"] = 5;
canvasindex["UpperEICAS"] = 4;
canvasindex["CptCDU"] = 2;

function refreshImage(imageid) {
    //var image = document.getElementById(imageid);
    var image = document.querySelector("#"+imageid);    

    var url = "http://localhost:5701/screenshot?type=png&canvasindex="+canvasindex[imageid];
    image.src = url;
    setSize(imageid);
    setTimeout(function() {
         refreshImage(imageid);
     }, refreshinterval);
   
}

function setSize(imageid){
    var e = document.getElementById("sizebox");
    var size = e.options[e.selectedIndex].value;
    var image = document.querySelector("#"+imageid);
    if (image != null){
        image.style.width=size+"px";
        image.style.height="auto";
    }
}

for (var i = 0; i < instruments.length; i++) {
    refreshImage(instruments[i]);
}
//refreshImage("CptND");
//refreshImage("CptPFD");


</script>
</body>
</html>

Discussion

The forum thread for discussing this feature is https://forum.flightgear.org/viewtopic.php?f=71&t=30642&start=15.

Related

References

References