Remote Properties

Remote Properties

Started in	04/2021 (early prototype)
Description	DDS based remote property tree server implementation
Contributor(s)	Erik Hofman
Status	under active development (04/2021)
Changelog	https://sourceforge.net/u/ehofman/profile/feed.rss

See Data_Distribution_Services_Support#Property_tree for the main article about this subject.

Also see Modularizing, parallelizing and distributing FlightGear and Distributed node system for FlightGear.

Background

One of the more common questions is, does the named pipe mechanism support "give me this property value", "set this property value" calls, i.e. would it be possible to use net_fdm to send named properties over to FlightGear

Erik wanted to setup a remote property tree where several instances of FlightGear could synchronise a subset of the property tree over the network. One thing he once had working was reducing a full property path to just a few bytes by using a lookup table on the server.

The client would quire a full property path and the server then returned a multi-byte (e.g. at least 32-bits) node id which the client then used to communicate with the server. That way you could:

• quire: get-id <full-path-to-property> returns node-id
• send: set node-id <new-value> returns true or false
• receive: get node-id returns node-value

One thing I particular do want to state - I suggested a very rough design but it’s one I am sure can work with the existing API for most users. It’s fine to make a new property backend but we absolutely cannot change all code that uses the SGPropertyNode class; we have a huge base of working code built upon it for the live tree, and unfortunately not much test coverage for that code. — James Turner (Jan 25th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API (was Re: A FGPythonSys implementation: ...). (powered by Instant-Cquotes)

I think that for single threaded use the current SGPropertyNode code is pretty good, especially combined with the command system which has the atomic multiple/increment/decrement/swap operation you propose. (And which I agree are valuable) Hence my thinking along lines where we keep multiple properties trees (or a master and shadow copies) using the existing API, to keep existing code happy. Now how we synchronise those could of course be exactly the kind of API you proposed, but really to me that /is/ just message passing; you have to have some locking in your property thread to regulate changes, the only question is if changes propagate instantly or are batched at some interval, which has some particular simulation advantages. — James Turner (Jan 25th, 2016). Re: [Flightgear-devel] Designing a thread-safe property tree API (was Re: A FGPythonSys implementation: ...). (powered by Instant-Cquotes)

Intro

See also Multi-Threading in FlightGear.

Introduces a small but powerful extension to the property tree in order to allow properties to be maintained in a different (remote) instance of a property tree, that is transparently accessed using a network socket.

READ/WRITE requests being transparently dispatched to the process/thread owning each individual property node.

So that properties may be maintained and provided by a separate property tree instance which does not have to live in the same process space as the rest of FlightGear (e.g. to sync state across multiple interlinked fgfs instance, in a multiplayer/dual-pilot setup or multi-instance setups like those commonly used at events like FSWeekend or LinuxTag)

Concept

This particular idea dates back to December 2004, when FlightGear core developers first talked about the idea of "remote properties" on the FlightGear Devel mailing list:

• "I would love to see all the FG core stuff going that route - one network capable API that everything can work through including FG itself. As far as I can see FG looks like it's already half way there."[1]

• "I have been working on extending the property code to add an SGRemoteProperty class to access properties on a remote host instead of locally." [2]

• "Most of the tree is cached locally, but queering the value of the end-node is done through a socket connection."[3]

• "I have this sort of working (but need more debugging).Next to the SGRemotePropertyNode I've also been considering queering multiple properties in one shot, instead of one by one."[4]

• "I think the current MP protocol could be enhanced to provide this function fairly easily. The first step would be to (re-add) function to export arbitary properties."[5]

• "The property system is already a great mechanism for communication among the sub-systems of FG. A great project would be to look at making the property system thread-safe, and follow that up by turning it into a "blackboard" that can be used by distributed copies of FlightGear."[6]

• "Simply put, the mechanics for doing this with FlightGear are already in place, you only need to take a slight detour over a communications link. This has its advantages too, such as added security (no possible code injection) and inherent networkability. Downside is that it takes a little more brain-food to make it work."[7]

It turns out that the concept of a "remote property" is very powerful because it would allow a property tree to be distributed across multiple processes, where each process may have its own internal property tree and transparently publish its own properties to a "master tree" and in turn subscribe to properties maintained in other property tree instances that are run in different processes.

This idea is particularly appealing, because it fully supports the approach of process-based parallelization and distribution in a "share nothing" fashion, where all state is explicitly propagated (duplicated) and may only be modified by the owning process (property tree).

Consequently, processes featuring their own instance of a property tree could run independent of the framerate (update rate) of the FlightGear main process, which in turn means that higher and more reliable update intervals would become possible.

A remote property could refer to nodes that are maintained in a different process, all reads would then be locally cached, while all write requests would be dispatched to the corresponding process (message queue).

In this sense, this is identical to message passing being used to access other property trees. Distributing the property tree becomes feasible because write requests will be implicitly serialized due to the single threaded nature of those host processes containing each property tree instance.

Use Case Scenarios

Remote properties would at least map to the following situations in FlightGear that require replicating shared state in a distributed fashion across multiple instances of FlightGear or related tools:

• multiplayer (probably the purest form of remote property use)
• autopilot (with remote properties available, A/Ps could be directly run outside the mainloop)
• synchronizing multiple running FlightGear instances
• shared, dual/multi control (multi crew vehicles)
• datalink channels (i.e. radio frequency)
• instructor console (access FlightGear internals)
• ATC radar client
• AI traffic system (use remote properties to instantiate and control AI objects remotely)
• Canvas MFD replication across multiple fgfs instances
• replay
• weather/environment server
• fdm server
• fgjs

More Background Info

• "The most flexible and powerful approach to maintaining server-side state would probably involve equipping each server with its own instance of a property tree, so that the server would end up being a property tree server supporting publish/subscribe messages which may include optional arguments to specify requested update intervals and update conditions, where all transmitted state would end up being put into the server's property tree namespace."[8]

• "The server would end up being a state variable server supporting dynamically configurable publish/subscribe methods in order to store, maintain and propagate state selectively and on a conditional basis."[9]

• "ideally, the whole fgfs protocol management system would be revamped to be based on the very same techniques: frequently users are requesting similar facilities for their own, custom protocol implementations (i.e. not necessarily related to fgms). Various related requests have been posted and discussed in the atlas RFE tracker." [10]

• "In this sense, it simply makes sense to implement everything on top of the same API, because the multiplayer protocol would then only become one "manifestation" of a custom protocol supported by fgfs."

• "well, given that all transmissions would be centered around sending/receiving property tree data/values, a far better approach to implementing the various feature requests here would be to extend the current property tree code so that it exposes its major methods (i.e. getters/setters) via an RPC interface, so that -at least optionally- each property tree instance could expose an RPC interface to the property tree."[11]

• "This could be implemented easily in a multi-platform fashion using UDP or even DBUS (which is likely to be also supported on WinTel platforms soon)."[12]

• "In fact, SimGear already has an UDP Socket class, so if we take the existing property tree code by deriving a subclass from SGPropertyNode & SocketUDP (or possibly even SGIOChannel to make it truly generic) and provide wrappers to map methods to RPC functions, we could quickly end up with an property tree implementation that provides RPC facilities in a multi-platform fashion, so that this could be employed for arbitrary IPC purposes, including but not limited to implementing multiplayer support."[13]

• "In addition to the current property tree methods, some additional helpers may be required - such as for example support for publish/subscribe messages in the RPC implementation. That way, the multiplayer server could simply boil down to a standalone version of a property tree that is remotely manipulated by arbitary -legitimate- fgfs clients which would then simply publish data and subscribe to server-side data. "[14]

• "This would in fact even facilitate improvements in FlightGear's architecture regarding multi-threading support, simply because it would encourage a subsystem/component design where individual subsystems could own their own instances of a property tree, so that running components in separate threads would be a no-brainer because read/write accesses would always be dispatched via IPC/RPC to the corresponding "own" property tree instance"[15]

• "Having a way to support "remote properties" would inevitably also require a way to integrate these with conventional local properties, in other words something that would allow us to mark certain properties as being remotely controlled, while "mounting" them in a local property tree."[16]

• "When we talk about "mounting remote properties" in a local property tree,we'll probably not just want to be able to mount individual properties, but possibly also whole remote property branches with all child nodes - or even a whole remote property tree that is stored somewhere else." [17]

• "ideally, such a mechanism would locally mirror a remote branch in a local mount point and allow for local writes to be transparently dispatched to the remote property tree."[18]

• "so that fgSetBool("/local/mount-point/foo",true) would result in dispatch via RPC to set the value of foo in the remote property tree."[19]

Implementation

Since the very beginning, the property tree has been designed to be very flexible:

• "I made the property manager flexible enough that you could provide any back-end retrieval by implementing, and provided default SGRawValue implementations for internal values, variable pointers, static functions, and object methods."[20]

• "When the property value is not managed internally in the SGPropertyNode, the SGPropertyNode will contain a reference to an SGRawValue which provides an abstract way to get, set, and clone the underlying value. The SGRawValue may change frequently during a session as a value is retyped or bound and unbound to various data source, but the abstract SGPropertyNode layer insulates the application from those changes."[21]

• "The SGPropertyNode class always keeps a *copy* of a raw value, not the original one passed to it; if you override a derived class but do not replace the clone method, strange things will happen.All derived SGRawValue classes must implement getValue, setValue, and clone for the appropriate type."[22]

flightgear/simgear/next/simgear/props/props.hxx#l237 still provides this flexibility, e.g. see line 259 in props.hxx for support for different backend implementations to read and write values using streams:

template<typename T>
class SGRawBase<T, 0> : public SGRawExtended
{
    virtual SGRawExtended* makeContainer() const;
    virtual std::ostream& printOn(std::ostream& stream) const;
    virtual std::istream& readFrom(std::istream& stream);
};

References

References