A NavDB web service: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
(Switch to the {{forum url}} and {{forum link}} templates for all forum links.)
(3 intermediate revisions by one other user not shown)
Line 3: Line 3:
{{WIP}}
{{WIP}}
-->
-->
== Challenges ==
there's no terminal at Kai-tek (realistic!) and the generic OSM skyscapers would be great, as well as the chequerboard and lead-in lights. (Really we should make these pieces conditional on the simulator date, since Kai-Tek is long closed - we do that for some other time-sensitivie landmarks).<ref>{{cite web
  |url    =  {{forum url|p=192365}}
  |title  =  <nowiki> Re: OSM buidings EHLE </nowiki>
  |author =  <nowiki> zakalawe </nowiki>
  |date  =  Oct 23rd, 2013
  |added  =  Oct 23rd, 2013
  |script_version = 0.40
  }}</ref>
This use scenario highlights a fundamental problem with our scenery approach, in dealing with historical periods. We just have no good way to have most of the nav data be current, but some portions reflect the world as it was in 1960 or 1980. I would prefer we maintained some global historical data sets of nav data, but in principle this would also mean maintaining historical scenery and airports, and selecting / de-selecting models based on data ranges. (So buildings appear based on their construction / demolition date). Obviously that’s a colossal amount of work which is not going to happen without effort. <ref>{{cite web
  |url    =  https://sourceforge.net/p/flightgear/mailman/message/35383393/
  |title  =  <nowiki> Re: [Flightgear-devel] apt.dat - add local version ? </nowiki>
  |author =  <nowiki> James Turner </nowiki>
  |date  =  Sep 21st, 2016
  |added  =  Sep 21st, 2016
  |script_version = 0.40
  }}</ref>
== Background ==
== Background ==
{{Main article|Navdata cache}}
{{Main article|Navdata cache}}
{{FGCquote
|1= we'll need the capability to to have two clients access the NavDB concurrently - I've hit lots of errors trying to do this
|2= {{cite web
  | url    = http://sourceforge.net/p/flightgear/mailman/message/34725710/
  | title  = <nowiki>[Flightgear-devel] HLA Update</nowiki>
  | author = <nowiki>Stuart Buchanan</nowiki>
  | date  = Dec 29th, 2015
  | added  = Dec 29th, 2015
  | script_version = 0.23
  }}
}}
A user once asked in the forums whether FG wouldsupport navaids not only from the nav.dat.gz file, but from files in custom sceneries.
A user once asked in the forums whether FG wouldsupport navaids not only from the nav.dat.gz file, but from files in custom sceneries.
For example, in ULLI airport there is a problem with glideslope and NDB at the O0outer markers and there are missing NDBs at middle markers. A user made a correct version of nav.dat.gz, but it worked only on his computer. He could distribute his scenery with the fixed nav.dat.gz, but if another player had made his own modifications in his own nav.dat, they would conflict. It would be possibls to write instructions how to find bad data and replace it with good data, but not every player wants to do such things, especially, if he has played MSFS earlier. That the main problem.<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=174016#p174016
For example, in ULLI airport there is a problem with glideslope and NDB at the O0outer markers and there are missing NDBs at middle markers. A user made a correct version of nav.dat.gz, but it worked only on his computer. He could distribute his scenery with the fixed nav.dat.gz, but if another player had made his own modifications in his own nav.dat, they would conflict. It would be possibls to write instructions how to find bad data and replace it with good data, but not every player wants to do such things, especially, if he has played MSFS earlier. That the main problem.<ref>{{cite web |url={{forum url|p=174016}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>Soitanen</nowiki>
     |author=<nowiki>Soitanen</nowiki>
Line 22: Line 55:


== Objective ==
== Objective ==
At least for the US, there's a ton of data freely available that can be extracted from such charts procedurally using a bit of scripting and OCR or regex. The corresponding data could be put into a database and updated regularly, according to the AIRAC cycle. But something like that should really be a central web service - in FG, we now have the means to easily download data via HTTP, and make web service requests. In other words, we could populate some kind of central navdb with our existing data, update the whole thing with the kind of data provided by the scripted discussed at [http://forum.flightgear.org/viewtopic.php?p{{=}}206529#p206529 Subject: A project to create a source of free geo-referenced instrume].
At least for the US, there's a ton of data freely available that can be extracted from such charts procedurally using a bit of scripting and OCR or regex. The corresponding data could be put into a database and updated regularly, according to the AIRAC cycle. But something like that should really be a central web service - in FG, we now have the means to easily download data via HTTP, and make web service requests. In other words, we could populate some kind of central navdb with our existing data, update the whole thing with the kind of data provided by the scripted discussed at {{forum link|p=206529|title=A project to create a source of free geo-referenced instrume}}.


At that point, we would have a web service that could run a cron job to regularly update the whole thing - maybe in combination with some kind of wiki-like front-end to allow contributors to update/edit and maintain entries for different countries and navaids, as well as procedures.<br/>
At that point, we would have a web service that could run a cron job to regularly update the whole thing - maybe in combination with some kind of wiki-like front-end to allow contributors to update/edit and maintain entries for different countries and navaids, as well as procedures.<br/>


The navdb would then be md5-hashed and fetched on demand, so that the navcache can be updated/rebuilt accordingly. <ref> {{cite web |url=http://forum.flightgear.org/viewtopic.php?p=213149#p213149
The navdb would then be md5-hashed and fetched on demand, so that the navcache can be updated/rebuilt accordingly. <ref> {{cite web |url={{forum url|p=213149}}
     |title=<nowiki>Re: 777 EFB: initial feedback</nowiki>
     |title=<nowiki>Re: 777 EFB: initial feedback</nowiki>
     |author=<nowiki>Hooray</nowiki>
     |author=<nowiki>Hooray</nowiki>
Line 35: Line 68:
For a complex EFB, it would be possible to create 100% of the charts procedurally, including charts for TPs (terminal procedures like SIDs/DPs, STARs and IAPs) - the main issue is lack of data. And even if we get our hands on a relatively recent AIRAC cycle, the FG side of things (navdb) will not match that data. And in  FG, we don't currently have the corresponding TP data. Converting/rendering existing charts (even PDFs from airnav.com)  should be straightforward, but the FG world would use a different set of navaids and FIXES. So, we'd be better off looking at how X-Plane handles this challenge, given that they also used to use DAFIF(T) originally.
For a complex EFB, it would be possible to create 100% of the charts procedurally, including charts for TPs (terminal procedures like SIDs/DPs, STARs and IAPs) - the main issue is lack of data. And even if we get our hands on a relatively recent AIRAC cycle, the FG side of things (navdb) will not match that data. And in  FG, we don't currently have the corresponding TP data. Converting/rendering existing charts (even PDFs from airnav.com)  should be straightforward, but the FG world would use a different set of navaids and FIXES. So, we'd be better off looking at how X-Plane handles this challenge, given that they also used to use DAFIF(T) originally.


In general, developers would probably favor procedural creation if we could get the data - and short of that, I'd use the approach discussed a while ago: [http://forum.flightgear.org/viewtopic.php?p{{=}}206529#p206529 Subject: A project to create a source of free geo-referenced instrume]<br/>
In general, developers would probably favor procedural creation if we could get the data - and short of that, I'd use the approach discussed a while ago: {{forum link|p=206529|title=A project to create a source of free geo-referenced instrume}}<br/>


Such charts would be centrally created, along with VRT (XML) files for geo-referencing them, so that the corresponding charts could be requested at run-time and downloaded on demand (or cached).
Such charts would be centrally created, along with VRT (XML) files for geo-referencing them, so that the corresponding charts could be requested at run-time and downloaded on demand (or cached).
We could probably use a simple form of OCR to extract navaid info from such charts to obtain relevant navaids for the given sector - it's just our navdb design that is kind of inflexible at the moment, but we could certain enrich/augment existing data using such updated data for navaids and fixes (or procedures).<ref> {{cite web |url=http://forum.flightgear.org/viewtopic.php?p=213143#p213143
We could probably use a simple form of OCR to extract navaid info from such charts to obtain relevant navaids for the given sector - it's just our navdb design that is kind of inflexible at the moment, but we could certain enrich/augment existing data using such updated data for navaids and fixes (or procedures).<ref> {{cite web |url={{forum url|p=213143}}
     |title=<nowiki>Re: 777 EFB: initial feedback</nowiki>
     |title=<nowiki>Re: 777 EFB: initial feedback</nowiki>
     |author=<nowiki>Hooray</nowiki>
     |author=<nowiki>Hooray</nowiki>
Line 48: Line 81:
[http://navdata.fgx.ch/ http://navdata.fgx.ch/]{{dead link}}
[http://navdata.fgx.ch/ http://navdata.fgx.ch/]{{dead link}}


The NavData project takes the latest xplane data, imports it to a postgis database spatially ready for querying on demand. It is intended as a "permanent" feed moving foward and making the latest xplane data available and updated for all, whether for caching local or online live. The data returned is formatted in a pratical way with a lot of the calculations already done, such as the middle of an airport, runway center point, orientation and displaced threshold etc. Also measures are returned in different units such as evelation both in metres and feet to make life easier for both calculation and presentation for end user application. <ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=207549#p207549
The NavData project takes the latest xplane data, imports it to a postgis database spatially ready for querying on demand. It is intended as a "permanent" feed moving foward and making the latest xplane data available and updated for all, whether for caching local or online live. The data returned is formatted in a pratical way with a lot of the calculations already done, such as the middle of an airport, runway center point, orientation and displaced threshold etc. Also measures are returned in different units such as evelation both in metres and feet to make life easier for both calculation and presentation for end user application. <ref>{{cite web |url={{forum url|p=207549}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>Hooray</nowiki>
     |author=<nowiki>Hooray</nowiki>
Line 63: Line 96:
Problem is to create and automated way to send this upstream to Robin the xplane maintainer.. but am well into this stuff
Problem is to create and automated way to send this upstream to Robin the xplane maintainer.. but am well into this stuff


Problem with navdata at the moment is I need some users to check the api and make it "usable" moving forward...' <ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=207675#p207675
Problem with navdata at the moment is I need some users to check the api and make it "usable" moving forward...' <ref>{{cite web |url={{forum url|p=207675}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>ac001</nowiki>
     |author=<nowiki>ac001</nowiki>
Line 73: Line 106:
At some point, someone even came up with a script to extract such stuff procedurally from airnav/TPP charts. This is something that me worthwhile to explore again.
At some point, someone even came up with a script to extract such stuff procedurally from airnav/TPP charts. This is something that me worthwhile to explore again.
API-wise, it would be a good idea to provide a mandatory version attribute, so that you can safely upgrade things later on, without features breaking. <ref>  
API-wise, it would be a good idea to provide a mandatory version attribute, so that you can safely upgrade things later on, without features breaking. <ref>  
{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=207695#p207695
{{cite web |url={{forum url|p=207695}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>Hooray</nowiki>
     |author=<nowiki>Hooray</nowiki>
Line 81: Line 114:
Also see: [http://sourceforge.net/p/flightgear/mailman/flightgear-devel/thread/536E6B56.2030209%40btinternet.com/#msg32324567 http://sourceforge.net/p/flightgear/mai ... sg32324567]
Also see: [http://sourceforge.net/p/flightgear/mailman/flightgear-devel/thread/536E6B56.2030209%40btinternet.com/#msg32324567 http://sourceforge.net/p/flightgear/mai ... sg32324567]


Robin is already working on improving the submission side (of apt.dat at least). See [http://developer.x-plane.com/2013/12/airport-authoring-sharing-airports/ http://developer.x-plane.com/2013/12/ai ... -airports/]  <ref> {{cite web |url=http://forum.flightgear.org/viewtopic.php?p=207698#p207698
Robin is already working on improving the submission side (of apt.dat at least). See [http://developer.x-plane.com/2013/12/airport-authoring-sharing-airports/ http://developer.x-plane.com/2013/12/ai ... -airports/]  <ref> {{cite web |url={{forum url|p=207698}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>Gijs</nowiki>
     |author=<nowiki>Gijs</nowiki>
Line 97: Line 130:
* with a GPL license.
* with a GPL license.


Then, we'd willing to hear about him - or her. <ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=152714#p152714
Then, we'd willing to hear about him - or her. <ref>{{cite web |url={{forum url|p=152714}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>f-ojac</nowiki>
     |author=<nowiki>f-ojac</nowiki>
Line 117: Line 150:
The whole work flow could be moved to a website, so that it would be easier contribute.
The whole work flow could be moved to a website, so that it would be easier contribute.


Using git as the backend (i.e. plain text files) would have the advantage, that we'd also get a revision history automatically.<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?p=152722#p152722
Using git as the backend (i.e. plain text files) would have the advantage, that we'd also get a revision history automatically.<ref>{{cite web |url={{forum url|p=152722}}
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |title=<nowiki>Re: Why does Flightgear use X-Plane airport data?</nowiki>
     |author=<nowiki>Hooray</nowiki>
     |author=<nowiki>Hooray</nowiki>
Line 132: Line 165:
|1= Sourcing only from the Xplane data myself, ground nets are absent and TWY names were almost invariably messy so I ended up turning them off because they were useless; only tarmac shapes looked good... after some work. Would you have a solution for properly named taxiways?
|1= Sourcing only from the Xplane data myself, ground nets are absent and TWY names were almost invariably messy so I ended up turning them off because they were useless; only tarmac shapes looked good... after some work. Would you have a solution for properly named taxiways?
|2= {{cite web
|2= {{cite web
   | url    = http://forum.flightgear.org/viewtopic.php?p=264699#p264699
   | url    = {{forum url|p=264699}}
   | title  = <nowiki>Re: Improving stability: No-scenery-emergency mode</nowiki>
   | title  = <nowiki>Re: Improving stability: No-scenery-emergency mode</nowiki>
   | author = <nowiki>mickybadia</nowiki>
   | author = <nowiki>mickybadia</nowiki>

Revision as of 13:05, 6 June 2019

This article is a stub. You can help the wiki by expanding it.

Challenges

there's no terminal at Kai-tek (realistic!) and the generic OSM skyscapers would be great, as well as the chequerboard and lead-in lights. (Really we should make these pieces conditional on the simulator date, since Kai-Tek is long closed - we do that for some other time-sensitivie landmarks).[1]


This use scenario highlights a fundamental problem with our scenery approach, in dealing with historical periods. We just have no good way to have most of the nav data be current, but some portions reflect the world as it was in 1960 or 1980. I would prefer we maintained some global historical data sets of nav data, but in principle this would also mean maintaining historical scenery and airports, and selecting / de-selecting models based on data ranges. (So buildings appear based on their construction / demolition date). Obviously that’s a colossal amount of work which is not going to happen without effort. [2]

Background

1rightarrow.png See Navdata cache for the main article about this subject.

Cquote1.png we'll need the capability to to have two clients access the NavDB concurrently - I've hit lots of errors trying to do this
— Stuart Buchanan (Dec 29th, 2015). [Flightgear-devel] HLA Update.
(powered by Instant-Cquotes)
Cquote2.png

A user once asked in the forums whether FG wouldsupport navaids not only from the nav.dat.gz file, but from files in custom sceneries. For example, in ULLI airport there is a problem with glideslope and NDB at the O0outer markers and there are missing NDBs at middle markers. A user made a correct version of nav.dat.gz, but it worked only on his computer. He could distribute his scenery with the fixed nav.dat.gz, but if another player had made his own modifications in his own nav.dat, they would conflict. It would be possibls to write instructions how to find bad data and replace it with good data, but not every player wants to do such things, especially, if he has played MSFS earlier. That the main problem.[3]

We have a workflow for scene-models that works well - people submit data (models), a central server / DB / etc collates them, and periodically new versions are available to everyone (terrasync, whether online / offline / manual). The exact same flow can work for nav data, ideally in the same place. We can augment Robin’s data (or not), maintain versions in sync with scenery (indeed, James Turner would move the data into the terrasync file tree next to Airports/ and Models/), etc.[4]

Objective

At least for the US, there's a ton of data freely available that can be extracted from such charts procedurally using a bit of scripting and OCR or regex. The corresponding data could be put into a database and updated regularly, according to the AIRAC cycle. But something like that should really be a central web service - in FG, we now have the means to easily download data via HTTP, and make web service requests. In other words, we could populate some kind of central navdb with our existing data, update the whole thing with the kind of data provided by the scripted discussed at A project to create a source of free geo-referenced instrume post on the forum This is a link to the FlightGear forum..

At that point, we would have a web service that could run a cron job to regularly update the whole thing - maybe in combination with some kind of wiki-like front-end to allow contributors to update/edit and maintain entries for different countries and navaids, as well as procedures.

The navdb would then be md5-hashed and fetched on demand, so that the navcache can be updated/rebuilt accordingly. [5]

EFB

For a complex EFB, it would be possible to create 100% of the charts procedurally, including charts for TPs (terminal procedures like SIDs/DPs, STARs and IAPs) - the main issue is lack of data. And even if we get our hands on a relatively recent AIRAC cycle, the FG side of things (navdb) will not match that data. And in FG, we don't currently have the corresponding TP data. Converting/rendering existing charts (even PDFs from airnav.com) should be straightforward, but the FG world would use a different set of navaids and FIXES. So, we'd be better off looking at how X-Plane handles this challenge, given that they also used to use DAFIF(T) originally.

In general, developers would probably favor procedural creation if we could get the data - and short of that, I'd use the approach discussed a while ago: A project to create a source of free geo-referenced instrume post on the forum This is a link to the FlightGear forum.

Such charts would be centrally created, along with VRT (XML) files for geo-referencing them, so that the corresponding charts could be requested at run-time and downloaded on demand (or cached). We could probably use a simple form of OCR to extract navaid info from such charts to obtain relevant navaids for the given sector - it's just our navdb design that is kind of inflexible at the moment, but we could certain enrich/augment existing data using such updated data for navaids and fixes (or procedures).[6]

Status

http://navdata.fgx.ch/[dead link]

The NavData project takes the latest xplane data, imports it to a postgis database spatially ready for querying on demand. It is intended as a "permanent" feed moving foward and making the latest xplane data available and updated for all, whether for caching local or online live. The data returned is formatted in a pratical way with a lot of the calculations already done, such as the middle of an airport, runway center point, orientation and displaced threshold etc. Also measures are returned in different units such as evelation both in metres and feet to make life easier for both calculation and presentation for end user application. [7]

Developer's Plans

We quote ac001: 'I'm the developer behind navdata.fgx.c and thanks to Hooray for alerting me as i dont follow the forums much...

So What I would want is to publich the xplane data.

But also have a way to "submit corrections", and maybe have a voting system of "yes this is correct".....

Problem is to create and automated way to send this upstream to Robin the xplane maintainer.. but am well into this stuff

Problem with navdata at the moment is I need some users to check the api and make it "usable" moving forward...' [8]

Coordinating with XPlane

Coordinating this with Robin Peel would be awesome, because of the sizable XP community, i.e. we could truly benefit here by leveraging our two communities to help update such data. At some point, someone even came up with a script to extract such stuff procedurally from airnav/TPP charts. This is something that me worthwhile to explore again. API-wise, it would be a good idea to provide a mandatory version attribute, so that you can safely upgrade things later on, without features breaking. [9]

Also see: http://sourceforge.net/p/flightgear/mai ... sg32324567

Robin is already working on improving the submission side (of apt.dat at least). See http://developer.x-plane.com/2013/12/ai ... -airports/ [10]

A NavDB wiki ?

If you know:

  • someone who is willing to develop, contruct and maitain the full airport diagrams database + its implementation;
  • a solution better than the formats used in XPlane;
  • someone who has the time to rebuild the whole toolchain to analyse this new format;
  • build a nice and crossplatform tool we can use to build airports;
  • with a GPL license.

Then, we'd willing to hear about him - or her. [11]

Advantages of staying with XPlane

There are so many advantages when reusing the X-Plane data and their format, they have a HUGE community of contributors - which FG simply doesn't have.

On the other hand, some web-based frontend to allow people to easily contribute and maintain airport/navdb data, would be great to have. Something pretty much like a "GIS Wiki", possibly based on PSQL or even git - so that people could go to a website, open an airport/navaid and file merge requests, review entries and help maintain all the data there would be useful.

The new scenemodels submission process is actually a very good example here.

Something like this could be very useful, not just for airports (runways, taxiways, aprons etc), but also for navaids and terminal procedures (SIDs/STARs and IAPs).

And I guess we could even get the X-Plane community to contribute to something like this. The whole work flow could be moved to a website, so that it would be easier contribute.

Using git as the backend (i.e. plain text files) would have the advantage, that we'd also get a revision history automatically.[12]

We need to learn the Open Street Map lesson here.


References

References