Using OSM Vector Data in FlightGear: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
Line 47: Line 47:


=== Random Buildings Scripting Interface (Hooray, Thorsten) ===
=== Random Buildings Scripting Interface (Hooray, Thorsten) ===
This would require some form of interface, so that stuart's random buildings can be aligned using networks of roads, rivers, railway tracks - like Gijs said, the randomness would be gone then


'''Hooray:''' we have had a fair share of discussions related to this. Stuart's work has certainly paved the way for even more improvements.
'''Hooray:''' we have had a fair share of discussions related to this. Stuart's work has certainly paved the way for even more improvements.

Revision as of 20:41, 30 January 2014

Background

This is referring to procedural/autogen scenery generation/enrichment (buildings, cities, roads, bridges and power lines) via OSM data.

The probem is that (as usual) people pull into different directions.

Actually, we've seen a number of related developments - such as IntelQube's "Instant City" addon, or the more recent effort by Rickbritto. So people are clearly interested in this. Like Thorsten also rightly pointed out, people are pulling into different directions, which are unfortunately somewhat conflicting. But overall, it's pretty obvious that people are interested in extending the scenery and populating it with buildings in particular.

Stuart has a 'random with certain moderately simple rules' algorithm with the aim to be as fast as possible without generating nonsense like overlapping or floating buildings. This probably means you can't have fancy night illumination, too much individual configuration or realistic arrangements beyond a point, and all placement info is encoded in the texture sheet (that's not a bad feature, because that means random buildings stay in sync with the underlying sheet!)

We've seen quite a few people working on this independently in 2013 and making very promising progress in different fields. Many of them being contributors able to build from source, and able to write C++ code. TorstenD mentioned a while ago, that he was also considering porting PixelCity to use Stuart's random buildings system (and thus, OSG) for these purposes. Now, we also have osgEarth as a secondary scenery engine, so it makes sense to coordinate things accordingly.

Besides, we also have guys working on improved generation of procedurally-created vegetation, road networks, bridges and buildings/cities, which are other areas that could greatly benefit from a little coordination with osgEarth support in FlightGear, to ensure that a generic approach is found over time and to avoid a "technology lock-in", like some people are complaining about related to Rembrandt.

Please see thee ongoing discussion at: http://wiki.flightgear.org/User_talk:F-JJTH#Implement_an_OSM_client_in_FG

The main point being that we should work out a way to expose the underlying APIs to FG, so that these can be used as the common back-end for OSM vector data access (probably via Tom's cppbind) - so that related efforts can benefit without having to duplicate work, and without having to use approaches that are not native to FG (such as python code).

As could be seen in 2013, we have an unprecedented number of people interested in this currently, who are able to contribute to the code - so if we could coordinate this a little, this could work out as well as the NavDisplay effort, which had a similar amount of people interested in related things.

We now have so many people doing work related to this, that the wiki is the only sane place to coordinate things - because some use the devel list, others only the forum - and others only IRC ... so better use the wiki for now.

Also, the wiki is the only future-safe method to organize things, in case we need to drop the effort for a "while" (weeks, months or years).

Objective

Any OSM-processing should ideally be exposed via FG APIs, so that OSM data can be processed by other subsystems (either via fgcommands, extension function or cppbind), to ensure that we cannot just visualize OSM data using the Canvas, but also use it affect the placement heuristics in Random Buildings according to Stuart's plans [1] or interface OSM with osgEarth.

Status

If you are interested in pre-processing and enriching vector data, you'll probably want to get in touch with some of the other folks who have done similar things, such as the osm2xp project. Also, if that's definitely your main interest, you may also want to look at TerraGear, which is the "FG scenery compiler suite" to build scenery.

That said, a "pre-processor" could also be implemented as a native FG component, i.e. within the main fgfs executable - quite probably even as a fully threaded component, so that pre-processing could take place in a simple worker thread using exclusive read/write access.

I'm just saying this because "compiling" scenery from scratch is generally understood not to be a simple task for most people here. In other words, even if you should come up with such an OSM-pre-processor to enrich FG scenery, the resulting data may not be usable by most people - not without an official scenery rebuild at least.

Thus, having a component that would be running inside (or next to) FG would be easier from a deployment point of view.

Efforts

Random Buildings (Stuart)

Stuart is the developer of the random buildings code (and the random trees), so he's got plenty of experience with procedural creation of scenery elements. To understand the C++ side of things, you'll definitely also need to be familiar with OpenGL and preferably OSG, i.e. 3D graphics and 3D maths. But even if that's not your thing, reusing Stuart's code to the largest possible degree would definitely be a good idea, because he's already tackled many of the difficult issues - and creating an interface (scripting or property tree) is definitely simpler than writing such a system from scratch.

Random Buildings Scripting Interface (Hooray, Thorsten)

This would require some form of interface, so that stuart's random buildings can be aligned using networks of roads, rivers, railway tracks - like Gijs said, the randomness would be gone then

Hooray: we have had a fair share of discussions related to this. Stuart's work has certainly paved the way for even more improvements. Like I mentioned, I would be interested in exposing the creation of procedural scenery to scripting space (or even just the property tree) so that customizing this aspect of FG is no longer possible purely from C++ space.

it would be great to have a way to dynamically assemble buildings and blocks of buildings and then combine them into a single geometry, so that the performance hit is decreased. But that would obviously require exposing some more hooks to Nasal scripts.

Now, what Stuart has done already addresses the needs in C++ space. Adding a Nasal interface to existing C++ code is much easier that way, and certainly easier than creating the whole thing from scratch.

And have better access to the placement heuristics, or rather the underlying criteria would be great. That's not just related to "materials" anymore, but also access to "ground networks", such as roads, rivers, railways.

The good thing is, adding a Nasal interface to make existing C++ code accessible is fairly trivial and much simpler than writing the original backend code. And this is clearly not a task that needs to be done by the developer of the original C++ code, as can be seen by all the existing Nasal extension functions, most of which were added by people who didn't really write the corresponding backend code.

A Nasal script that may query a scenery tile and get detailed information on these properties, could come up with the heuristics to place models realistically. Coming up with the models procedurally would be a different matter. But it seems, Stuart has addressed this already in OSG space. So the really difficult work has probably already been done. Coming up with an interface to access his code from scripting space should be simpler than writing his code from scratch.

Hooray and Thorsten have some more involved dynamical city-building scheme in mind - finding an algorithm which dynamically generates a street pattern and places a plausible mockup of a city dependent on its knowledge of the underlying terrain. Much of the performance here depends on what models are placed - if models without an xml wrapper are used, it's going to be relatively fast, if not, then not. That's different from the situation with weather where we could never use models without xml wrapper, because the rotation effect needed to be declared there (merging the models to a whole block is also important though).

PixelCity (Hooray, TorstenD)

Once you do take a look, you'll also find a number of threads related to the open source PixelCity project, which uses a fixed rendering pipeline (i.e. no shaders) - but which is nonetheless extremely impressive. Like Torsten mentioned a while ago, this would need to be ported to OSG (and shaders) to be usable in FG, but this could probably be done by merging it with Stuart's code, because that's a very good (and working!) example of how the FlightGear/SimGear side of things works.

It really looks cool from the distance and close up. Implementing it into FlightGear requires porting the OpenGL calls to be OpenSceneGraph compatible, probably by implementing osgDrawable. The current implementation only shows night-time views, so this has to be tweaked, too. Another issue to take care of is interaction with existing scenery models, so the generated buildings do not interfere with our scenery models. The same is true for our landcover, roads and elevation data. Definitely worth some thinking. Probably as a google summer of code


Also see AutoGen Scenery for FlightGear.

osgEarth (poweroftwo)

OSM & Canvas (TheTom)

TheTom is also working on something to show tilemaps inside a canvas. I've now pushed a function to fgdata (https://gitorious.org/fg/fgdata/commit/41b6c9c688ce78d8c907a07c42b9ea4d8290da0c) which allows building templates for the tile urls.


Also see Canvas & Tiled Maps Note that osgEarth will probably soon be linked into FG by default according to [2], which means that GDAL/OGR can be relied upon, i.e. supporting GeoTIFF processing would be another useful option for any mapping/charting needs implemented via Tom's Canvas system.

Procedural Airports (psadro_gm, zakalawe, papillon81)

As was said previously, X-Plane has been doing that for years - and meanwhile we may have the computing power (and idle cores!) to actually sonsider doing that in FG, too. So it would be a great addition and would quite certainly lower the entry barrier for people doing scenery development if airport changes became directly visible without having to go through the hassle of running TG tools.

fully dynamically created airports is another really long-standing scenery related idea, that's been discussed among core developer on and off for over a decade already, including David Megginson and Andy Ross

http://forum.flightgear.org/viewtopic.php?p=176132#p176151

What about a purely symbolic representation? Store just centerline
and width for each taxiway, and keep the existing polygonal
representation for "tarmac" areas. Throw out the "pre-cut" airport

This means that the tile loader will be forced to do some
computational geometry at load time to decide on the actual polygons
to be rendered by OpenGL. But that's not likely to be too much --
even the biggest airpots have no more than 30 taxiways with no more
than ~4 curves on each. Figure an individual curve has no more than
10 segments, with 6 polygons for each segment (two yellow paint, four
pavement). So that's a few thousand polygons generated and intersect
per airport loaded. Not so bad, really -- note that because it can
happen in the loader thread, and therefore make good use of the
multi-dispatch multi-core CPUs that are becoming so popular these
days.

The biggest advantage is that (if you do it with the runways too) it
means that the airports can be generated at runtime and edited by the
user without doing a terragear scenery regeneration.

we already have all the data to generate the ground polygons, inside FlightGear - that's exactly how the airfield chart / ground-radar work. And we can do it adaptively (more bezier interpolation on faster machines) and also get airports editable in mostly-realtime (unless the airport boundary changes).

The trick is to make an OSG 'loader' which creates the airport geometry from the apt.dat definitions, which of course is exactly what genapts does - this is why I was working with Chris to make it fast enough to run inside the loader thread, in FG. I'd love to fully generate the apts at runtime. LEMD is probably the most complex layout, and genapt currently takes about 2.5 seconds to generate. I imagine it could be generated in 2 stages for LOD. 1 for runways and taxiways, which are really quick, the 2nd stage for the line data which is slower.

Procedural Buildings & Cities (radi & vanosten)

Also see Osm2city.py

Radi has focused on OSM buildings/bridges in the past, but is now interested in procedural city generation in general. OSM xml data for LOWI region alone is 50 MB. Might be a bit heavy a download for some users. Radi discussed this with James at FSweekend last year: we could pre-process OSM data, extract what's useful for us, and ship the outcome with our scenery. OSM buildings for LOWI would then add perhaps 1MB or so. OTOH, if this is generated at run-time (and then cached), users could decide themselves if/what OSM features they wish/can afford.

vanosten: I understand there are advantages to do stuff on the fly - it might be even faster to do calculations than reading detailed info like ac-files from the file system. However I wonder whether it really is the same thing that people try to accomplish. Together with Radi I want with osm2city make a plausible world, which satisfies also someone on sightseeing with a helicopter, who knows the area. The other is generating a plausible world. The first one needs a lot of parametrization and logic on a per object level. If the first one can be satisfied with on-the-fly processing, then I would look into porting code to C++ and learning C++. If not, then potentially we should leave the initiatives apart?

I agree that we might have to go with less details and a few / couple of buildings - maybe even replacing existing OSM floorplans in place. It is not about matching reality - it is about making something that looks realistic. I have built some paper based algorithms and the next thing is to try the stuff in programming. I will most probably go for a Java based prototype using the JTS Topology Suite library. If the prototype prooves sucessful and estimates for the next step do not look too bad, then I need to take a decision with your input, whether to port it to C++ / GEOS or C++ / Simgear. However this would mean that i need to learn C++.

I finally got bob.pl working using a custom Java program, which also reads the height to place the buildings (I will share the java program once it works outside Eclipse without hassle). Basically I now have 9000 objects around LSZR at the lake of Constance in Switzerland. See following image (which looks good, but which I think still shows a case for intelligently generating all those buildings, which are not yet in OSM datasource).

9000 objects means 9000 *.ac files - a lot of which could be reduced to a few, if objects would be intelligently replaced by shared objects (e.g. for residential houses, not so much for other buildings). Which leads to the following questions:

  • Right now startup is very slow due to scenery loading. Would the startup be significantly (linearily) be faster, if e.g. 60-70% of the buildings were shared buildings?
  • At night all the buildings generated by bob.pl are without lights. Could that be changed or would that again be a case for using shared buildings?
  • bob.pl is written in Perl FG is written in C++. Is there a policy against writing some of the scenery tools in e.g. Java, if one would like to contribute?
  • A previous post by Soitanen in this thread used OBJECT_STATIC_AGL to dynamically get the height over sea level for the buildings. However I guess that precalculating during scenery generation is worth the hassle in order to reduce CPU?

Populate FG world with OSM data (F-JJTH)

F-JJTH's idea here is to populate FG world with OSM data. OSM provides some API for HTTP request ( [3] ) Once data are downloaded we could be able to draw them on-the-fly in FG. In other words: it's similar to the osm2fg but in-live.

BTW, this feature is not at the top of my priority and I would wait the osgEarth integration before all.


Using the HTTP APIs is a bit overlapping with the recent work on HTTP bindings for Nasal via cppbind, but also with poweroftwo's osgEarth integration, which uses curl to provide fully threaded access to this kind of data. Also, Stuart repeatedly mentioned on the forums that he would be supportive of exposing the placement heuristics for his random buildings code-seconded by Zakalawe [4], so that these can be affected via corresponding APIs - we've had quite a bit of progress regarding procedural scenery generation in 2013, including buildings & cities (radi, Soitanen/osm2fg) [5] [6], roads, rivers - even railways (vivian) and procedural bridge generation (radi) [7], but also procedural power lines (vanosten) [8]. In other words, there's some serious -and unprecented manpower- to be leveraged and coordinated here, including quite a few folks who are able to build from source and able to write C++ code. So this deserves being coordinated among all interested parties. And it would clearly make sense not to just expose things to the Canvas system, but to expose the corresponding APIs so that other subsystems and users can access these and use these for the purposes outlined above.

OSM2FG (Saitonen)

Powerlines (owenpsmith)