Osm2city worldbuild: Difference between revisions

From FlightGear wiki
Jump to navigation Jump to search
No edit summary
m (cleanup)
 
(8 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The osm2city worldbuild is an attempt by merspieler started in May 2019 of providing global coverage with osm2city scenery.
{{lowercase title}}
The '''osm2city worldbuild''' is an attempt by merspieler started in May 2019 of providing global coverage with osm2city scenery. The 1st worldbuild has been completed.


== Machine ==
== Machine ==
Since I was disappointed by the information I could get on the world scenery 2.0 build, espectailly on the machine it run on, I've decided to give some insight, on what is needed to run an osm2city worldbuild.
Since I was disappointed by the information I could get on the world scenery 2.0 build, espectailly on the machine it run on, I've decided to give some insight, on what is needed to run an osm2city worldbuild.


As of January 2020 the build is run on a server with two 8-core/16-thread CPUs and 96GB RAM. While at first the CPU seems to be the bottleneck, once you get into densly packed areas the RAM becomes the limiting factor. I've seen a single thread taking up more than 55GB of RAM so make sure, you've got enough. For larger systems I'd recommend to go with running one thread per 4GB RAM as many threads don't even need 1GB which gives room for some threads to grow beyond that. On smaller systems be sure to monitor the RAM usage and reduce thread count if neccesary.
As of January 2020 the build is run on a server with two 8-core/16-thread CPUs and 96GB RAM. While at first the CPU seems to be the bottleneck, once you get into densly packed areas the RAM becomes the limiting factor. I've seen a single thread taking up more than 80GB of RAM and even caused out-of-memory problems on this server. As a general rule, watch how much RAM is currently used and adjust the number of running processes accordingly.


For storage there are five 4TB HDDs configured in a RAIDZ2 with an effective capacity of just below 12TB. Currently this is more than is needed.
For storage I've started with five 4TB HDDs configured in a RAIDZ2 with an effective capacity of just below 12TB but changed to RAID10 (with one spare drive, about 8TB effective) to improve IO performance. Currently 8TB this is more than is needed.
As of writing the global osm data in the database has a size of 2.3TB but this requirement can be lowered (with a penalty in runtime) by only importing the required area from the highly compressed .osm.pbf file. With this approach, the initial try worked relativly well on a 1TB HDD with enough headroom.
As of writing the global osm data in the database has a size of 2.3TB but this requirement can be lowered (with a penalty in runtime) by only importing the required area from the highly compressed .osm.pbf file. With this approach, the initial try worked relativly well on a 1TB HDD with enough headroom. There are quite a number of random IOPS which causes some IO wait time, up to 80% of the CPU on my Server so an SSD would certainly benificial for some of the data or as L2ARC.
 
== Rollouts on LTS ==
''See [[OSM2City 1st Worldbuild|'''information and performance tips for the OSM2City 1st worldbuild''']] .''
 
The 1st rollout on the FlightGear [[Release Plan|Long Term Stable]] (LTS) stream was in [[Changelog 2020.3|FlightGear 2020.3.7]] point release. FlightGear 2020.3.7 LTS will automatically download from [[TerraSync]]. FlightGear 2020.3.6 and earlier will not automatically download OSM2City data. A small amount of existing experimental OSM2City data in an old format was removed from TerraSync.
 
 
 
[[Category:Osm2city]]
[[Category:Custom scenery]]

Latest revision as of 12:15, 22 March 2021

The osm2city worldbuild is an attempt by merspieler started in May 2019 of providing global coverage with osm2city scenery. The 1st worldbuild has been completed.

Machine

Since I was disappointed by the information I could get on the world scenery 2.0 build, espectailly on the machine it run on, I've decided to give some insight, on what is needed to run an osm2city worldbuild.

As of January 2020 the build is run on a server with two 8-core/16-thread CPUs and 96GB RAM. While at first the CPU seems to be the bottleneck, once you get into densly packed areas the RAM becomes the limiting factor. I've seen a single thread taking up more than 80GB of RAM and even caused out-of-memory problems on this server. As a general rule, watch how much RAM is currently used and adjust the number of running processes accordingly.

For storage I've started with five 4TB HDDs configured in a RAIDZ2 with an effective capacity of just below 12TB but changed to RAID10 (with one spare drive, about 8TB effective) to improve IO performance. Currently 8TB this is more than is needed. As of writing the global osm data in the database has a size of 2.3TB but this requirement can be lowered (with a penalty in runtime) by only importing the required area from the highly compressed .osm.pbf file. With this approach, the initial try worked relativly well on a 1TB HDD with enough headroom. There are quite a number of random IOPS which causes some IO wait time, up to 80% of the CPU on my Server so an SSD would certainly benificial for some of the data or as L2ARC.

Rollouts on LTS

See information and performance tips for the OSM2City 1st worldbuild .

The 1st rollout on the FlightGear Long Term Stable (LTS) stream was in FlightGear 2020.3.7 point release. FlightGear 2020.3.7 LTS will automatically download from TerraSync. FlightGear 2020.3.6 and earlier will not automatically download OSM2City data. A small amount of existing experimental OSM2City data in an old format was removed from TerraSync.