osm-export-tool

osm-export-tool copied to clipboard

Large countries with dense feature objects have to be split into various sections (i.e. Tanzania, Indonesia, Japan) for exporting to HDX. These are hosted on the site as various sections, so that users have to download each individually and then merge them together. Would be great if they were automatically merged by the tool before being pushed onto HDX.

An alternate solution, which I think will be cleaner and more reliable to implement, is to make the export tool support large country datasets without having to section them.

The limiting factor in the design right now is Overpass API. Large exports fail when Overpass times out on complex queries that return lots of data. Using Overpass lets us have minutely or same-day updates, which is required for HOT activations.

If it is true that all the large, sectioned exports like TZA and IDN occur on a monthly or biweekly basis, we can instead create a separate export process that uses the weekly planet pbf from planet.openstreetmap.org, which shouldn't have a problem with creating a single export per country (this is how the other extract tools that don't need minutely updates work).

Maybe you should investigate the use of lz4 compression instead of the default zlib for Overpass API, and deploy the newest 0.7.55 release, both of which improve query performance.

Also, can you post some query example where you experience issues? Do you see the same issues on https://lz4.overpass-api.de/api/ as well?