Implement download-diff

[zstadler]

import-update repeatedly pulls the changes from the replication server and applies them according the replication_interval defined in the ${IMPOSM_CONFIG_FILE} file. It works well for the daily updates available for Planet and Geofabrik. It is not applicable for minutely updates because it cannot be active in parallel with the required refresh of the mbtiles by export-list.sh. This limitation is caused by a PostgreSQL restriction preventing read access to materialized views while they are being updated.

Therefore, a continuous minutely update of mbtiles would run export-list.sh in-between subsequent runs of import-diff. AFAIK, there is no tool in openmaptiles-tools to provide the required input for import-diff.

The download-diff would use the replication URL created by download-osm in the ${IMPOSM_CONFIG_FILE} file to:

• Download the required updates from the replication, according to the latest timestamp in ${area}.pbf
• Assemble them into the changes.osc.gz file as input for required by import-diff
• Update the ${area}.pbf file

The minutely updates flow would repeat the following sequence:

1. Execute download-diff to combine all new minutely updates into the changes.osc.gz file.
2. Execute import-diff to update the PostGIS DB and create the list of expired tiles.
3. Extend the list of expired tiles to all zoom levels between MIN_ZOOM and MAX_ZOOM, similar to @frodrigo's tile_multiplier.py
4. Execute export-list.sh to update the expired tiles in the mbtiles file.

A simpler, yet less-optimized flow could run quickstart.sh in-between runs of download-diff.

The implementation is likely to use osmupdate and osmconvert, already available in the openmaptiles-tools docker image.

@nyurik and @frodrigo, your thoughts?

[frodrigo]

Note. The replacement of materialized view by table is in progress. The 3. is an independent issue.