Have a rio download command to mirror remote datasets

[daniel-j-h]

trafficstars

Hey lovely people :hugs: again and again I'm catching myself writing small scripts to download e.g. remote slippy map tiles and mirror them on my disk. As I learned a while ago, rasterio already supports slippy map endpoints (by providing a xml file).

What do you think of having a sub-command to download and mirror remote datasets? It seems like rasterio knows about these datasets already, and all that's missing is a sub-command where we just mirror and download them. Then users could also easily e.g. point rasterio to a WMS endpoint and mirror it to disk as a slippy map directory.

To give you a specific use case, the latest use for this is me writing a small script to quickly download and mirror a part of the imagery-index wms or tms source to check it out locally e.g.

https://github.com/ideditor/imagery-index/blob/877e755a9b52cf2233ef712153608af8d95bd85c/sources/europe/germany/Berlinaerialphotograph2019.json

Thoughts?

[sgillies]

@daniel-j-h GDAL's resolution of WMS tiles is hidden by its API and is not available to rasterio. We don't know exactly which tiles it has requested or retrieved and we only see the materialized output. I think this means that rasterio will not be extra useful for mirroring a WMS.

The GDAL project has a contributed program named gdal2tiles: https://gdal.org/programs/gdal2tiles.html. It also doesn't exactly mirror a WMS, but you could effectively mirror if you gave it the right parameters. As far as I know, you could give it a WMS endpoint and it would generate directories of tiles for you.

That said, a modern rewrite of https://github.com/OSGeo/gdal/blob/master/gdal/swig/python/scripts/gdal2tiles.py using rasterio and mercantile could make a handy rio plugin. It suddenly occurs to me that the rio-mbtiles plugin and https://github.com/mapbox/mbutil could serve you well. rio-mbtiles can read a slippy map XML file and write an mbtiles database. Then mbutil could extract the database into a directory of files. You'd still need to write a new XML file as an entry point into the tiles on disk, but in some cases that would be very alike to the original XML file.

[daniel-j-h]

Ah, too bad we do not have direct access the the WMS API. I was thinking, would it work to run rio convert with a WMS as an input source and a local Slippy Map output source? This should mirror remote tiles to a local directory, no?

What is the difference of rio clipping Slippy Map tiles out of a raster compared to gdal2tiles? rasterio should already be capable to replace gdal2tiles if I'm not missing something?