intake-stac icon indicating copy to clipboard operation
intake-stac copied to clipboard

Slow creation with dynamic STAC catalogs

Open TomAugspurger opened this issue 3 years ago • 0 comments

In https://github.com/intake/intake-stac/blob/b1451497a6ea40b939f340e397fafad082fcc47a/intake_stac/catalog.py#L123, intake-stac will recurse into child objects. This will end up making many HTTP requests for large, dynamic STAC catalogs served over a STAC API like https://planetarycomputer.microsoft.com/api/stac/v1.

I believe that all the necessary information is provided at the https://planetarycomputer.microsoft.com/api/stac/v1/collections endpoint. https://github.com/stac-utils/pystac-client handles all the logic for interacting with STAC APIs efficiently (it has subclasses for pystac.Collection, etc.). Something like

pystac_client.Client.open("https://planetarycomputer.microsoft.com/api/stac/v1/").get_collections()

should efficiently get the child collections for APIs that implement the /collections endpoint (I'm unsure about child catalogs; we don't use them).

Might have some overlap with https://github.com/intake/intake-stac/issues/66.

TomAugspurger avatar Jul 05 '22 17:07 TomAugspurger