eo:cloud_cover queryable broken

[m-mohr]

For example in https://github.com/stac-utils/pgstac/edit/main/src/pgstac/migrations/pgstac.0.7.9-0.7.10.sql (and most other migrations), the $ref to the eo:cloud_cover is broken.

It is https://stac-extensions.github.io/eo/v1.1.0/schema.json#/definitions/fieldsproperties/eo:cloud_cover but should be https://stac-extensions.github.io/eo/v1.1.0/schema.json#/definitions/fields/properties/eo:cloud_cover.

Generally, I think it would be much easier for tooling if the schemas would just be copied over instead of using $ref links.

[constantinius]

How can we fix this in a running instance without waiting for a patch?

[jonas-eberle]

How can we fix this in a running instance without waiting for a patch?

I guess you can do an update of eo:cloud_cover entry of the database table ("queryables") directly.

[vincentsarago]

I guess you can just do 👇

DO $$
  BEGIN
    INSERT INTO queryables (name, definition, property_wrapper, property_index_type) VALUES
    ('eo:cloud_cover','{"$ref": "https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fields/properties/eo:cloud_cover"}','to_int','BTREE');
  EXCEPTION WHEN unique_violation THEN
    RAISE NOTICE '%', SQLERRM USING ERRCODE = SQLSTATE;
  END
$$;

[vincentsarago]

Note: This is coming from previous version https://github.com/stac-utils/pgstac/pull/160/files#diff-2274efb3ca918a5fdd73a71e90b8bbd97b180963ea131085df5aa0195455c5d6

@bitner my understanding is that we will have to hardcode something in the migration steps to fix this!

[constantinius]

In a running instance, I solved it like this:

SET search_path TO pgstac;
UPDATE queryables SET definition = '{"$ref": "https://stac-extensions.github.io/eo/v1.0.0/schema.json#/definitions/fields/properties/eo:cloud_cover"}' WHERE name = 'eo:cloud_cover';

[bitner]

You'll need to use the UPDATE statement as @constantinius mentions.

I'll get a fix in for any instances that have this exact setting for eo_cloudcover, but I will make sure the migration does not change anything that anyone may have manually put in for eo_cloudcover.

I will note that if you don't want to use refs, you can always add your own entry with the full schema rather than the ref (the queryables table is intended to be actively managed). The migrations should not change anything that is already set. Refs vs full schema is definitely a choice that different implementations may want to use. The implementation that I do most of my development for prefers using the refs.

[m-mohr]

@bitner Just for "$ref": I'm implementing a browser-based tool (i.e. STAC Browser) that can show a form for queryables, but loading all the dozens of related schemas and dereferncing them freezes the Browser for multiple seconds. In this use-case it's really bad to have $refs, especially if you link to various extensions. Maybe that's even something that should be an option for the queryables endpoint (?dereference=1 or so). Edit: Opened https://github.com/opengeospatial/ogcapi-features/issues/840 for some guidance...

[bitner]

@m-mohr yes, exactly, so you may want to update the definition for eo:cloud_cover on your instance to be the non-referenced schema json rather than the $ref.

I think that something like ?dereference=1 would be better done at the client or stac-fastapi (or other application tier that uses pgstac) level and would probably want to use some sort of cache in front of the "dereferencer" rather than in pgstac (which does not have the ability to go out and fetch urls unless we use extensions that are not readily available on hosted DBaaS).

[m-mohr]

Yes, this option would better be implemented in the actual API implementation.

As STAC Browser is working for all STAC APIs, I can't update that in my instance (unless everybody gives me access to their DBs, I don't host an API myself) ;-)