Publish dataset programmatically

[dthib]

Hi,

Is your feature request related to a problem? Please describe. I have dynamic datasets and I want to update the pygeoapi programatically with these new datasets as soon as they are available. More precisely: the job get the new data, push it to posgres and I would like to declare this new dataset in pygeoapi with metadata (update date, etc)

Describe the solution you'd like I was think of a API route to be able to add a new provider.

Describe alternatives you've considered Another option would be to use an url for remote config file and be able to trigger a config reload once the remote config file has been updated

Thanks for this great work !

[KoalaGeo]

If your updating PostGres, and the pygeoapi config for the connection stays the same, could you not run the pygeoapi deployment on a schedule in sync with the postgres updates and it would then pick up the new data?

[dthib]

Thanks for your reply. I am not familiar with the config file but if my understanding of the documentation is correct, the config file must be changed when a new dataset (eg: a postgis table) is added. Am i right ? Thanks

[KoalaGeo]

You're correct. If you're adding a new table in postgis then yes the config will need updating.

I was assuming the new data would be added to the existing postgis schema:table and you wouldn't be making a new table.

[dthib]

ok thanks for the confirmation. Any advice in this case ? Best!

[KoalaGeo]

Maybe you could stick with the one table, and amend your new data with a "last updated" datetime attribute, then the user could use the api filtering to only pull out the latest data.

Something like the Canadian weather service - https://api.weather.gc.ca/collections

Or BGS sensor services (these are proxies of https://sensors.bgs.ac.uk/ which uses the OGC SensorThingsAPI standard):

• Observations - https://ogcapi.bgs.ac.uk/collections/BGSSTAObservations/items
• Things (sensors) - https://ogcapi.bgs.ac.uk/collections/BGSSTAThings/items
• Datastreams - https://ogcapi.bgs.ac.uk/collections/BGSSTADatastreams/items

Otherwise yes, you're going to need to somehow programmatically generate a config file.

[dthib]

Thanks for the reply. The data does not have the same structure/fields so using the same table is not possible. I am planning to specify a remote config file url which is updated by another process. At startup, pygeoapi is downloading the file. Thanks again, Best

[Moritz-Langer]

@dthib in the case of configuring a new table everytime this is quite straight forward after finding out which tables are relevant for your yaml file. (This however is quite specific to your database).

After finding out which tables/views etc are relevant, if there are new collections you need to append your config file (yaml) or rebuild entirely. You can do this quite easily using python and If necessary I can give you some guidance. You could also add to this script the restarting of pygeoapi if there are changes and run that as a cronjob.

Let me know if this is still relevant and we can discuss this further.

My understanding would be, that this is not particularly relevant to the pygeoapi core and is something custom.

Anyone else's thoughts?

[github-actions[bot]]

As per RFC4, this Issue has been inactive for 90 days. In order to manage maintenance burden, it will be automatically closed in 7 days.

[github-actions[bot]]

As per RFC4, this Issue has been closed due to there being no activity for more than 90 days.