Adding support for processing microservices

[herzogrh]

Is your feature request related to a problem? Please describe. Right now, processes for the OGC API Processes can only be implemented in Python.

Describe the solution you'd like It would be great to connect the pygeoapi with processing microservices via eg. Websockets connections. In that way, it is possible to execute processes in other environments than in Python, but still make them accessible through pygeoapi's OGC API processes and job management. As an example, one could connect a node.js processing microservice with pygeoapi and run Insight Maker simulations. Another benefit is that one can dynamically add and remove processing microservices without having to recompile pygeoapi.

Once a microservice establishes a connection to the pygeoapi endpoint, it registers a process that is then added to the list of processes in pygeoapi. Whenever an execution request is made by a client, the proposed feature will forward the inputs through the eg. websocket connection and the microservice will start the calculation. Once the results are calculated, the microservice transmits the results back to be stored in the job management.

Benefits are, as pointed out by @geotom in #1484:

• to offload requests to remote workers
• scale processing horizontally
• a dynamic service registration/discovery

Describe alternatives you've considered Implementing a job management and OGC API Processing complient endpoints in every single programming language, which sounds like an awful lot of work to me.

Additional context

In the urban digital twin project Connected Urban Twins we identified the use case of multi-dimensional simulations running on various different modelling paradigms and software stacks. Therefore it is necessary to easily spin up and deploy various processing microservices via standardised APIs such as the OGC API Processes. As pygeoapi is a great implementation of OGC API processes and provides both an interface and an integrated job management, the feature to connect Websocket processing microservices would be greatly appreciated. We implemented a rough version ourselves which can be found here as a fork of the pygeoapi: Urban Model Server

[geotom]

Great initiative @herzogrh

I have a lot of ideas and past implementation experience for WPS. Making the microservice job management agnostic is definitely something I would prefer. A plugin could ship a default protocol/API leaving it open to users implement their own transports. I have already had a look to https://github.com/airtai/faststream.

Other brokers/queues I would like to see or might be useful are

• MQTT
• Websocket
• Celery

Additionally I would love to see

• schemas & validation
• service versions
• namespaces

[tomkralidis]

Keeping mind the pygeoapi focus on OGC API standards, note that Pub/Sub support is coming via the forthcoming OGC API - Pub/Sub efforts. This will essentially allow pygeoapi to publish/subscribe to a broker, which can then be tied loosely enough from/to/with other services.

[herzogrh]

Thanks for the info @tomkralidis! If I understand correctly, OGC API Pub/Sub would be the preferred way of subscribing to a processing microservice. I suppose that still, there would have to be a link between Pub/Sub and API Processes. So that if subscribed to a broker, pygeoapi would publish the process registered by the broker via its API processes endpoint.

[tomkralidis]

Sure. I don't have an overview/picture of the workflow, but in OGC API - Pub/Sub, pygeoapi would basically publish messages to a Pub/Sub broker (MQTT, AMQP, WebSockets, Kafka, etc.) that would trigger additional workflow.

[github-actions[bot]]

This Issue has been inactive for 90 days. As per RFC4, in order to manage maintenance burden, it will be automatically closed in 7 days.

[github-actions[bot]]

As per RFC4, this Issue has been closed due to there being no activity for more than 90 days.