[💡 FEATURE REQUEST]: Google Pub-Sub driver

[joaopbnogueira]

Plugin

JOBS

I have an idea!

Hey guys,

In addition to the currently supported jobs drivers, it would be great if Google Pub Sub was added to the list.

This would mean that we could use the cloud-native services of the two major cloud providers (AWS & GCP).

Thanks!

EDIT: Driver repo: https://github.com/roadrunner-server/google-pub-sub

[rustatian]

Hey, @joaopbnogueira 👋🏻 Nice feature request. I'll add to the v2023.2.0 milestone.

[rustatian]

Hey, @joaopbnogueira 👋🏻 Could you please add a few use cases of how do you plan to use this driver, what features you currently use, etc...

[joaopbnogueira]

Hi @rustatian I have to be honest, I haven't given it much thought, but to give you some context, the idea is that we're going to use Google PubSub at the org level for cross-service communication and this is where the need comes from. I'm mostly interested in the consumer part of things, and publishing would be a nice-to-have at this stage.

I imagine, that there will need to be a configuration section containing, the project id and subscription id at the very least, to be able to subscribe to a given topic, as well as options to configure the way of working (pull vs push). For pull subscriptions I assume the most interesting kind is "streaming pull" while for push an endpoint would somehow have to be provided.

Regarding IAM the simplest approach would probably be to provide a service account json file.

Other advanced scenarios could include:

• create and configure topics & subscriptions
• define schemas

Thank you having a look at this

[rustatian]

Thank you very much for your use-case 👍🏻 I guess, I finish working on this driver this week (work out of the work 😄), but as for the streaming part: technically, the stream would be spread over all workers, since from the google-pub-sub it's just a set of messages... For sure, you may use only 1 worker or use prefetch: 1 to have a model like: consume -> worker -> Ack -> Next -> Worker ...

[rustatian]

It's also hard to test that driver, I need to figure out how to use the emulator inside the GitHub actions:

• https://cloud.google.com/sdk/docs/components
• https://cloud.google.com/pubsub/docs/emulator
• https://github.com/google-github-actions/setup-gcloud

[joaopbnogueira]

Hey @rustatian just a quick disclaimer first that I've never used it, but why not a docker image? https://github.com/marcelcorso/gcloud-pubsub-emulator

Or at least using the Dockerfile as reference to how to set it up.

[rustatian]

Oh, thanks for pointing me to this 👍🏻 That will definitely help me, at least, with the local tests 😃

[rustatian]

Hey @joaopbnogueira 👋 Google pub-sub driver would be released with RR v2024.2.0. Docs will be updated soon, and you would be able to at least try this driver locally (I'll include how-to local configuration) or with real gpub-sub 😃 Repo is here: https://github.com/roadrunner-server/google-pub-sub

[joaopbnogueira]

Thank you a lot @rustatian 🚀