[RFC] Supporting Queues / Messaging Pipelines across Cloud Providers (Ex: AWS SQS)

[rsnk96]

Feature Request

A commonly used feature in Cloud Platforms is their message-passing/queueing services, such as

• AWS: SQS
• Azure: [Service-Bus, Storage Queues]
• GCP: [Cloud Tasks, Cloud Pub-Sub]

These are often used in conjunction with their other services, such as Storage (Ex: to send a message-notification automatically when a new file is uploaded to a Container/Bucket) and Compute (Ex: To pass information between different compute instances, or to take an action whenever a new file is uploaded to Storage). However, it is a pain right now to deal with the different CSP's syntaxes for using these queues, and would be convenient if we could centralize how we carry out at least basic operations like:

1. Read n messages from queue
2. Add new messages into the queue
3. Flush the queue / Delete n items from the queue

[rsnk96]

Hi @Kami Can you help me with how to get this feature request reviewed? (I saw your activity being the most recent, sorry if someone else is better equipped to review this request 😅 )

[Kami]

Yeah, I think libcloud.queue / libcloud.queing API would make sense.

Since queuing APIs are usually quite poll heavy (at least for reading / retrieving messages), I think it would also make sense to expose async API in addition to the sync one.

This would also make it a first abstraction in Libcloud which would native expose async APIs - which makes a lot of sense in this day and age. I know that some library users build their own async primitives on top of other Libcloud APIs which don't expose async methods.

Ideally the next step would be a proposal in a form of a PR which implement the base API. Implementation for at least 2-3 providers is also a good idea to make sure that the base API is generic enough and not biased towards a single provider (that's how we approached other APIs as well - base API + 3 driver implementations).

[rsnk96]

👍 will slowly develop a proposal with that approach in mind.

[rsnk96]

Ideally the next step would be a proposal in a form of a PR which implement the base API.

@Kami - can you clarify if you mean the 1st step would be to develop the end-user API, without any backend? (Just to sample the UX)

Also, can you clarify if the IRC Chat (#libcloud) is still the way to go for more immediate support, or would Github Issues / any other mode be the preferred option now?

[Kami]

@rsnk96 Correct, yeah.

But usually to develop a good and provider agnostic (as much as possible) end user API, it involves prototyping with at least 2-3 providers (to ensure API is not biased against a single provider or similar).

And for now, Github issues should work. When you think you have a proposal / prototype ready, feel free to share a link to it on the mailing list.