Apache Pulsar support in ArgoLabs Dataflow

[alexec]

In Dataflow, we currently have support for Kafka, Jetstream, NATS Streaming and other sources and sinks. It would be amazing to add support for Pulsar too. We know that we plenty of use cases in the OSS community and this would get a lot of love.

The Dataflow team are not Pulsar experts. My ask is - would anyone be interested in implementing the changes needed?

Here is a pull request for Jetstream, breaking it down:

1. Add manifests to install a dev Pulsar into Kubernetes cluster.
2. Add APIs to specify a Pulsar source and sink (e.g. the URL, authentication).
3. Implement the source and sink.
4. Write test infrastructure (e.g. check the right number of messages get written to a sink).
5. Write tests (e.g. does the right number of message get written, even if the pod is deleted).
6. Write examples.
7. Write docs.

I'll raise a ticket against the main repo, and the Go client repo (there might be engineers there with the exact skills needed).

[wolfstudy]

@alexec Good ideas, Maybe we can send a PIP or email for this change.

[alexec]

Awesome! What is a PIP please?

[wolfstudy]

Awesome! What is a PIP please?

Reference to:

https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-%28PIP%29 https://github.com/apache/pulsar/wiki/Pulsar-Improvement-Proposal-Template