[Parent issue] Support for Apache Pulsar

[hpvd]

Since there is a continious demand for it, just an overview for current state of Support for Apache Pulsar:

• when looking into beams i/o connector matrix pulsar is not listed yet: https://beam.apache.org/documentation/io/connectors/

• but there is a sdk package Package org.apache.beam.sdk.io.pulsar https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/pulsar/package-summary.html

• in v2.39 of beam a relatively basic pulsar io was introduced: Implemented Apache PulsarIO https://beam.apache.org/blog/beam-2.39.0/ for details see: https://issues.apache.org/jira/browse/BEAM-8218

• there are some open issues in beam for pulsar: https://github.com/apache/beam/issues?q=is%3Aissue+pulsar+is%3Aopen

  • #30688
  • #30152
  • #22054
  • #22027
  • #22025
  • #21714
  • #21560
  • #31079

• there were also some non merged pull request marked as stale and automatically closed, which may be worth to have a look on https://github.com/apache/beam/pulls?q=is%3Apr+pulsar+is%3Aclosed+label%3Astale

  • #26410
  • #22375
  • #22028
  • #22026
  • #17473
  • #16749

• in general it would be a good longterm target, to reach the same quality of integration in Beam as there is for Kafka

[hpvd]

regarding the beginning of pulsar integration from https://www.youtube.com/watch?v=L_Pm2yKgA6g

[damccorm]

Hey @hpvd - have you been able to make any progress on this issue? I agree that the current state of the IO isn't great right now, so there's some good contribution opportunity :)