Support SparkV3.0 and Pulsar 2.6 +

[wenboma100]

Is your feature request related to a problem? Please describe. Current connector cannot support SparkV3.0 because of lots of changes brought by SparkV3.0;

Describe the solution you'd like The connector can support Spark V3.0 and Pulsar 2.6+ version;

Describe alternatives you've considered

Additional context

[nburkecogito]

hi everyone!

I'm curious as to if there has been any headway into getting this issue resolved? @jianyun8023 do you have a publicly available branch?

[reswqa]

hi everyone!

I'm curious as to if there has been any headway into getting this issue resolved? @jianyun8023 do you have a publicly available branch?

Very glad that you are paying attention to our project. We are mainly doing pulsar-flink connector's work recently, and spark has not invested much effort here. When the recent work is finished, I may take time to support spark3.0 datasource v2 api. Of course, if you are willing to contribute to this work, we will be very grateful!

[sijie]

@nburkecogito - What is your timeline for supporting SparkV3.0?

[gfeldman]

I took a stab at adapting the code here to a datasource v2. I am happy to share what I have. I have it reading my avro pulsar sources, but it is very much POC code. I am not sure if this should be a PR or there should be a separate branch or repo for the v2 source because the code is quite different.

[Aegeaner]

I have upgraded pulsar-spark to Spark 3.0 & Pulsar 2.7.0，please review. https://github.com/streamnative/pulsar-spark/pull/42 @sijie @jiazhai

[sijie]

@Aegeaner Thank you for working on that! We will take a look