pulsar-spark icon indicating copy to clipboard operation
pulsar-spark copied to clipboard
verified publisher icon streamnative

[BUG] unnecessary subscription created when processing batches

Open hanguyen6 opened this issue 4 years ago • 0 comments

Describe the bug

Why do we need to create subscription when reader API is used and Spark manages offsets on its own. I can see new subscriptions got created and deleted by PulsarMetadataReader before and after committing a batch but dont see it useful. Is that legacy code and should be removed ?

Expected behavior

  • No subscription creation needed when reader API is used

Additional context Spark version 2.4.5 and Pulsar version 2.5.0

hanguyen6 avatar Mar 31 '21 14:03 hanguyen6