Does FlinkPulsarSource support multiple thread to consume one topic?

[Johnnyhooyo]

now it's seems to only single thread for each topic even partitioned topic.

[jianyun8023]

@Johnnyhooyo ~Yes, if you need faster consumption, you can try to increase the parallelism of flink job.~ Sorry, the number of Pulsar topic partitions and Flink parallelism is 1:1 .

[Johnnyhooyo]

@Johnnyhooyo Yes, if you need faster consumption, you can try to increase the parallelism of flink job.

In my job, the topic has 2 partition, and parallelism of source is 4. then just 2 thread consume the message.

I see the connector use reader to consume message, and the reader not support "subscriptionType.shared"

[Johnnyhooyo]

look at that, the parallelism of job is 4 @jianyun8023

[shulaoh]

hi , I hit same problem. I set the pulsar partition to 1, enable-key-hash-range to true, and use flink source with 4 threads to read it, only one thread can consume message.

seems shared read doesn't take effect!

[shulaoh]

just curious why this issue is not responded since it is a fundamental feature...

[jianyun8023]

@Johnnyhooyo @shulaoh Sorry, the number of Pulsar topic partitions and Flink parallelism is 1:1. This is because the connector uses the Pulsar Reader API to consume data, and the Reader uses exclusive mode to consume, so only one reader can consume a partition.

If you want to use Pulsar consume API to solve this limitation, you can read about Pulsar Source in Flink 1.14.