flink-connector-pulsar icon indicating copy to clipboard operation
flink-connector-pulsar copied to clipboard

[FLINK-37299][Connector/Pulsar] Flink stateless startup cannot contin…

Open yebai1105 opened this issue 9 months ago • 2 comments

Purpose of the change

Controls whether Flink continues to consume from the location recorded by the consumer group or from the location specified by StartCursor when it starts from stateless.

1、If 'pulsar.source.resetSubscriptionCursor' = 'true', each time a Flink task is restarted, it will consume according to the consumption location specified by the StartCursor configuration. 2、If 'pulsar.source.resetSubscriptionCursor' = 'fasle', no matter what value 'StartCursor' is set to, each restart will start consuming from the location recorded by the consumer group.

Brief change log

The value of pulsar.source.resetSubscriptionCursor determines whether Flink continues to consume from the location recorded by the consumer group or from the location specified by StartCursor when it starts from stateless.

Verifying this change

This change is a minor change and don't have any tests.

Significant changes

(Please check any boxes [x] if the answer is "yes". You can first publish the PR and check them afterwards, for convenience.)

  • [ ] Dependencies have been added or upgraded
  • [ ] Public API has been changed (Public API is any class annotated with @Public(Evolving))
  • [ ] Serializers have been changed
  • [ ] New feature has been introduced
    • If yes, how is this documented? (not applicable / docs / JavaDocs / not documented)

yebai1105 avatar Feb 11 '25 08:02 yebai1105

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

boring-cyborg[bot] avatar Feb 11 '25 08:02 boring-cyborg[bot]

@yebai1105 If the StartCursor for consumption is set to "earliest", will there be any issues just by changing this part? Looking at the code, there is no other place where the cursor is set except here. It seems that the "subscriptionInitialPosition" needs to be set when creating the consumer in the PulsarPartitionSplitReader.

thomasg19930417 avatar Apr 10 '25 05:04 thomasg19930417

Awesome work, congrats on your first merged pull request!

boring-cyborg[bot] avatar Oct 17 '25 18:10 boring-cyborg[bot]