pulsar-client-go icon indicating copy to clipboard operation
pulsar-client-go copied to clipboard

fix: seek race

Open nodece opened this issue 1 year ago • 1 comments

Motivation

Broker's seek logic:

  1. Close the consumer, not connection.
  2. Move the read position of the cursor.
  3. Return response to the client.

Seek issue:

  1. Seek and reconnect requests in the different goroutines, which should be executed serially.
  2. Message loss after seek operation based on time: the partitioned topic has multiple consumer partitions, when all seek request is done, the parent consumer will clean up the receive queue, and at the same time, the consumer partition re-reads the message asynchronous. when a message is received, the consumer partition will push the message to the parent consumer, there is a concurrency issue with message cleaning and message push.

Modifications

  • Use the same goroutine to perform the seek and reconnect.
  • The parent consumer pauses receiving messages from all consumer partitions before performing the seek operation and then cleans up the message queue, when the seek is done on the consumer partition, continues to push the message to the parent consumer.

nodece avatar Aug 01 '24 07:08 nodece

Could you review this PR again? @RobertIndie

nodece avatar Aug 11 '24 15:08 nodece