kafka icon indicating copy to clipboard operation
kafka copied to clipboard

KAFKA-17400: Added share fetch purgatory for delaying share fetch requests which cannot be completed because of reaching record lock partition limit

Open adixitconfluent opened this issue 1 year ago • 1 comments

About

Introduced a share fetch purgatory on the broker which delays share fetch requests that cannot be completed instantaneously. Introduced 2 new classes -

  1. DelayedShareFetch - Contains logic to instantaneously complete or force complete a share fetch request on timeout.
  2. DelayedShareFetchKey - Contains the key which can be used to watch the entries within the share fetch purgatory.
  3. ShareFetchUtils - This utility class contains functionalities required for post-processing once the replica manager fetch is completed.

There are many scenarios which can cause a share fetch request to be delayed and multiple scenarios when a delayed share fetch can be attempted to be completed. In this PR, we are only targeting the case when record lock partition limit is reached, the ShareFetch should wait for up to MaxWaitMs for records to be released

Testing

The code has been tested with the help of unit tests.

adixitconfluent avatar Aug 22 '24 18:08 adixitconfluent

Hi @junrao, @mumrah and @AndrewJSchofield, please re-review my code whenever you get a chance. Thanks.

adixitconfluent avatar Aug 29 '24 08:08 adixitconfluent

Hi @junrao, I have addressed your comments. Please re-review whenever you can. Thanks!

adixitconfluent avatar Aug 31 '24 15:08 adixitconfluent

The tests failing for JDK 21 and Scala 2.13 are unrelated to the PR. I ran the four of them locally and they all passed.

adixitconfluent avatar Sep 11 '24 17:09 adixitconfluent