pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[fix][broker] fix consume stuck of shared streaming dispatcher

Open coderzc opened this issue 3 years ago • 1 comments

Fixes #18314

Motivation

In #18289, we set sendInProgress to false only after the last entry in a batch of messages has been delivered, but this will make consume stuck, because StreamingEntryReader.pendingReads may is not empty when without available entries from the ledger and lead to ctx.isLast() always be false.

https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/streamingdispatch/StreamingEntryReader.java#L172-L177

Modifications

Because case of #18289 only happen when readType == ReadType.Replay, so use readType == ReadType.Normal || ctx.isLast() instead of ctx.isLast().

Documentation

  • [ ] doc
  • [ ] doc-required
  • [x] doc-not-needed
  • [ ] doc-complete

Matching PR in forked repository

PR in forked repository: https://github.com/coderzc/pulsar/pull/26

coderzc avatar Nov 03 '22 07:11 coderzc

@poorbarcode PTAL

coderzc avatar Nov 04 '22 06:11 coderzc

Codecov Report

Merging #18315 (5045881) into master (b31c5a6) will decrease coverage by 0.82%. The diff coverage is 40.44%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master   #18315      +/-   ##
============================================
- Coverage     46.98%   46.16%   -0.83%     
+ Complexity    10343     9936     -407     
============================================
  Files           692      667      -25     
  Lines         67766    65699    -2067     
  Branches       7259     7029     -230     
============================================
- Hits          31842    30332    -1510     
+ Misses        32344    31946     -398     
+ Partials       3580     3421     -159     
Flag Coverage Δ
unittests 46.16% <40.44%> (-0.83%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
.../org/apache/bookkeeper/mledger/impl/EntryImpl.java 69.62% <0.00%> (-11.27%) :arrow_down:
...lsar/broker/service/RedeliveryTrackerDisabled.java 50.00% <ø> (ø)
...va/org/apache/pulsar/broker/service/ServerCnx.java 48.91% <ø> (+0.22%) :arrow_up:
...ersistentStreamingDispatcherMultipleConsumers.java 0.00% <0.00%> (ø)
.../java/org/apache/pulsar/client/impl/ClientCnx.java 30.16% <ø> (ø)
...a/org/apache/pulsar/client/impl/TableViewImpl.java 0.00% <0.00%> (ø)
...ar/client/impl/conf/ProducerConfigurationData.java 84.70% <ø> (-0.18%) :arrow_down:
...va/org/apache/pulsar/client/impl/ConsumerImpl.java 15.07% <12.50%> (+0.03%) :arrow_up:
...keeper/mledger/impl/cache/RangeEntryCacheImpl.java 45.79% <22.44%> (-6.31%) :arrow_down:
...eeper/mledger/impl/cache/InflightReadsLimiter.java 30.35% <30.35%> (ø)
... and 93 more

codecov-commenter avatar Nov 07 '22 03:11 codecov-commenter

@codelipenghui @congbobo184 @Technoboy- @mattisonchao Could you take a look?

poorbarcode avatar Nov 09 '22 09:11 poorbarcode

@poorbarcode @codelipenghui PTAL

coderzc avatar Nov 11 '22 03:11 coderzc

/pulsarbot run-failure-checks

coderzc avatar Nov 14 '22 08:11 coderzc

@codelipenghui PTAL

coderzc avatar Nov 15 '22 02:11 coderzc