[Enhancement] Make cursor caching eligibility logic reactive since reads don't get cached until checkCursorsToCacheEntries has been called

[lhotari]

Search before asking

• [X] I searched in the issues and found nothing similar.

Motivation

Currently broker cache doesn't cache entries until checkCursorsToCacheEntries has been called. It get called as part of PulsarStats.updateStats which gets called every statsUpdateFrequencyInSecs (60s by default). This could be a problem when a topic gets assigned to a broker and a large number of consumer connect in a large fanout scenario. The read request deduplication solution (PendingReadsManager added in #17241) will handle the problem for consumers at the same point of consumption, but caching would be useful for consumers that are not consuming exactly at the same location.

Solution

Improve the cursor caching eligibility logic to be reactive instead of calculated every 60s.

Alternatives

No response

Anything else?

discussion thread about broker cache design: https://lists.apache.org/thread/xm095hnjo0cffbdy8ckysmzzm90gsbnp broker cache default tuning: #23466

Are you willing to submit a PR?

• [ ] I'm willing to submit a PR!

[lhotari]

A revisited broker entry cache solution is in progress in lhotari/pulsar#209

[lhotari]

"PIP-430: Pulsar Broker cache improvements: refactoring eviction and adding a new cache strategy based on expected read count", #24444 covers this