bookkeeper
bookkeeper copied to clipboard
AutoRecovery memory leak problem.
BUG REPORT
In Auditor, we will checkAllLedger in the scheduled task scheduleCheckAllLedgersTask
.
https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/Auditor.java#L1333
In ReplicationWorker, we will check ledgerFragment to replicate.
https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/ReplicationWorker.java#L369
Both will invoke openLedgerNoRecovery
. It will register self to AbstractZkLedgerManager.
https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadOnlyLedgerHandle.java#L105-L107
In ReadOnlyLedgerHandler close, it will unregistrer self to AbstractZkLedgerManager, but it only remove listener in AbstractZkLedgerManager#listeners, not remove watcher in ZKWatchManager. So ZKWatchManager dataWatcher will be more and more.
https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java#L412-L426
There is a heap dump we found in product.
![image](https://user-images.githubusercontent.com/22524871/175764518-5449bd49-7740-4c23-b1c9-d754057e7bcd.png)