AutoRecovery memory leak problem.

[horizonzy]

BUG REPORT

In Auditor, we will checkAllLedger in the scheduled task scheduleCheckAllLedgersTask.

https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/Auditor.java#L1333

In ReplicationWorker, we will check ledgerFragment to replicate.

https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/replication/ReplicationWorker.java#L369

Both will invoke openLedgerNoRecovery. It will register self to AbstractZkLedgerManager.

https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/client/ReadOnlyLedgerHandle.java#L105-L107

In ReadOnlyLedgerHandler close, it will unregistrer self to AbstractZkLedgerManager, but it only remove listener in AbstractZkLedgerManager#listeners, not remove watcher in ZKWatchManager. So ZKWatchManager dataWatcher will be more and more.

https://github.com/apache/bookkeeper/blob/677ccec3eb84f5be1b3556537871e14eb5e8359c/bookkeeper-server/src/main/java/org/apache/bookkeeper/meta/AbstractZkLedgerManager.java#L412-L426

There is a heap dump we found in product.