bookkeeper
bookkeeper copied to clipboard
apache
Bookie compaction halts without any errors or diagnostic information.
BUG REPORT
Describe the bug
Bookie compaction halts without any errors or diagnostic information, even with the DEBUG log level enabled.
We observed that every time it silently stopped, it was at entry log 1884, corresponding to the file 75c.log.
# {pod="pulsar-bookie-4",namespace="pulsar"}
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18113 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage - Ledger exists. ledger: 18120 : true
2025-04-22T09:12:37,546+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-22T09:12:37,547+0000 [GarbageCollectorThread-6-1] DEBUG org.apache.bookkeeper.bookie.DefaultEntryLogger - Recovering ledgers maps for log 1884 at offset: 282035685
2025-04-22T09:12:38,518+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again.
When the entry log 75c.log was moved to a different location and compaction was manually triggered, the bookie successfully reclaimed storage.
Naively checking the consistency of the affected file via:
/pulsar/bin/bookkeeper shell readlog 1884
did not reveal any obvious problems with the metadata.
We couldn't find any relevant information at the metrics level either.
The problem occurred again on a different bookie at a different time, with exactly the same symptoms and resolution.
To Reproduce
We cannot reliably reproduce the issue. The only noteworthy observation is that both impacted entry log files were closed during a bookie shutdown.
Expected behavior
- Compaction should not stop due to problems with a single file.
- If compaction halts, it should provide diagnostic information to help identify the issue.
- There should be a CLI tool or command to check the consistency of an entry log file (if such a tool exists, it may not be well-documented).
Screenshots
N/A
Additional context
Environment:
- Pulsar Version: 4.0.0
- Apache Bookie Version: 4.17.1
- Deployment: Deployed via Pulsar Helm Charts to Kubernetes with Karpenter (pods may be rescheduled multiple times a day).
Configuration:
gcWaitTime: "300000"
isForceGCAllowWhenNoSpace: "true"
majorCompactionInterval: "10800"
majorCompactionThreshold: "0.8"
minorCompactionInterval: "360"
minorCompactionThreshold: "0.2"
According to your log, Bookie goes to org.apache.bookkeeper.bookie.GarbageCollectorThread#extractMetaFromEntryLogs, but when executing org.apache.bookkeeper.bookie.GarbageCollectorThread#doGcEntryLogs, it does not print "Deleting entryLogId {} as it has no active ledgers!".
Therefore, I believe that your entryLog still has active ledgers and cannot be completely deleted.
@StevenLuMT Thank you for responding. I understand why you might think that way; however, please refer to the sample logs provided below and at the bottom.
We have experienced multiple attempts at compaction ending in the same manner—always extracting metadata from entry log 1884, followed by radio silence (even in DEBUG mode, as mentioned in the issue).
The system is active, and there should be data available for compaction, especially if these attempts are spread across multiple hours.
Additionally, as shown below, I reverted the movement of the potentially corrupted file, triggered compaction manually, and observed the same behaviour. I then moved the file to a different location and triggered compaction again. This time, the process moved forward and completed successfully:
# {pod="pulsar-bookie-4",namespace="pulsar"}
#
2025-04-23T07:19:57,718+0000 [vert.x-worker-thread-10] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Forced garbage collection triggered by thread: vert.x-worker-thread-10
2025-04-23T07:19:57,721+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Garbage collector thread forced to perform GC before expiry of wait time.
2025-04-23T07:19:57,729+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-23T07:19:59,317+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again.
2025-04-23T07:20:40,733+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-23T07:21:26,091+0000 [vert.x-worker-thread-3] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Forced garbage collection triggered by thread: vert.x-worker-thread-3
2025-04-23T07:21:26,092+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Garbage collector thread forced to perform GC before expiry of wait time.
2025-04-23T07:21:26,099+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Enter major compaction, suspendMajor false
2025-04-23T07:21:26,099+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Do compaction to compact those files lower than 0.8
2025-04-23T07:21:26,099+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets before compaction [10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 0, 0, 0, 2, 0, 0, 1, 0, 17]
2025-04-23T07:21:26,099+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction progress 0 / 3, current compaction entryLogId: 1981
2025-04-23T07:21:38,457+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 1981 after compaction
2025-04-23T07:21:38,544+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1981
2025-04-23T07:21:53,196+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 1950 after compaction
2025-04-23T07:21:53,252+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1950
2025-04-23T07:21:56,453+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Creating a new entry log file : createNewLog = false, reachEntryLogLimit = true
2025-04-23T07:21:56,455+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerBase - Flushing entry logger 1983 back to filesystem, pending for syncing entry loggers : [BufferedChannel{logId=1983, logFile=/pulsar/data/bookkeeper/ledgers/current/7bf.log, ledgerIdAssigned=-1}].
2025-04-23T07:21:56,457+0000 [pool-4-thread-1] INFO org.apache.bookkeeper.bookie.EntryLoggerAllocator - Created new entry log file /pulsar/data/bookkeeper/ledgers/current/7c1.log for logId 1985.
2025-04-23T07:22:08,155+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogManagerForSingleEntryLog - Synced entry logger 1983 to disk.
2025-04-23T07:22:08,182+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 1982 after compaction
2025-04-23T07:22:08,252+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1982
2025-04-23T07:22:08,254+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 0, 0, 0, 2, 0, 0, 1, 0, 17], compacted [0, 0, 0, 0, 2, 0, 0, 1, 0, 0]
2025-04-23T07:22:08,254+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - GarbageCollectorThread-6-1 Set forceGarbageCollection to false after force GC to make it forceGC-able again.
Logs with multiple attempts of compaction, spanning hours, newlines between the runs for clarity:
# {pod="pulsar-bookie-4"} |="Garbage"
2025-04-20T05:22:07,567+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-20T05:22:07,139+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-20T05:22:07,137+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-20T05:20:51,019+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-20T05:20:51,017+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-20T05:20:32,847+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-20T05:20:32,845+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-20T05:20:18,606+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-20T05:20:18,605+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-20T05:20:18,602+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-20T05:20:18,599+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-20T05:20:18,597+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-20T05:20:17,483+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-20T05:20:17,483+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-20T05:20:17,480+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-20T05:20:04,529+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-20T05:20:04,528+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-20T05:20:04,526+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-20T05:20:04,524+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-20T05:20:04,522+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-20T05:20:04,436+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1727.log : No ledgers map index found on entryLogId 1727
2025-04-20T05:20:04,435+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1727
2025-04-20T05:20:04,433+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1847
2025-04-20T05:20:04,433+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1921
2025-04-20T05:20:04,432+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1921 as it has no active ledgers!
2025-04-20T05:20:04,432+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1921.log : readerIndex(0) + length(4) exceeds writerIndex(0): PooledUnsafeDirectByteBuf(ridx: 0, widx: 0, cap: 1024)
2025-04-20T05:20:04,431+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1921
2025-04-20T05:20:04,218+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1920.log : No ledgers map index found on entryLogId 1920
2025-04-20T05:20:04,216+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1920
2025-04-20T05:10:03,651+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T05:10:03,651+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T05:10:03,645+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T05:04:20,795+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T05:04:20,795+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T05:04:20,790+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T04:58:09,456+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-20T04:58:09,028+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-20T04:58:09,026+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-20T04:56:52,908+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-20T04:56:52,906+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-20T04:56:34,736+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-20T04:56:34,735+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-20T04:56:20,498+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-20T04:56:20,496+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-20T04:56:20,493+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-20T04:56:20,490+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-20T04:56:20,487+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-20T04:56:19,373+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-20T04:56:19,372+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-20T04:56:19,370+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-20T04:56:07,331+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-20T04:56:07,330+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-20T04:56:07,326+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-20T04:56:07,324+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-20T04:56:07,322+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-20T04:56:07,320+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1727.log : No ledgers map index found on entryLogId 1727
2025-04-20T04:56:07,318+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1727
2025-04-20T04:56:07,309+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1847
2025-04-20T04:46:06,837+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T04:46:06,837+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T04:46:06,832+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T04:35:17,100+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T04:35:17,100+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T04:35:17,095+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T02:22:42,881+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-20T02:22:42,453+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-20T02:22:42,451+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-20T02:21:26,332+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-20T02:21:26,331+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-20T02:21:08,161+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-20T02:21:08,159+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-20T02:20:53,918+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-20T02:20:53,917+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-20T02:20:53,915+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-20T02:20:53,913+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-20T02:20:53,911+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-20T02:20:52,798+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-20T02:20:52,797+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-20T02:20:52,794+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-20T02:20:40,835+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-20T02:20:40,833+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-20T02:20:40,831+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-20T02:20:40,829+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-20T02:20:40,822+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-20T02:10:40,128+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T02:10:40,128+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T02:10:40,121+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T02:03:52,232+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-20T02:03:51,804+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-20T02:03:51,802+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-20T02:02:35,684+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-20T02:02:35,682+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-20T02:02:17,512+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-20T02:02:17,510+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-20T02:02:03,271+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-20T02:02:03,269+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-20T02:02:03,267+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-20T02:02:03,265+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-20T02:02:03,262+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-20T02:02:02,149+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-20T02:02:02,148+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-20T02:02:02,145+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-20T02:01:50,185+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-20T02:01:50,181+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-20T02:01:50,179+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-20T02:01:50,176+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-20T02:01:50,166+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-20T01:51:49,632+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T01:51:49,632+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T01:51:49,627+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-20T00:36:48,074+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-20T00:36:47,646+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-20T00:36:47,644+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-20T00:35:31,525+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-20T00:35:31,524+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-20T00:35:13,355+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-20T00:35:13,352+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-20T00:34:59,111+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-20T00:34:59,110+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-20T00:34:59,108+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-20T00:34:59,107+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-20T00:34:59,105+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-20T00:34:57,991+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-20T00:34:57,990+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-20T00:34:57,988+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-20T00:34:45,952+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-20T00:34:45,951+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-20T00:34:45,949+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-20T00:34:45,947+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-20T00:34:45,945+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-20T00:34:45,945+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1855
2025-04-20T00:34:45,942+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1855 as it has no active ledgers!
2025-04-20T00:34:45,940+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1855
2025-04-20T00:34:45,938+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1727.log : No ledgers map index found on entryLogId 1727
2025-04-20T00:34:45,937+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1727
2025-04-20T00:34:45,937+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1854
2025-04-20T00:34:45,934+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1854 as it has no active ledgers!
2025-04-20T00:34:45,932+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1854
2025-04-20T00:34:45,932+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1853
2025-04-20T00:34:45,928+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1853 as it has no active ledgers!
2025-04-20T00:34:45,925+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1853
2025-04-20T00:34:45,925+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1852
2025-04-20T00:34:45,923+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1852 as it has no active ledgers!
2025-04-20T00:34:45,921+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1852
2025-04-20T00:34:45,921+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1850
2025-04-20T00:34:45,916+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1850 as it has no active ledgers!
2025-04-20T00:34:45,913+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1850
2025-04-20T00:34:45,903+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1847
2025-04-20T00:24:45,333+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-20T00:24:45,333+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-20T00:24:45,327+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T23:27:47,624+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T23:27:47,197+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T23:27:47,195+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T23:26:31,076+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T23:26:31,074+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T23:26:12,904+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T23:26:12,903+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T23:25:58,663+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T23:25:58,661+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T23:25:58,660+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T23:25:58,657+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T23:25:58,655+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T23:25:57,545+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T23:25:57,542+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T23:25:57,538+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T23:25:45,577+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T23:25:45,574+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T23:25:45,570+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T23:25:45,567+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T23:25:45,561+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T23:15:45,117+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T23:15:45,117+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T23:15:45,112+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T23:10:35,001+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T23:10:35,000+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T23:10:34,977+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T23:05:28,922+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T23:05:28,922+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T23:05:28,912+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T22:55:41,529+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T22:55:41,529+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T22:55:41,524+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T22:50:21,249+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T22:50:21,245+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T22:50:21,235+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T22:41:56,477+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T22:41:56,476+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T22:41:56,472+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T22:38:03,670+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T22:38:03,670+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T22:38:03,665+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T22:09:37,611+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T22:09:37,610+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1882
2025-04-19T22:09:37,609+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1882 as it has no active ledgers!
2025-04-19T22:09:36,661+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T22:09:36,660+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T22:09:36,232+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T22:09:36,230+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T22:08:20,111+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T22:08:20,110+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T22:08:01,939+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T22:08:01,938+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T22:07:47,697+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T22:07:47,696+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T22:07:47,694+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T22:07:47,692+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T22:07:47,690+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T22:07:46,577+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T22:07:46,575+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T22:07:46,573+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T22:07:34,613+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T22:07:34,610+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T22:07:34,608+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T22:07:34,606+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T22:07:34,598+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T21:57:33,752+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T21:57:33,752+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T21:57:33,745+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T18:47:25,711+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T18:47:24,763+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T18:47:24,762+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T18:47:24,334+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T18:47:24,332+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T18:46:08,216+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T18:46:08,214+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T18:46:08,214+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1876
2025-04-19T18:46:08,213+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1876 as it has no active ledgers!
2025-04-19T18:46:08,212+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1876.log : Not all ledgers were found in ledgers map index. expected: 1768711269 -- found: 0 -- entryLogId: 1876
2025-04-19T18:46:08,211+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1876
2025-04-19T18:45:50,042+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T18:45:50,040+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T18:45:35,800+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T18:45:35,798+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T18:45:35,796+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T18:45:35,794+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T18:45:35,792+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T18:45:34,679+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T18:45:34,678+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T18:45:34,675+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T18:45:22,715+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T18:45:22,713+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T18:45:22,711+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T18:45:22,709+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T18:45:22,704+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T18:35:21,844+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T18:35:21,843+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T18:35:21,832+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T18:30:12,747+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T18:30:12,742+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T18:30:12,719+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T18:23:11,044+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T18:23:10,096+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T18:23:10,095+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T18:23:09,667+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T18:23:09,665+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T18:21:53,546+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T18:21:53,546+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T18:21:53,545+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1876.log : Not all ledgers were found in ledgers map index. expected: 1768711269 -- found: 0 -- entryLogId: 1876
2025-04-19T18:21:53,544+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1876
2025-04-19T18:21:35,374+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T18:21:35,372+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T18:21:21,131+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T18:21:21,130+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T18:21:21,128+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T18:21:21,126+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T18:21:21,124+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T18:21:20,010+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T18:21:20,009+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T18:21:20,007+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T18:21:08,038+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T18:21:08,037+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T18:21:08,034+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T18:21:08,030+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T18:21:08,016+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T18:11:07,264+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T18:11:07,263+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T18:11:07,259+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T18:02:54,555+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T18:02:54,554+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T18:02:54,549+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T17:56:26,360+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T17:56:26,360+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T17:56:26,355+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T17:52:48,526+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T17:52:48,526+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T17:52:48,521+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T17:47:00,608+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T17:46:59,660+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T17:46:59,659+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T17:46:59,232+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T17:46:59,230+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T17:45:43,113+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T17:45:43,112+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T17:45:43,111+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1876.log : Not all ledgers were found in ledgers map index. expected: 1768711269 -- found: 0 -- entryLogId: 1876
2025-04-19T17:45:43,110+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1876
2025-04-19T17:45:43,110+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log metadata for 1874
2025-04-19T17:45:43,108+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Deleting entryLogId 1874 as it has no active ledgers!
2025-04-19T17:45:42,354+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1874.log : No ledgers map index found on entryLogId 1874
2025-04-19T17:45:42,353+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1874
2025-04-19T17:45:24,183+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T17:45:24,181+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T17:45:09,944+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T17:45:09,942+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T17:45:09,940+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T17:45:09,938+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T17:45:09,933+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T17:45:08,823+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T17:45:08,821+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T17:45:08,817+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T17:44:56,856+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T17:44:56,855+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T17:44:56,852+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T17:44:56,850+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T17:44:56,845+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T17:34:56,345+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T17:34:56,344+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T17:34:56,339+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T16:12:30,472+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T16:12:29,525+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T16:12:29,523+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T16:12:29,094+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T16:12:29,093+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T16:11:12,974+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T16:11:12,974+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T16:11:12,973+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1876.log : Not all ledgers were found in ledgers map index. expected: 1768711269 -- found: 0 -- entryLogId: 1876
2025-04-19T16:11:12,972+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1876
2025-04-19T16:11:12,218+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1874.log : No ledgers map index found on entryLogId 1874
2025-04-19T16:11:12,217+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1874
2025-04-19T16:10:54,046+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T16:10:54,045+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T16:10:39,808+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T16:10:39,806+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T16:10:39,805+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T16:10:39,802+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T16:10:39,797+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T16:10:38,686+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T16:10:38,684+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T16:10:38,680+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T16:10:26,720+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T16:10:26,717+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T16:10:26,715+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T16:10:26,713+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T16:10:26,706+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T16:00:25,212+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T16:00:25,212+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T16:00:25,207+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
2025-04-19T15:15:24,289+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1884
2025-04-19T15:15:23,341+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1882.log : No ledgers map index found on entryLogId 1882
2025-04-19T15:15:23,340+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1882
2025-04-19T15:15:22,911+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1880.log : Negative position
2025-04-19T15:15:22,910+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1880
2025-04-19T15:14:06,792+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1878.log : Negative position
2025-04-19T15:14:06,791+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1878
2025-04-19T15:14:06,790+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1876.log : Not all ledgers were found in ledgers map index. expected: 1768711269 -- found: 0 -- entryLogId: 1876
2025-04-19T15:14:06,789+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1876
2025-04-19T15:14:06,036+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1874.log : No ledgers map index found on entryLogId 1874
2025-04-19T15:14:06,034+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1874
2025-04-19T15:13:47,864+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1872.log : Not all ledgers were found in ledgers map index. expected: 1969555200 -- found: 0 -- entryLogId: 1872
2025-04-19T15:13:47,862+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1872
2025-04-19T15:13:33,626+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1869.log : Not all ledgers were found in ledgers map index. expected: 793994612 -- found: 0 -- entryLogId: 1869
2025-04-19T15:13:33,625+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1869
2025-04-19T15:13:33,621+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1868
2025-04-19T15:13:33,617+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1867
2025-04-19T15:13:33,614+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1866
2025-04-19T15:13:32,502+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1738.log : Negative position
2025-04-19T15:13:32,500+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1738
2025-04-19T15:13:32,497+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1862
2025-04-19T15:13:20,536+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.DefaultEntryLogger - Failed to get ledgers map index from: 1860.log : No ledgers map index found on entryLogId 1860
2025-04-19T15:13:20,535+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1860
2025-04-19T15:13:20,533+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1859
2025-04-19T15:13:20,531+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1858
2025-04-19T15:13:20,526+0000 [GarbageCollectorThread-6-1] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Extracting entry log meta from entryLogId: 1857
2025-04-19T15:03:19,998+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Major Compaction : enabled=true, threshold=0.8, interval=10800000
2025-04-19T15:03:19,998+0000 [main] INFO org.apache.bookkeeper.bookie.GarbageCollectorThread - Minor Compaction : enabled=true, threshold=0.2, interval=360000
2025-04-19T15:03:19,987+0000 [main] INFO org.apache.bookkeeper.bookie.ScanAndCompareGarbageCollector - Over Replicated Ledger Deletion : enabled=true, interval=86400000, maxConcurrentRequests=1000
Also please see metrics of space reclaimed by compaction, there is a gap and huge jump once the file is moved:
I roughly understand your description. You mean that the damaged files may affect the progress of GC?
@StevenLuMT Yeah, that's how it looks to me. Obviously I can't share the whole file, but are there any commands to verify its consistency?
execute /pulsar/bin/bookkeeper shell readlog 1884 to get all Lid and Eid
Then execute /pulsar/bin/bookkeeper shell readledger to read the data.
I understand that readledger will check the index and entryLog, which will verify all files. If these exist, then it can be ruled out that it is a data problem.
@StevenLuMT I've tried, but seems like the ledger is long gone.:
kubectl exec -n pulsar pod/pulsar-bookie-4 -- /pulsar/bin/bookkeeper shell readlog 1884 > ./1884_meta.log
# there are 7005 entries
❯ grep Lid 1884_meta.log | cut -d" " -f8,9 | wc -l
7005
# all for the same ledger 17772
❯ grep Lid 1884_meta.log | cut -d" " -f8,9 | grep Lid=17772 | wc -l
7005
# entries start at 0 till 7004
❯ grep Lid 1884_meta.log | cut -d" " -f8,9 | grep Lid=17772 | head
Lid=17772, Eid=0,
Lid=17772, Eid=1,
Lid=17772, Eid=2,
Lid=17772, Eid=3,
Lid=17772, Eid=4,
Lid=17772, Eid=5,
Lid=17772, Eid=6,
Lid=17772, Eid=7,
Lid=17772, Eid=8,
Lid=17772, Eid=9,
❯ grep Lid 1884_meta.log | cut -d" " -f8,9 | grep Lid=17772 | tail
Lid=17772, Eid=6995,
Lid=17772, Eid=6996,
Lid=17772, Eid=6997,
Lid=17772, Eid=6998,
Lid=17772, Eid=6999,
Lid=17772, Eid=7000,
Lid=17772, Eid=7001,
Lid=17772, Eid=7002,
Lid=17772, Eid=7003,
Lid=17772, Eid=7004,
# reading the ledger 17772 entries from 0 to 7004 for bookie 4
kubectl exec -n pulsar pod/pulsar-bookie-4 -- /pulsar/bin/bookkeeper shell readledger -l 17772 -fe 0 -le 7005 -b pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181 > ./1884_entrylog_ledger_17772_data.log
2025-04-24T09:13:31,724+0000 [BookieClientScheduler-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand - Failed to read entry 0 -- No such ledger exists on Bookies 2025-04-24T09:13:31,729+0000 [main] ERROR org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand - Error future.get while reading entries from ledger 17772 java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?] ... 2025-04-24T09:13:38,371+0000 [BookieClientScheduler-OrderedExecutor-0-0] ERROR org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand - Failed to read entry 7004 -- No such ledger exists on Bookies 2025-04-24T09:13:38,371+0000 [main] ERROR org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand - Error future.get while reading entries from ledger 17772 java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
If I don't specify bookie to read from like:
/pulsar/bin/bookkeeper shell readledger -l 17772 -fe 0 -le 7005
it still fails:
Caused by: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
# additional I've tried to read the entrylog file with message body
kubectl exec -n pulsar pod/pulsar-bookie-4 -- /pulsar/bin/bookkeeper shell readlog 1884 -m > ./1884_meta_with_message.log
# when filtering out sensitive data and metadata there were no traces of any errors
All parameters are assigned, refer to the following example:
bookieId=***:3181
lid=17772
eidStart=0
eidEnd= 7005
./bin/bookkeeper shell readledger -bookie $bookieId -msg -ledgerid $lid -firstentryid $eidStart -lastentryid $eidEnd
@StevenLuMT I assume you meant (flags with double dash, readledger command):
bookieId=pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181
lid=17772
eidStart=0
eidEnd=7005
/pulsar/bin/bookkeeper shell readledger --bookie $bookieId --msg --ledgerid $lid --firstentryid $eidStart --lastentryid $eidEnd
Same error for all entries:
2025-04-24T13:23:03,922+0000 [main] ERROR org.apache.bookkeeper.tools.cli.commands.bookie.ReadLedgerCommand - Error future.get while reading entries from ledger 17772 java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsException: No such ledger exists on Bookies
We have a pretty strict retention on pulsar topics, so it's not surprising that it was removed from metadata.
export BOOKIE_CONF=/pulsar/conf/bookkeeper.conf
lid=17772
./bin/bookkeeper shell ledgermetadata -ledgerid $lid |grep LedgerMetadata
Get the ensembles where the ledgerId is located
export BOOKIE_CONF=/pulsar/conf/bookkeeper.conf
# bookieId cannot be customized and must be one of the ensembles in the previous command.
bookieId=pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181
lid=17772
./bin/bookkeeper shell listledgers -meta -bookieid $bookieId |grep $lid
Use this command to confirm whether the bookieId contains the ledgerId. Confirm that the correct Bookie node is requested.
export BOOKIE_CONF=/pulsar/conf/bookkeeper.conf
# bookieId cannot be customized and must be one of the ensembles in the previous command.
bookieId=pulsar-bookie-4.pulsar-bookie.pulsar.svc.cluster.local:3181
lid=17772
eidStart=0
eidEnd=7005
/pulsar/bin/bookkeeper shell readledger --bookie $bookieId --msg --ledgerid $lid --firstentryid $eidStart --lastentryid $eidEnd
Finally, execute this
@StevenLuMT
pulsar-bookie-4:~/bookkeeper/ledgers/current$ lid=17772
pulsar-bookie-4:~/bookkeeper/ledgers/current$ /pulsar/bin/bookkeeper shell ledgermetadata -ledgerid $lid
...
2025-04-25T07:11:22,398+0000 [main] ERROR org.apache.bookkeeper.bookie.BookieShell - Got an exception
com.google.common.util.concurrent.UncheckedExecutionException: java.util.concurrent.ExecutionException: org.apache.bookkeeper.client.BKException$BKNoSuchLedgerExistsOnMetadataServerException: No such ledger exists on Metadata Server
Some random fresh ledger just to confirm the command works:
pulsar-bookie-4:~/bookkeeper/ledgers/current$ /pulsar/bin/bookkeeper shell ledgermetadata -ledgerid 18374 | grep LedgerMetadata
2025-04-25T07:08:11,871+0000 [main] INFO org.apache.bookkeeper.tools.cli.commands.client.LedgerMetaDataCommand - LedgerMetadata{formatVersion=3, ensembleSize=3, writeQuorumSize=3, ackQuorumSize=2, state=OPEN, digestType=CRC32C, REDACTED
So still the same, we had some experience with these components already, so I'm pretty sure it's just not there, checking from metadata side as well:
[zk: pulsar-zookeeper:2181(CONNECTED) 11] ls /ledgers/00/0001/L7772 Node does not exist: /ledgers/00/0001/L7772
It shouldn't be a problem during compaction right? If bookie discovers that entries are not a part of active ledger they should be considered for removal during compaction if threshold is met.
Well, currently it seems that ledgerId=17772 has been expired and cleared, which is in line with expectations; there are other ledgerIds in the current entrgLog?
@StevenLuMT Current entryLog has multiple different ledgerIds mentioned in the metadata. Our cluster is not massive, but we have multiple topics that should be constantly active. It feels strange that entrylog 1884 had only 17772 ledger entries.
Well, the size is indeed a problem, but it may also be because the gc task finally merged the data.