GC thread shuts down slowly when compacting

[erobot]

BUG REPORT

Describe the bug

GC thread shuts down slowly when compacting.

To Reproduce

Steps to reproduce the behavior:

1. A bookie is major compacting.
2. Stop the bookie.
3. The bookie does not exit in a reasonable time.

Expected behavior

Bookie exits in a reasonable time.

Additional context

Version

4.15.4

Problem

I think the problem is that the stop flag running has been moved after compacting.compareAndSet. When running is not set to false, entry log continues to compact one after one and the shutdown thread is hard to set the compacting.

shutdown from 4.14.8: https://github.com/apache/bookkeeper/blob/965c3627328787e2750f41c2a4eda6c1d709d7c6/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L528-L535

shutdown from 4.15.4: https://github.com/apache/bookkeeper/blob/44f87efb8121a9f5428e7ddc70656aee85f563bf/bookkeeper-server/src/main/java/org/apache/bookkeeper/bookie/GarbageCollectorThread.java#L652-L665

Stack dump

component-shutdown-thread is waiting gc thread to shutdown and has been waiting for 5153.82s.

"component-shutdown-thread" #177 prio=5 os_prio=0 cpu=3148.35ms elapsed=5153.82s tid=0x00007fac601f4880 nid=0x14dee waiting on condition  [0x00007fb97d0e4000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep([email protected]/Native Method)
        at org.apache.bookkeeper.bookie.GarbageCollectorThread.shutdown(GarbageCollectorThread.java:662)
        - locked <0x00007fbcce3885f0> (a org.apache.bookkeeper.bookie.GarbageCollectorThread)
        at org.apache.bookkeeper.bookie.storage.ldb.SingleDirectoryDbLedgerStorage.shutdown(SingleDirectoryDbLedgerStorage.java:267)
        at org.apache.bookkeeper.bookie.storage.ldb.DbLedgerStorage.shutdown(DbLedgerStorage.java:222)
        at org.apache.bookkeeper.bookie.BookieImpl.shutdown(BookieImpl.java:865)
        at org.apache.bookkeeper.bookie.BookieImpl.shutdown(BookieImpl.java:834)
        at org.apache.bookkeeper.proto.BookieServer.shutdown(BookieServer.java:195)
        - locked <0x00007fbcce2863c0> (a org.apache.bookkeeper.proto.BookieServer)
        at org.apache.bookkeeper.server.service.BookieService.doClose(BookieService.java:101)
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.close(AbstractLifecycleComponent.java:123)
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$close$6(LifecycleComponentStack.java:154)
        at org.apache.bookkeeper.common.component.LifecycleComponentStack$$Lambda$370/0x00007fbc24476a90.accept(Unknown Source)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.close(LifecycleComponentStack.java:154)
        at org.apache.bookkeeper.common.component.ComponentStarter$ComponentShutdownHook.run(ComponentStarter.java:47)
        at java.lang.Thread.run([email protected]/Thread.java:840)