Failed to acquire lock for block

[humengyu2012]

Alluxio Version: 2.9.2

Describe the bug Our workers have a lot of warning logs:

2023-04-11 13:20:57,743 WARN  BlockLockManager - Failed to acquire lock for block 452934501286 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:20:57,743 WARN  AsyncBlockRemover - Failed to remove block 452934501286 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-9, error Can not acquire lock to remove block 452934501286 for session -4 after 60000 ms
2023-04-11 13:21:08,808 WARN  BlockLockManager - Failed to acquire lock for block 452934501302 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:21:08,808 WARN  BlockLockManager - Failed to acquire lock for block 968263467008 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 98
2023-04-11 13:21:08,809 WARN  AsyncBlockRemover - Failed to remove block 452934501302 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-2, error Can not acquire lock to remove block 452934501302 for session -4 after 60000 ms
2023-04-11 13:21:08,809 WARN  AsyncBlockRemover - Failed to remove block 968263467008 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-0, error Can not acquire lock to remove block 968263467008 for session -4 after 60000 ms
2023-04-11 13:21:10,087 WARN  BlockLockManager - Failed to acquire lock for block 968263467011 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 88
2023-04-11 13:21:10,087 WARN  AsyncBlockRemover - Failed to remove block 968263467011 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-6, error Can not acquire lock to remove block 968263467011 for session -4 after 60000 ms
2023-04-11 13:21:17,498 WARN  BlockLockManager - Failed to acquire lock for block 968263467010 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 97
2023-04-11 13:21:17,498 WARN  AsyncBlockRemover - Failed to remove block 968263467010 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-3, error Can not acquire lock to remove block 968263467010 for session -4 after 60000 ms
2023-04-11 13:21:24,094 WARN  BlockLockManager - Failed to acquire lock for block 452934501179 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 213
2023-04-11 13:21:24,094 WARN  AsyncBlockRemover - Failed to remove block 452934501179 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-8, error Can not acquire lock to remove block 452934501179 for session -4 after 60000 ms
2023-04-11 13:21:40,388 WARN  BlockLockManager - Failed to acquire lock for block 452934500506 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:21:40,388 WARN  AsyncBlockRemover - Failed to remove block 452934500506 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-7, error Can not acquire lock to remove block 452934500506 for session -4 after 60000 ms
2023-04-11 13:21:41,295 WARN  BlockLockManager - Failed to acquire lock for block 452934500989 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 213
2023-04-11 13:21:41,295 WARN  AsyncBlockRemover - Failed to remove block 452934500989 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-5, error Can not acquire lock to remove block 452934500989 for session -4 after 60000 ms
2023-04-11 13:21:49,504 WARN  BlockLockManager - Failed to acquire lock for block 452934501048 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:21:49,504 WARN  AsyncBlockRemover - Failed to remove block 452934501048 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-4, error Can not acquire lock to remove block 452934501048 for session -4 after 60000 ms
2023-04-11 13:21:51,069 WARN  BlockLockManager - Failed to acquire lock for block 452934500976 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 213
2023-04-11 13:21:51,069 WARN  AsyncBlockRemover - Failed to remove block 452934500976 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-1, error Can not acquire lock to remove block 452934500976 for session -4 after 60000 ms
2023-04-11 13:21:57,743 WARN  BlockLockManager - Failed to acquire lock for block 452934501263 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:21:57,744 WARN  AsyncBlockRemover - Failed to remove block 452934501263 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-9, error Can not acquire lock to remove block 452934501263 for session -4 after 60000 ms
2023-04-11 13:22:08,809 WARN  BlockLockManager - Failed to acquire lock for block 452934500930 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 213
2023-04-11 13:22:08,809 WARN  BlockLockManager - Failed to acquire lock for block 452934501052 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 212
2023-04-11 13:22:08,809 WARN  AsyncBlockRemover - Failed to remove block 452934500930 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-2, error Can not acquire lock to remove block 452934500930 for session -4 after 60000 ms
2023-04-11 13:22:08,809 WARN  AsyncBlockRemover - Failed to remove block 452934501052 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-0, error Can not acquire lock to remove block 452934501052 for session -4 after 60000 ms
2023-04-11 13:22:10,088 WARN  BlockLockManager - Failed to acquire lock for block 452917723138 after 60000 MILLISECONDS.  session: -4, blockLockType: WRITE, lock reference count = 4990
2023-04-11 13:22:10,088 WARN  AsyncBlockRemover - Failed to remove block 452917723138 instructed by master. This is best-effort and will be tried later. threadName block-removal-service-6, error Can not acquire lock to remove block 452917723138 for session -4 after 60000 ms

To Reproduce Steps to reproduce the behavior (as minimally and precisely as possible)

Expected behavior A clear and concise description of what you expected to happen.

Urgency Describe the impact and urgency of the bug.

Are you planning to fix it Please indicate if you are already working on a PR.

Additional context Add any other context about the problem here.