[BUG] controller node - failed to mark destroy compacted objects

Open Aaron0930 opened this issue 9 months ago • 9 comments

Version & Environment

AutoMQ version : v1.2.1
Operating System : Amazon Linux 2023

What went wrong?

some nodes of automq cluster faced issue due to run out of bandwidth. After that, one of controller and brokers keep reporting errors event the cluster is healthy now

Additional information

Please attach any relevant logs, backtraces, or metric charts. broker error: [2025-04-07 11:34:12,883] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor)

controller error: [2025-04-07 11:34:12,883] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager),

Apr 07 '25 11:04 Aaron0930

try to restart impacted broker and controller nodes, it does not help. https://github.com/AutoMQ/automq/issues/765 look like similar issue, but the previous solution was for automq-for-rocketmq

Apr 07 '25 11:04 Aaron0930

Is there any other error log in controller.log?

Apr 07 '25 12:04 superhx

here is controller log [2025-04-07 11:34:12,883] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager)

Apr 07 '25 12:04 Aaron0930

Other error controller logs: [2025-04-07 12:08:59,961] ERROR [QuorumController id=0] Controller took 20164 µs to process write event: commitStreamObject (org.apache.kafka.controller.QuorumController)

[2025-04-07 12:20:10,312] ERROR [QuorumController id=0] object 23535896 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)

Apr 07 '25 12:04 Aaron0930

Could you get more logs about object 6495919, 154573100

Apr 15 '25 11:04 superhx

@superhx I am Aaron's colleague, here are some recent error logs.

message [2025-04-15 12:48:08,792] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:47:08,260] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:47:08,260] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:47:08,260] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:46:08,203] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:46:08,202] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:46:08,202] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:45:08,148] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:45:08,147] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:45:08,147] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:44:08,091] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:44:08,091] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:44:08,091] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:43:08,031] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:43:08,031] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:43:08,031] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:42:07,443] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:42:07,434] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:42:07,434] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:41:07,372] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:41:07,371] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:41:07,371] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:40:07,305] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:40:07,304] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:40:07,304] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:39:07,250] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:399) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:39:07,249] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:39:07,249] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:38:07,182] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:38:07,181] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:38:07,181] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:37:07,107] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:37:07,106] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:37:07,106] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:36:07,052] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:36:07,051] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:36:07,051] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:35:06,998] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:35:06,992] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:35:06,992] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:34:06,939] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:34:06,938] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:34:06,938] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:33:06,881] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:33:06,880] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:33:06,880] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:32:04,856] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:32:04,855] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:32:04,855] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:31:03,988] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:31:03,987] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:31:03,987] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:29:59,874] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:29:59,874] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:29:59,874] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:28:42,227] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=MINOR_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:399) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:28:42,226] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:28:42,226] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:27:40,050] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:27:40,050] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:27:40,050] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:26:39,281] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:26:39,280] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:26:39,280] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:25:38,408] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:25:38,407] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:25:38,407] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:24:38,350] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:24:38,348] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:24:38,347] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:23:38,231] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:23:38,231] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:23:38,231] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:22:37,526] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:22:37,525] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:22:37,525] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:21:37,469] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:21:37,472] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:21:37,469] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager) [2025-04-15 12:20:37,391] ERROR [QuorumController id=0] [CommitStreamObject]: failed to mark destroy compacted objects. compactedObjects=[6495919, 154573100], req=CommitStreamObjectRequestData(nodeId=2, nodeEpoch=1743911169270, objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, sourceObjectIds=[6495919, 154573100], streamEpoch=14, attributes=0, operations=[2, 2]) (org.apache.kafka.controller.stream.StreamControlManager) [2025-04-15 12:20:37,392] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],5035,type=CLEANUP_V1,req=CompactStreamObjectRequest{objectId=-1, objectSize=0, streamId=5035, startOffset=-1, endOffset=-1, streamEpoch=14, attributes=0, sourceObjectIds=[6495919, 154573100], operations=[DEEP_DELETE, DEEP_DELETE]} (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.cleanupExpiredObject(StreamObjectCompactor.java:250) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:168) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: org.apache.kafka.common.errors.s3.CompactedObjectsNotFoundException: The compacted objects are not found. [2025-04-15 12:20:37,391] ERROR [QuorumController id=0] object 6495919 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)

Apr 15 '25 12:04 jerome-j-20230331

@jerome-j-20230331

Are there any logs related to object 6495919 before the first 'failed to mark destroy compacted' happens?
Could you try to rolling restart all the controller nodes and recheck whether the error recover

Apr 15 '25 13:04 superhx

@superhx I restarted the three controller nodes one by one. The error "object 154377965 not exist when mark destroy object (org.apache.kafka.controller.stream.S3ObjectControlManager)" in controller.log has temporarily disappeared, but after manually rebalancing the cluster partitions, such errors still appear. Restarting the three controllers again can solve the problem, but other similar errors still occur:. server.log: [2025-04-16 07:19:57,393] INFO [CompactionManager id=2] Compaction started (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,394] INFO [CompactionManager id=2] Get 30 stream set objects from metadata (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,691] INFO [CompactionManager id=2] Throttle compaction read to 3492623 bytes/s, expect to complete in no less than 9min (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,691] INFO [CompactionManager id=2] Compact 30 stream set objects, total 30 objects to compact (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,691] INFO [CompactionManager id=2] 30 stream set objects as compact candidates, total compaction size: 1886016707 (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,694] INFO [CompactionAnalyzer id=2] 30 stream set objects to compact after filter (com.automq.stream.s3.compact.CompactionAnalyzer) [2025-04-16 07:19:57,711] INFO [CompactionAnalyzer id=2] Current stream num in stream set: 310, max: 20000, stream object num: 42, max: 10000 (com.automq.stream.s3.compact.CompactionAnalyzer) [2025-04-16 07:19:57,716] INFO [CompactionManager id=2] Analyze compaction plans complete, cost 25ms (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,716] INFO [CompactionManager id=2] Compaction plans: expect to generate 50 Stream Object, 1 stream set object with size 253553038 in 9 iterations, objects excluded: [] (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:57,717] INFO [CompactionManager id=2] Compaction progress 1/9, read from 30 stream set objects, total size: 209598660 (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:19:58,356] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6397, max=570.0, mean=107.1786775050805} callback={count=6397, max=2085.0, mean=87.62419884320775} ack={count=6397, max=65.0, mean=19.47412849773331} (kafka.server.streamaspect.ElasticKafkaApis) [2025-04-16 07:20:05,996] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=104857600/104857600, append={count=6456, max=206926.0, mean=21550.96375464684}, callback={count=6456, max=1.1073328E7, mean=1262797.1220570013}, ack={count=6263, max=220723.0, mean=20970.326201500877} (kafka.log.streamaspect.ElasticLog) [2025-04-16 07:20:57,342] WARN [GetStreamObjects]: stream: 4711, startOffset: 0, endOffset: 1454976172, limit: 2147483647, and search in metadataCache failed with exception: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null (kafka.log.stream.s3.metadata.StreamMetadataManager) [2025-04-16 07:20:57,343] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],4711,type=CLEANUP_V1,req=null (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:156) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null at kafka.log.stream.s3.metadata.StreamMetadataManager.lambda$getStreamObjects$8(StreamMetadataManager.java:182) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at kafka.log.stream.s3.metadata.StreamMetadataManager.getStreamObjects(StreamMetadataManager.java:187) at kafka.log.stream.s3.objects.ControllerObjectManager.getStreamObjects(ControllerObjectManager.java:274) ... 15 more [2025-04-16 07:20:58,042] INFO [CompactionManager id=2] Compaction progress 2/9, read from 30 stream set objects, total size: 209385517 (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:20:58,432] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6236, max=551.0, mean=114.69611930724824} callback={count=6236, max=2091.0, mean=93.22370109044259} ack={count=6236, max=83.0, mean=20.784637588197562} (kafka.server.streamaspect.ElasticKafkaApis) [2025-04-16 07:21:06,017] INFO log append cost, permitAcquireFail={count=0, max=0.0, mean=0.0}, remainingPermit=104857600/104857600, append={count=6295, max=159318.0, mean=22574.997776012708}, callback={count=6295, max=6848319.0, mean=1262421.6057188245}, ack={count=6147, max=159464.0, mean=21690.954612005855} (kafka.log.streamaspect.ElasticLog) [2025-04-16 07:21:57,344] WARN [GetStreamObjects]: stream: 4711, startOffset: 0, endOffset: 1454984131, limit: 2147483647, and search in metadataCache failed with exception: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null (kafka.log.stream.s3.metadata.StreamMetadataManager) [2025-04-16 07:21:57,344] ERROR [STREAM_OBJECT_COMPACT_FAIL],[UNEXPECTED],4711,type=CLEANUP_V1,req=null (com.automq.stream.s3.compact.StreamObjectCompactor) java.util.concurrent.ExecutionException: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) at com.automq.stream.s3.compact.StreamObjectCompactor.compact0(StreamObjectCompactor.java:156) at com.automq.stream.s3.compact.StreamObjectCompactor.compact(StreamObjectCompactor.java:116) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:418) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compactV1(S3StreamClient.java:402) at com.automq.stream.s3.S3StreamClient$StreamWrapper.compact(S3StreamClient.java:375) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$6(S3StreamClient.java:155) at java.base/java.util.ArrayList.forEach(ArrayList.java:1511) at com.automq.stream.s3.S3StreamClient.lambda$startStreamObjectsCompactions$7(S3StreamClient.java:155) at com.automq.stream.utils.ThreadUtils.lambda$wrapRunnable$0(ThreadUtils.java:74) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.kafka.metadata.stream.S3Object.getTimestamp()" because "objectMetadata" is null at kafka.log.stream.s3.metadata.StreamMetadataManager.lambda$getStreamObjects$8(StreamMetadataManager.java:182) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682) at kafka.log.stream.s3.metadata.StreamMetadataManager.getStreamObjects(StreamMetadataManager.java:187) at kafka.log.stream.s3.objects.ControllerObjectManager.getStreamObjects(ControllerObjectManager.java:274) ... 15 more [2025-04-16 07:21:57,564] INFO [CompactionManager id=2] Compaction progress 3/9, read from 30 stream set objects, total size: 209371118 (com.automq.stream.s3.compact.CompactionManager) [2025-04-16 07:21:58,434] INFO [KafkaApi-2] produce cost (in microseconds), produce={count=6642, max=1138.0, mean=115.77220716651611} callback={count=6642, max=1964.0, mean=92.86013249021379} ack={count=6642, max=124.0, mean=21.08130081300813} (kafka.server.streamaspect.ElasticKafkaApis)

Apr 16 '25 07:04 jerome-j-20230331

It seams the S3Object was deleted but the S3StreamObject wasn't deleted. It should not happen.

I will find the reason. You can delete streamId=4711 related topic to fix it.

Apr 16 '25 07:04 superhx