[BUG] The metric of s3 object count keeps increasing with IllegalStateException in log when no producer&consumer

[keashem]

Version & Environment

based on automq 1.2.2，3 brokers were running in 3 node seperatly。the S3 WAL and S3 storage used CubeFS

cat /etc/os-release NAME="CentOS Linux" VERSION="7 (Core)" ID="centos" ID_LIKE="rhel fedora" VERSION_ID="7" PRETTY_NAME="CentOS Linux 7 (Core)" ANSI_COLOR="0;31" CPE_NAME="cpe:/o:centos:centos:7" HOME_URL="https://www.centos.org/" BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7" CENTOS_MANTISBT_PROJECT_VERSION="7" REDHAT_SUPPORT_PRODUCT="centos" REDHAT_SUPPORT_PRODUCT_VERSION="7"

What went wrong?

The metric of s3 object count keeps increasing with IllegalStateException in log when no producer&consumer

[2025-02-18 21:09:18,270] ERROR open test-topic-40 failed, retry open after 1s (kafka.log.streamaspect.ElasticUnifiedLog$) java.lang.IllegalStateException at kafka.log.streamaspect.ElasticLogLoader.deleteSegmentsIfLogStartGreaterThanLogEnd$1(ElasticLogLoader.scala:129) at kafka.log.streamaspect.ElasticLogLoader.recoverLog(ElasticLogLoader.scala:182) at kafka.log.streamaspect.ElasticLogLoader.load(ElasticLogLoader.scala:69) at kafka.log.streamaspect.ElasticLog$.apply(ElasticLog.scala:710)

[2025-02-18 21:09:18,270] ERROR [ElasticLog partition=test-topic-17 epoch=79] failed to open elastic log, trying to close streams. (kafka.log.streamaspect.ElasticLog$) java.lang.IllegalStateException at kafka.log.streamaspect.ElasticLogLoader.deleteSegmentsIfLogStartGreaterThanLogEnd$1(ElasticLogLoader.scala:129) at kafka.log.streamaspect.ElasticLogLoader.recoverLog(ElasticLogLoader.scala:182) at kafka.log.streamaspect.ElasticLogLoader.load(ElasticLogLoader.scala:69) at kafka.log.streamaspect.ElasticLog$.apply(ElasticLog.scala:710)

What should have happened instead?

The metric of s3 object count shouldn't increasing when no producer&consumer

How to reproduce the issue?

Additional information

Please attach any relevant logs, backtraces, or metric charts.

[daniel-y]

Hi, @keashem, thanks for your feedback. We'll look into this issue soon, also feel free to submit a pull request to fix it. And, managing an AutoMQ cluster can be challenging, so if you need any support from our team, we're happy to help. We can set up a WeChat group or Slack channel for you.