slow message loading when topic is compacted and filter is oldest

[wanis-fahmy]

Issue submitter TODO list

• [X] I've looked up my issue in FAQ
• [X] I've searched for an already existing issues here
• [X] I've tried running main-labeled docker image and the issue still persists there
• [X] I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

We have several compacted (cleanup policy: compact) topics in our clusters, when we try to load messages in any of them using the "oldest" filter, it takes a significant amount of time to load (around 1 minute or more). Worthy to note that this doesn't occur when selecting the "newest" filter.

The topic size is around 17MB.

We monitored the CPU and memory of kafbat and increased it significantly in efforts to get a faster load time, but it didn't help. The app logs and console logs looks normal

We have significantly larger topics that are not compacted and they load the messages instantly, regardless of which filter.

Expected behavior

messages in compacted topics are loaded faster, like normal topics that have delete cleanup policy

Your installation details

running the latest main 1196f9f docker image deployed in EKS as a normal k8s deployment without helm.

Steps to reproduce

create a compacted topic and load it with some significant data

Screenshots

Logs

No response

Additional context

No response

[github-actions[bot]]

Hi wanisfahmyDE! 👋

Welcome, and thank you for opening your first issue in the repo!

Please wait for triaging by our maintainers.

As development is carried out in our spare time, you can support us by sponsoring our activities or even funding the development of specific issues. Sponsorship link

If you plan to raise a PR for this issue, please take a look at our contributing guide.

[xuyuanz]

Similar problem we noticed: for compact topics, oldest is working fine for us. But when we switched to use newest, it took forever to load.

[iliax]

@Haarolean this should be fixed due to https://github.com/kafbat/kafka-ui/pull/406

[Haarolean]

@Haarolean this should be fixed due to #406

@wanisfahmyDE @xuyuanz please verify, the image is available under ghcr.io/kafbat/kafka-ui:main

[kapybro[bot]]

Further user feedback is requested. Please reply within 7 days or we might close the issue.

[wanis-fahmy]

hey @Haarolean, yes the image is ghcr.io/kafbat/kafka-ui:main. the commit SHA is 1196f9f

[kapybro[bot]]

Thanks for the additional feedback! We'll get back to your issue soon.

[Haarolean]

@wanisfahmyDE so does it work now?

[wanis-fahmy]

no it doesn't, I created the issue when we were running already on ghcr.io/kafbat/kafka-ui:main

[xuyuanz]

yes switched to use kafka-ui:main. Now it loading forever for "oldest" messages.

[SubhraB]

Similar problem we noticed: for compact topics, oldest is working fine for us. But when we switched to use newest, it took forever to load. yes switched to use kafka-ui:main. Now it loading forever for "oldest" messages.

Exact same thing we are facing, we are Newest takes forever with helm chart version 1.4.5. When switched to main, it was the other way round.

[Haarolean]

Probly 'other way around' issue is related to #249, lol. I'll hide these comments as this is most likely unrelated. This issue is open which means we haven't fixed that, yet. Please upvote the issue and stay tuned!