slow message loading when topic is compacted and filter is oldest

[wanisfahmyDE]

Issue submitter TODO list

• [X] I've looked up my issue in FAQ
• [X] I've searched for an already existing issues here
• [X] I've tried running main-labeled docker image and the issue still persists there
• [X] I'm running a supported version of the application which is listed here

Describe the bug (actual behavior)

We have several compacted (cleanup policy: compact) topics in our clusters, when we try to load messages in any of them using the "oldest" filter, it takes a significant amount of time to load (around 1 minute or more). Worthy to note that this doesn't occur when selecting the "newest" filter.

The topic size is around 17MB.

We monitored the CPU and memory of kafbat and increased it significantly in efforts to get a faster load time, but it didn't help. The app logs and console logs looks normal

We have significantly larger topics that are not compacted and they load the messages instantly, regardless of which filter.

Expected behavior

messages in compacted topics are loaded faster, like normal topics that have delete cleanup policy

Your installation details

running the latest main 1196f9f docker image deployed in EKS as a normal k8s deployment without helm.

Steps to reproduce

create a compacted topic and load it with some significant data

Screenshots

Logs

No response

Additional context

No response