Jorge Esteban Quilcate Otoya

Looking at https://github.com/aiven/tiered-storage-for-apache-kafka/issues/124, we should consider using `segmentid` as part of the prefix. The main downside I see is that we lose the hierarchical structure that could be valuable when...

> For the record, I'm pretty happy with our current structure and created this ticket to formally have this discussion :) I do as well--the hierarchical structure is easier to...

Sure, no priority for this until we have the RSM implementation merged into the new-implementation branch. Just keeping track of the discussion on #206, let's revisit later.

Thanks @hardy-luo! This is happening on the broker-side, so most probably not solvable within the plugin itself. 3.6.1 is also a very early version with Tiered Storage support; could you...

@hardy-luo thanks for reaching out. Could you share a bit more on the need to increase the throughput on the upload side? We were considering to allow this by parallelizing...

@catalinmer thanks for reporting this. Could you also share which Kafka version are you using? Haven't seem this NPE in the past. From the stacktrace the error seem to be...

Aha, now that we've confirmed the version I can see how this NPE is possible: First null could be assigned to the producer state file: https://github.com/apache/kafka/blob/726d0d604d794f528a6e68a9fddf4158aa6c5e54/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L936 and then it's used...

@catalinmer my understanding is that the older messages stay as they are while the broker is backward-compatible. In the case of Tiered Storage my guess is that this scenario was...

Sounds related to https://issues.apache.org/jira/browse/KAFKA-16511 -- could you try this on 3.7.1 and see if it's still an issue?

@bingkunyangvungle sorry for the late reply. I have failed to find some time to look into this. Will try again this week. In the meantime, there are a couple of...