Ivan Yurchenko
Ivan Yurchenko
@jeqo I isolated all the reflection operations in a separate package + added a comment on this. Even when this becomes non-experimental, most likely the reflection magic stays: the AWS...
A bit harder than that. The Linux kernel requires direct IO to be aligned by the logical block size. So some padding is needed and all the metadata around it.
Hi @bingkunyangvungle Thanks for reporting this. This is a know issue originating from how the broker side handles things. And we're working on a workaround on the plugin level and...
Yeah, we're working on it
@HenryCaiHaiying The former supersedes the latter.
We are probably fine with the integrity guarantees object storages give. However, we should set MD5 digests during uploads (e.g. in S3; where it's supposed to be automatic per the...
For the record, I'm pretty happy with our current structure and created this ticket to formally have this discussion :)
> This way we can have use the manifest structure as an index for the segmentid-based structure. wdyt? Could you please elaborate? Also what we get with this?
I don't think this the prefixing is relevant for S3 any more (see my comment in https://github.com/aiven/tiered-storage-for-apache-kafka/issues/124#issuecomment-1571995894), but may be still relevant for GCS and Azure, it's unclear and poorly...
AWS S3 used to recommend a (pseudo-)randomized prefix to solve this issue. However some years ago they changed something drastically in how S3 scales and now it's not needed anymore....