Henry Haiying Cai comments

Results 46 comments of


                                            Henry Haiying Cai

S3 download speed is too slow when consumer needs to read data from S3 storage

@ivanyu Looping back to check the progress on this issue.

S3 download speed is too slow when consumer needs to read data from S3 storage

Thanks for the contribution, left a small comment in one of the PR

S3 download speed is too slow when consumer needs to read data from S3 storage

@jeqo with this PR: https://github.com/Aiven-Open/tiered-storage-for-apache-kafka/pull/429, do we still need #403 ?

S3 key/folder structure for performance

The easiest way to achieve this is is to add the segment_id at the beginning of the S3 path in `fileNamePrefix` method: https://github.com/aiven/tiered-storage-for-apache-kafka/blob/main/s3/src/main/java/io/aiven/kafka/tiered/storage/s3/S3StorageUtils.java#L34 Something like: s3://bucket/hashed-segment-id/topic/partition/segment.log

S3 key/folder structure for performance

The remote storage metadata is actually stored separately in a Kafka topic (the default implementation of KIP-405). We are supposed to find/resolve the remote location of the topic/partition/segment driven from...

S3 key/folder structure for performance

From the AWS doc, I don't see how how they improve the performance without prefix randomization. I remember read somewhere else that AWS can analyze the request pattern and starts...