litdata icon indicating copy to clipboard operation
litdata copied to clipboard

Published 20 hours ago • verified publisher icon Lightning-AI

Add Support for Custom S3 Configuration in s5cmd

Open csy1204 opened this issue 4 months ago • 0 comments

🚀 Feature

Currently, custom S3 configurations are only supported in the S3 client. I would like to request that s5cmd also support custom S3 configurations to provide consistency across tools and enhance its flexibility for various use cases.

Motivation

https://github.com/Lightning-AI/litdata/blob/b9aa903bd9c98cd96ee989394fdaa1a38f8036f0/src/litdata/streaming/downloader.py#L52-L56

  • storage_options does not apply to s5cmd

Pitch

Alternatives

Additional context

#236

csy1204 avatar Oct 10 '24 07:10 csy1204