s3-utils
s3-utils copied to clipboard
concat should support parallel writing
The reason I was interested in this tool was to merge ~2TB (3gb chunks) of data without downloading it to a middleman server first. Doing this in serial will take too long. I'm not great with rust but I I ended up creating a copycat project in javascript to do this https://gitlab.com/joshwillik/s3-concat (in case you want to read my implementation)