Bora Berke Sahin

Below are the results of a benchmark test of small files. With checksum verification, it slows down around %50. |Scenario| Summary | |:---|:---| | upload small files | 'PR:478' ran...

This PR is still in draft as it is depending on migrating to aws-sdk-go-v2. Even if migration to v2 happens, this implementation can still stay under discussion as it slows...

Current version of s5cmd uses `Content-MD5` when uploading objects to verify integrity of an object as explained [here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/checking-object-integrity.html#checking-object-integrity-md5). However, it does not check the integrity of an object on download....

Here is the benchmark results comparing master with this PR: ### Benchmark summary: |Scenarios | File Size | File Count | |:---|:---|:---| | small files | 1M | 10000 |...

Some updates about this PR: - aws-sdk-go-v2 doesn't natively work for google cloud. The issue can be seen [here](https://github.com/aws/aws-sdk-go-v2/issues/1816). A workaround is possible by removing the `content-encoding` but this may...

I have updated this PR and it is ready for review.

Any updates on getting this PR merged?

Hey @josh-fell, I've created this PR based on your comments [here](https://github.com/apache/airflow/pull/38001#issuecomment-2023850316). I would really appreciate it if you could take a look at it!

Hi @josh-fell, did you have a chance to look at this PR? Would appreciate your comments!

> > rerun endpoint does not accept body, which means parameters like steps_override, schema_override, threads_override, cause cannot be passed. Current implementation always uses rerun endpoint if retry_from_failure is set to...