s3fs icon indicating copy to clipboard operation
s3fs copied to clipboard

Published 20 hours ago • verified publisher icon PyFilesystem

Check integrity during download/upload

Open CMCDragonkai opened this issue 5 years ago • 0 comments

I suspect s3fs doesn't check integirty during upload/download. This is sometimes causing problems.

The normal aws CLI does do this automatically, and we can pass an md5hash during the upload.

  • https://aws.amazon.com/premiumsupport/knowledge-center/data-integrity-s3/
  • https://docs.aws.amazon.com/cli/latest/topic/s3-faq.html

Perhaps there's a way to do this behind the scenes here? And this would mean when streaming the file to s3, we would first have to calculate the hash.

Or maybe this is out of the scope of s3fs.

CMCDragonkai avatar Apr 14 '20 02:04 CMCDragonkai