packaging icon indicating copy to clipboard operation
packaging copied to clipboard

Published 20 hours ago • verified publisher icon hhvm

`aws s3 sync` is slow

Open Atry opened this issue 2 years ago • 2 comments

We use aws s3 sync to upload deb packages from EBS to S3 https://github.com/hhvm/packaging/blob/6d301967b15491b4c504947ec9bf0215ffb1a803/bin/update-debianish-repo#L49-L50

According to the document, aws s3 sync will compare the content of all the 2TB files even when we just added 5GB files. It would be good to avoid checking the 2TB files.

Atry avatar Jul 21 '22 18:07 Atry

Even though it is still not ideal, I think Fred had partially optimized it, because previously we had to downloading the 2TB files before the updating, which is worse.

Atry avatar Jul 21 '22 18:07 Atry

Why not mount S3 with s3fs and directly run reprepro in the mounted path, instead of aws s3 sync?

Atry avatar Jul 21 '22 22:07 Atry