aptly icon indicating copy to clipboard operation
aptly copied to clipboard

S3 publishing speedup

Open cavedon opened this issue 1 year ago • 2 comments

Description of the Change

When publishing to an S3 bucket two prefixes can be used: a prefix in the storage configuration (in aptly.conf), and a prefix in the publish configuration (in the aptly DB). When updating a publishing to an S3 bucket:

  • when using a prefix in the publish configuration, aptly will needlessly list the content of the whole bucket (or under the storage prefix, if configured)
  • if the number of files published in the bucket is very large, listing it may take a significant amount of time, because it is done sequentially in batches of 1000 keys.

This pull request addresses both issue.

Checklist

  • [x] unit-test added (if change is algorithm)
  • [x] functional test added/updated (if change is functional)
  • [x] man page updated (if applicable)
  • [ ] bash completion updated (if applicable)
  • [x] documentation updated
  • [x] author name in AUTHORS

cavedon avatar Apr 24 '23 22:04 cavedon

Seems the ci is broken: https://github.com/aptly-dev/aptly/issues/1173

randombenj avatar May 01 '23 14:05 randombenj

good idea, unfortunately aptly has moved to github.com/aws/aws-sdk-go-v2/service/s3 which makes rebasing this PR a bit difficult.

Would you mind rebasing it ?

neolynx avatar Apr 21 '24 10:04 neolynx