aptly
aptly copied to clipboard
S3 publishing speedup
Description of the Change
When publishing to an S3 bucket two prefixes can be used: a prefix in the storage configuration (in aptly.conf), and a prefix in the publish configuration (in the aptly DB). When updating a publishing to an S3 bucket:
- when using a prefix in the publish configuration, aptly will needlessly list the content of the whole bucket (or under the storage prefix, if configured)
- if the number of files published in the bucket is very large, listing it may take a significant amount of time, because it is done sequentially in batches of 1000 keys.
This pull request addresses both issue.
Checklist
- [x] unit-test added (if change is algorithm)
- [x] functional test added/updated (if change is functional)
- [x] man page updated (if applicable)
- [ ] bash completion updated (if applicable)
- [x] documentation updated
- [x] author name in
AUTHORS
Seems the ci is broken: https://github.com/aptly-dev/aptly/issues/1173
good idea, unfortunately aptly has moved to github.com/aws/aws-sdk-go-v2/service/s3 which makes rebasing this PR a bit difficult.
Would you mind rebasing it ?