distribution
distribution copied to clipboard
distribution
Upgrade AWS SDK to /v2
Description
The current s3 storage implementation depends on:
github.com/aws/aws-sdk-go v1.55.5
However, this SDK has been in maintenance mode since July 31, 2024 and will be EOL on July 31, 2025.
A bit more details here: Announcing end-of-support for AWS SDK for Go (v1) effective July 31, 2025
The supported SDK is the v2 version: https://github.com/aws/aws-sdk-go-v2
Would you be open to a PR to bump the supported AWS SDK in the storage implementation?
We welcome PRs, but I'm not sure we'll merge the SDK v2 bump into main before we do a stable v3 release.
It'd be a huge change. That said, yeah go ahead please, let's start a discussion at least
Hey!
We have been working on aws-sdk-v2 migration in our fork, the final MR in the series is here.
It should be reasonably on-par with the bugfixes/changes in the distribution/distribution as I ported all the relevant changes before the migration. We run tests against both MinIO and AWS S3 api. It should work with alternative provides but I haven't had a chance to verify it - any feedback is welcome!
Just like with Azure storage driver changes I am unable to prepare MR myself, but if you would be interested in porting it, I will be happy to answer all the questions.
Oh that's awesome! I haven't had a chance to look at the code, but would be happy to bridge the gap between the MR you have and a PR in this repository. (I also have testing in place for MinIO and AWS S3).
ps. not sure what you meant by "Just like with Azure storage driver changes I am unable to prepare MR myself".
Just like with Azure storage driver changes I am unable to prepare MR myself
I was referring to https://github.com/distribution/distribution/issues/4571
Oh that's awesome! I haven't had a chance to look at the code, but would be happy to bridge the gap between the MR you have and a PR in this repository.
This is great! Happy to answer any questions you may have.
(I also have testing in place for MinIO and AWS S3).
BTW, MinIO has issues with checksum handling compared to native AWS S3 API (issue, issue), checksum_disabled config option must be set to true in order to run container registry using MinIO as a backend while using aws-sdk-go-v2.
@mfridman so now that we have v3 feel free to hack on the PR for V2 upgrade.
@vespian Curious, why did you keep the v1 SDKs around instead of dropping it wholesale in favor of the /v2 SDK?
Note to self, migration guide is here: https://docs.aws.amazon.com/sdk-for-go/v2/developer-guide/migrate-gosdk.html
To give our customers a safe fallback and option to test things out before doing the switch.
In case of any troubles we can simply ask them to move back to v1 while we fix v2. The rewrite itself is pretty extensive and storage is the 2nd most critical part of the registry codebase IMO, right after authN/authZ.
We did the same approach (i.e. v1+v2) with Azure driver refactoring, but this is not necessary per se for migrating the codebase.
Gotcha, thanks for the info. Just gathering all the requirements. I'll chip away at this in my spare time, but if anyone else comes along with an implementation quicker that's also great!
For anyone wondering the SDKv2 broke many "S3 compatible providers" I have documented them here https://github.com/vmware-tanzu/velero/issues/8265
We ended up having to ship https://github.com/openshift/velero-plugin-for-legacy-aws for those who require v1 while the compatible providers work on updating to match v2 expectations.