velero
velero copied to clipboard
vmware-tanzu
Aws Pod Identity provider not working
What steps did you take and what happened:
I configured an IAM Role and bind it to the velero service account using EKS Pod Identity.
I deployed the velero server with the helm chart and the following errors happened :
time="2024-02-13T09:55:13Z" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/1/0, BackupStorageLocation \"default\" is unavailable: rpc error: code = Unknown desc = could not load config: invalid endpoint host, \"169.254.170.23\", only loopback hosts are allowed)" controller=backup-storage-location logSource="pkg/controller/backup_storage_location_controller.go:178"
What did you expect to happen:
The server should be able to retrieve AWS credentials using the pod identity agent.
Anything else you would like to add:
This seems to be an SDK version issue, AWS provides the minimum SDK version to use for Pod Identity : https://docs.aws.amazon.com/eks/latest/userguide/pod-id-minimum-sdk.html.
Command used to deploy the server
helm upgrade velero vmware-tanzu/velero \
--namespace claranet-system \
--create-namespace \
--set credentials.useSecret=false \
--set configuration.backupStorageLocation[0].name=default \
--set configuration.backupStorageLocation[0].provider=aws \
--set configuration.backupStorageLocation[0].bucket=<bucket_name> \
--set configuration.backupStorageLocation[0].config.region=eu-west-3 \
--set configuration.volumeSnapshotLocation[0].name=default \
--set configuration.volumeSnapshotLocation[0].provider=aws \
--set configuration.volumeSnapshotLocation[0].config.region=eu-west-3 \
--set initContainers[0].name=velero-plugin-for-aws \
--set initContainers[0].image=velero/velero-plugin-for-aws:v1.9.0 \
--set initContainers[0].volumeMounts[0].mountPath=/target \
--set initContainers[0].volumeMounts[0].name=plugins
Environment:
- Velero version (use
velero version): 1.13.0 - Velero features (use
velero client config get features): NOT SET - Kubernetes version (use
kubectl version): v1.29.0-eks-c417bb3 - Kubernetes installer & version: EKS
- Cloud provider or hardware configuration: AWS
- OS (e.g. from
/etc/os-release):
Vote on this issue!
This is an invitation to the Velero community to vote on issues, you can see the project's top voted issues listed here.
Use the "reaction smiley face" up to the right of this comment to vote.
- :+1: for "I would like to see this bug fixed as soon as possible"
- :-1: for "There are more important bugs to focus on right now"
Thanks !
Charles
Is there an update to this issue? Seems as though just bumping the AWS SDK to a minimum release-2023-11-14 would solve this issue.
Pod Identities are the AWS recommended approach going forward for providing credentials to your pods for AWS services.
Most recent bump was to v1.24.1 which is newer than 2023-11-14.
Thank you, but I'm not seeing that released yet, right? The changelog is still "unreleased" and I didn't see the commit in the latest 1.13.2 tag.
I agree on this, the AWS SDK is set to v1.24.1 on the main branch but the change has not been released in the v1.13.2 version. Any idea when this can be released ?
If it's not required in 1.13.z, then you can expect it in the next 1.14 https://github.com/vmware-tanzu/velero/wiki/1.14-Roadmap
1.13.z may never get it if there's not a need due to CVEs etc.
IIUC this issue is not a regression of a previously working feature right?
No not a regression at all. Just wanted an update on when this can get released on order to fully migrate to AWS pod identity. This can wait the 1.14 for sure.
@maxrabin According to the document https://docs.aws.amazon.com/eks/latest/userguide/pod-id-configure-pods.html, I think it should work, although it was not tested.
