k8s.io
k8s.io copied to clipboard
kubernetes
image-promo job is hitting quota limits
The image-promo prow job is failing a lot more frequently than usual (6x in 5h): https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/post-k8sio-image-promo
It looks like it's hitting quota limits on various regional GCP registries. Maybe the total amount of images is too high for the current limits.
two examples:
11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig: fetching \"asia-south1-docker.pkg.dev/k8s-artifacts-prod/images/kube-proxy-ppc64le:sha256-11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig\": GET https://asia-south1-docker.pkg.dev/v2/k8s-artifacts-prod/images/kube-proxy-ppc64le/manifests/sha256-11347b163438dc07aabfb3ddb651c785b4830f379c1ca6b57bd4939e7e29c895.sig: TOOMANYREQUESTS: Quota exceeded for quota metric 'Requests per project per user' and limit 'Requests per project per user per minute per user' of service 'artifactregistry.googleapis.com' for consumer 'project_number:388270116193'." diff=12ms
time="15:40:40.771" level=fatal msg="run `cip run`: promote images: signing images: replicating signatures: copying signature europe-west9-docker.pkg.dev/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig to australia-southeast1-docker.pkg.dev/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller:sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig: HEAD https://australia-southeast1-docker.pkg.dev/v2/k8s-artifacts-prod/images/capi-ipam-ic/cluster-api-ipam-in-cluster-controller/manifests/sha256-2fa62384935b0233f68acf75fcb12bbe149b7f122e83d4e5f67f157e73998732.sig: unexpected status code 429 Too Many Requests (HEAD responses have no body, use GET for details)" diff=958ms
The image promoter makes a really high amount of API calls because of the approach to image signatures.
We have not changed the quotas in the infrastructure projects.
/transfer kubernetes-sigs/promo-tools /sig release
@BenTheElder: Something went wrong or the destination repo kubernetes/kubernetes-sigs/promo-tools does not exist.
In response to this:
The image promoter makes a really high amount of API calls because of the approach to image signatures.
We have not changed the quotas in the infrastructure projects.
/transfer kubernetes-sigs/promo-tools /sig release
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
The image promoter is at https://github.com/kubernetes-sigs/promo-tools
Happened today too for a CAPV release:
https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/post-k8sio-image-promo/1776261613632884736
This is not getting visibility to the right people, please file a bug with kubernetes-sigs/promo-tools or kubernetes/sig-release.
K8s-Infra is not changing the limits on the backing registries, they are roughly equivalent to the previous non-configurable GCR limits and they're necessary to prevent trivial DOS, even the project itself must not be permitted to self-DOS, and there is no way to exempt a single user.
Opened the issue at promo-tools. Maybe I can find some time to take a look where it could get optimised. Thanks for pointing to the right direction @BenTheElder ! Really appreciating it 🚀
/close
Please check:
- https://github.com/kubernetes-sigs/promo-tools/issues/1271
@ameukam: Closing this issue.
In response to this:
/close
Please check:
- https://github.com/kubernetes-sigs/promo-tools/issues/1271
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.