image-reflector-controller icon indicating copy to clipboard operation
image-reflector-controller copied to clipboard

Published 20 hours ago • verified publisher icon fluxcd

Backoff on ImageRepo reconciliations on 429 Too many Requests error

Open goutamtadi1 opened this issue 3 years ago • 0 comments

Describe the bug

In spite of receiving 429 Too many requests errors from Image Repositories, flux is still trying to reconcile. This will potentially lead to account blocking.

Steps to reproduce

  1. When having a image policy like below:
apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImagePolicy
metadata:
  name: some-name
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: some-repo
  filterTags:
    pattern: "^main-[a-fA-F0-9]+-(?P<ts>.*)"
    extract: "$ts"
  policy:
    alphabetical:
      order: asc
  1. And if there are more than one page of tags to pull from in target quay image repository ( more than 25)
  2. With ImageRepo declared as follows:
apiVersion: image.toolkit.fluxcd.io/v1alpha1
kind: ImageRepository
metadata:
  name: some-repo
  namespace: flux-system
spec:
  secretRef:
    name: flux-quay-pull-secret
  image: quay.io/someorg/someimage
  interval: 1h
  1. Flux image reconciliation happens over multiple pages even though there is 429 Too many requests error initially

Expected behavior

Flux reconciliation on image repositories should backoff

Screenshots and recordings

{"level":"error","ts":"2021-08-10T17:34:15.769Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:15.867Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:15.992Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:16.130Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:16.267Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:16.455Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:16.741Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:17.163Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:17.909Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:19.298Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:21.982Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:27.211Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}
{"level":"error","ts":"2021-08-10T17:34:37.572Z","logger":"controller-runtime.manager.controller.imagerepository","msg":"Reconciler error","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageRepository","name":"someimage","namespace":"flux-system","error":"GET https://quay.io/v2/astronomer/someimage/tags/list?n=50&next_page=REDACTED: unexpected status code 429 Too Many Requests: <html>\r\n<head><title>429 Too Many Requests</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>429 Too Many Requests</h1></center>\r\n<hr><center>nginx/1.12.1</center>\r\n</body>\r\n</html>\r\n"}

OS / Distro

Linux

Flux version

0.16.1

Flux check

► checking prerequisites ✗ flux 0.16.1 <0.16.2 (new version is available, please upgrade) ✔ kubectl 1.21.2 >=1.18.0-0 ✔ Kubernetes 1.20.7 >=1.16.0-0 ► checking controllers ✔ image-automation-controller: deployment ready ► ghcr.io/fluxcd/image-automation-controller:v0.7.0 ✔ image-reflector-controller: deployment ready ► ghcr.io/fluxcd/image-reflector-controller:v0.7.1 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v0.10.0 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v0.11.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v0.10.0 ✔ all checks passed

Git provider

No response

Container Registry provider

quay.io

Additional context

No response

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

goutamtadi1 avatar Aug 10 '21 18:08 goutamtadi1