image-automation-controller icon indicating copy to clipboard operation
image-automation-controller copied to clipboard

Published 20 hours ago • verified publisher icon fluxcd

Intermittent unexpected EOF

Open RonnanSouza opened this issue 2 years ago • 2 comments

The Issue

Several ImageUpdateAutomations are intermittently failing our clusters. We use git sources pointing to our Gitlab using deploy keys to authenticate through SSH.

Artifacts

Flux Check
► checking prerequisites
✗ flux 0.37.0 ✔ Kubernetes 1.23.12 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.27.0
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.27.0
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.23.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.31.0
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.29.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.32.1
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git Source

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  name: provisioning-api
  namespace: provisioning-api
  gitImplementation: go-git
  ignore: |
    /*
    !/kubernetes/kube-stag-us-cilium.tfgco.com
  interval: 15m
  ref:
    branch: master
  secretRef:
    name: provisioning-api-git-credentials
  timeout: 60s
  url: ssh://[email protected]/infrastructure/provisioning-api

ImageUpdateAutomation

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: provisioning-api
  namespace: provisioning-api
spec:
  git:
    checkout:
      ref:
        branch: master
    commit:
      author:
        email: [email protected]
        name: flux2-bot
      messageTemplate: Automated image update by Flux
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: provisioning-api
    namespace: provisioning-api
  update:
    path: /kubernetes/kube-stag-us-cilium.tfgco.com
    strategy: Setters

Logs

image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.486Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"}}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.486Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"},"ref":{"branch":"master"},"working":"/tmp/provisioning-api-provisioning-api1860084948"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.862Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","count":1,"manifests-path":"/tmp/provisioning-api-provisioning-api1860084948/kubernetes/kube-stag-us-cilium.tfgco.com"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:43.864Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","working":"/tmp/provisioning-api-provisioning-api1860084948"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"error","ts":"2023-01-19T23:22:45.020Z","msg":"unexpected EOF","name":"provisioning-api","namespace":"provisioning-api","reconciler kind":"ImageUpdateAutomation","annotations":null,"error":"error","stacktrace":"github.com/fluxcd/pkg/runtime/events.(*Recorder).AnnotatedEventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:137\ngithub.com/fluxcd/pkg/runtime/events.(*Recorder).Eventf\n\tgithub.com/fluxcd/pkg/[email protected]/events/recorder.go:114\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).event\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:593\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile.func2\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:163\ngithub.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile\n\tgithub.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:404\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.020Z","logger":"events","msg":"unexpected EOF","type":"Warning","object":{"kind":"ImageUpdateAutomation","namespace":"provisioning-api","name":"provisioning-api","uid":"c4762892-cde9-4ac4-a32d-af8bf726449f","apiVersion":"image.toolkit.fluxcd.io/v1beta1","resourceVersion":"1051457684"},"reason":"error"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"error","ts":"2023-01-19T23:22:45.044Z","msg":"Reconciler error","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"005ea767-5e32-48ec-9abc-f28c2d33aa21","error":"unexpected EOF","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:326\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.809Z","msg":"fetching git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"}}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:45.810Z","msg":"attempting to clone git repository","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","gitrepository":{"namespace":"provisioning-api","name":"provisioning-api"},"ref":{"branch":"master"},"working":"/tmp/provisioning-api-provisioning-api2243875591"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:46.168Z","msg":"updating with setters according to image policies","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","count":1,"manifests-path":"/tmp/provisioning-api-provisioning-api2243875591/kubernetes/kube-stag-us-cilium.tfgco.com"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:46.169Z","msg":"ran updates to working dir","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","working":"/tmp/provisioning-api-provisioning-api2243875591"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"info","ts":"2023-01-19T23:22:47.573Z","msg":"Committed and pushed change 8e20661c6840a226463667890b5bfec937fda422 to master\nAutomated image update by Flux","name":"provisioning-api","namespace":"provisioning-api","reconciler kind":"ImageUpdateAutomation","reason":"info","annotations":null}
image-automation-controller-bfb56c4-zrkpf manager {"level":"debug","ts":"2023-01-19T23:22:47.573Z","logger":"events","msg":"Committed and pushed change 8e20661c6840a226463667890b5bfec937fda422 to master\nAutomated image update by Flux","type":"Normal","object":{"kind":"ImageUpdateAutomation","namespace":"provisioning-api","name":"provisioning-api","uid":"c4762892-cde9-4ac4-a32d-af8bf726449f","apiVersion":"image.toolkit.fluxcd.io/v1beta1","resourceVersion":"1051457715"},"reason":"info"}
image-automation-controller-bfb56c4-zrkpf manager {"level":"info","ts":"2023-01-19T23:22:47.575Z","msg":"pushed commit to origin","controller":"imageupdateautomation","controllerGroup":"image.toolkit.fluxcd.io","controllerKind":"ImageUpdateAutomation","ImageUpdateAutomation":{"name":"provisioning-api","namespace":"provisioning-api"},"namespace":"provisioning-api","name":"provisioning-api","reconcileID":"0e3c73b4-da7e-4f04-8c0a-65aaaa617111","revision":"8e20661c6840a226463667890b5bfec937fda422","branch":"master"}

RonnanSouza avatar Jan 19 '23 23:01 RonnanSouza