flux2 icon indicating copy to clipboard operation
flux2 copied to clipboard

upgrading to v0.31.3 results in a 404

Open masterkain opened this issue 2 years ago • 5 comments

Describe the bug

today on one of my clusters (EKS) I've seen this during an upgrade from v0.31.1:

✔ cloned repository
► generating component manifests
✔ generated component manifests
✔ component manifests are up to date
✔ reconciled components
► determining if source secret "flux-system/flux-system" exists
✔ source secret up to date
► generating sync manifests
✔ generated sync manifests
✔ sync manifests are up to date
► applying sync manifests
✔ reconciled sync configuration
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✗ failed to download artifact from http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/e36b5ede775cff77408cdc3b51091ddc3d666dd0.tar.gz, status: 404 Not Found
► confirming components are healthy
✔ helm-controller: deployment ready
✔ image-automation-controller: deployment ready
✔ image-reflector-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
✗ bootstrap failed with 1 health check failure(s)

not sure if it does any harm but maybe it's worth reporting, it keeps doing that on further attempts

Steps to reproduce

after upgrading flux locally:

flux bootstrap github --context=mycontext --owner=myowner --repository=cluster --path=clusters/mycluster --branch=main --read-write-key --components-extra=image-reflector-controller,image-automation-controller

Expected behavior

no failed health checks

Screenshots and recordings

No response

OS / Distro

macOS 12.4

Flux version

v0.31.3

Flux check

► checking prerequisites
✔ Kubernetes 1.22.9-eks-a64ea69 >=1.20.6-0
► checking controllers
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v0.22.1
✔ image-automation-controller: deployment ready
► ghcr.io/fluxcd/image-automation-controller:v0.23.4
✔ image-reflector-controller: deployment ready
► ghcr.io/fluxcd/image-reflector-controller:v0.19.2
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v0.26.2
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v0.24.0
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v0.25.9
► checking crds
✔ alerts.notification.toolkit.fluxcd.io/v1beta1
✔ buckets.source.toolkit.fluxcd.io/v1beta2
✔ gitrepositories.source.toolkit.fluxcd.io/v1beta2
✔ helmcharts.source.toolkit.fluxcd.io/v1beta2
✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1
✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2
✔ imagepolicies.image.toolkit.fluxcd.io/v1beta1
✔ imagerepositories.image.toolkit.fluxcd.io/v1beta1
✔ imageupdateautomations.image.toolkit.fluxcd.io/v1beta1
✔ kustomizations.kustomize.toolkit.fluxcd.io/v1beta2
✔ providers.notification.toolkit.fluxcd.io/v1beta1
✔ receivers.notification.toolkit.fluxcd.io/v1beta1
✔ all checks passed

Git provider

GitHub

Container Registry provider

No response

Additional context

No response

Code of Conduct

  • [X] I agree to follow this project's Code of Conduct

masterkain avatar Jun 29 '22 22:06 masterkain

I have the same issue on an upgrade

kustomization/core.flux-system
failed to download artifact from http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/985578b4b6083839811d59477e92d40d83654db2.tar.gz, status: 404 Not Found
revision
master/985578b4b6083839811d59477e92d40d83654db2

timothystewart6 avatar Jun 30 '22 02:06 timothystewart6

Can you please post here flux get source git flux-system and flux get ks flux-system

stefanprodan avatar Jun 30 '22 08:06 stefanprodan

Hi, if this is still happening, can you also share the status of the GitRepository object with kubectl -n flux-system get gitrepo flux-system -o yaml. The status can provide more information about the state of the source artifact, if there's any issue in that.

darkowlzz avatar Jun 30 '22 09:06 darkowlzz

I tried again today and got no errors, anyway:

kain@kain-osx cluster % flux get source git flux-system
NAME       	REVISION    	SUSPENDED	READY	MESSAGE
flux-system	main/baff7f0	False    	True 	stored artifact for revision 'main/baff7f03dcfdd6635be4de0e9aedb272aee8209f'
kain@kain-osx cluster % flux get ks flux-system
NAME       	REVISION    	SUSPENDED	READY	MESSAGE
flux-system	main/baff7f0	False    	True 	Applied revision: main/baff7f0
kain@kain-osx cluster % kubectl -n flux-system get gitrepo flux-system -o yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: GitRepository
metadata:
  annotations:
    reconcile.fluxcd.io/requestedAt: 2022-06-30 21:21:12.017358735 +0000 UTC m=+1287229.872381964
  creationTimestamp: "2022-05-14T23:17:22Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 1
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: flux-system
  namespace: flux-system
  resourceVersion: "37038373"
  uid: 2e7d7c92-a9a0-42ef-8104-ea5d9c3bb9c7
spec:
  gitImplementation: go-git
  interval: 1m0s
  ref:
    branch: main
  secretRef:
    name: flux-system
  timeout: 60s
  url: ssh://[email protected]/xxx/xxx
status:
  artifact:
    checksum: 5035720dafc4325397bb733eb411673fc0a470ba4f6b88da468769ce0adda748
    lastUpdateTime: "2022-06-30T21:21:14Z"
    path: gitrepository/flux-system/flux-system/baff7f03dcfdd6635be4de0e9aedb272aee8209f.tar.gz
    revision: main/baff7f03dcfdd6635be4de0e9aedb272aee8209f
    size: 944967
    url: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/baff7f03dcfdd6635be4de0e9aedb272aee8209f.tar.gz
  conditions:
  - lastTransitionTime: "2022-06-30T21:21:14Z"
    message: stored artifact for revision 'main/baff7f03dcfdd6635be4de0e9aedb272aee8209f'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: Ready
  - lastTransitionTime: "2022-06-30T21:21:14Z"
    message: stored artifact for revision 'main/baff7f03dcfdd6635be4de0e9aedb272aee8209f'
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: ArtifactInStorage
  contentConfigChecksum: sha256:fcbcf165908dd18a9e49f7ff27810176db8e9f63b4352213741664245224f8aa
  lastHandledReconcileAt: 2022-06-30 21:21:12.017358735 +0000 UTC m=+1287229.872381964
  observedGeneration: 1
  url: http://source-controller.flux-system.svc.cluster.local./gitrepository/flux-system/flux-system/latest.tar.gz

masterkain avatar Jun 30 '22 21:06 masterkain

Here's my output

➜  home-ops git:(master) flux get source git flux-system
NAME            REVISION        SUSPENDED       READY   MESSAGE

flux-system     master/0fc8d21  False           True    stored artifact for revision 'master/0fc8d21831f141228440ab1949edf60ba3120001'
➜  home-ops git:(master) flux get ks flux-system
NAME            REVISION        SUSPENDED       READY   MESSAGE

flux-system     master/0fc8d21  False           True    Applied revision: master/0fc8d21

The error stopped around 6/30/2022 10:30 CST

timothystewart6 avatar Jun 30 '22 23:06 timothystewart6

This was fixed in Flux 0.32.0

stefanprodan avatar Aug 12 '22 09:08 stefanprodan