flux2
flux2 copied to clipboard
fluxcd
Bootstrap fails with "context deadline exceeded"
Describe the bug
After bootstrapping from a Gitea repository with flux bootstrap git, the bootstrap fails with:
◎ waiting for Kustomization "flux-system/flux-system" to be reconciled
✗ Get "https://k3s.local/apis/kustomize.toolkit.fluxcd.io/v1/namespaces/flux-system/kustomizations/flux-system": context deadline exceeded
► confirming components are healthy
✔ helm-controller: deployment ready
✔ kustomize-controller: deployment ready
✔ notification-controller: deployment ready
✔ source-controller: deployment ready
✔ all components are healthy
✗ bootstrap failed with 1 health check failure(s)
This was working fine until I recently added a SealedSecret for my Cert Manager issuer. The initial commit for adding the Issuer and Sealed Secret worked fine within flux. I re-created the k3s cluster with my Terraform deployment because I wanted to disable the k3s helm controller. After my cluster came up, I tried to bootstrap flux and discovered this problem.
From what I can discern, Kustomize is trying to build the flux-system kustomization and for some reason it is looking for the SealedSecrets CRD which isn't installed yet (see additional context section). Shouldn't the dependsOn clause cause Sealed Secrets to be installed first and this error would go away?
For what it's worth, if I remove the cert-manager portion of my repository it deploys without any errors.
Steps to reproduce
- Bootstrap a fresh cluster from a git repository that has already defined deployments for Cert Manager, with a SealedSecret, and an Issuer which references the Sealed Secret
Expected behavior
The bootstrap should succeed
Screenshots and recordings
No response
OS / Distro
Fedora CoreOS
Flux version
2.0.1
Flux check
► checking prerequisites ✔ Kubernetes 1.27.4+k3s1 >=1.24.0-0 ► checking controllers ✔ helm-controller: deployment ready ► ghcr.io/fluxcd/helm-controller:v0.35.0 ✔ kustomize-controller: deployment ready ► ghcr.io/fluxcd/kustomize-controller:v1.0.1 ✔ notification-controller: deployment ready ► ghcr.io/fluxcd/notification-controller:v1.0.0 ✔ source-controller: deployment ready ► ghcr.io/fluxcd/source-controller:v1.0.1 ► checking crds ✔ alerts.notification.toolkit.fluxcd.io/v1beta2 ✔ buckets.source.toolkit.fluxcd.io/v1beta2 ✔ gitrepositories.source.toolkit.fluxcd.io/v1 ✔ helmcharts.source.toolkit.fluxcd.io/v1beta2 ✔ helmreleases.helm.toolkit.fluxcd.io/v2beta1 ✔ helmrepositories.source.toolkit.fluxcd.io/v1beta2 ✔ kustomizations.kustomize.toolkit.fluxcd.io/v1 ✔ ocirepositories.source.toolkit.fluxcd.io/v1beta2 ✔ providers.notification.toolkit.fluxcd.io/v1beta2 ✔ receivers.notification.toolkit.fluxcd.io/v1 ✔ all checks passed
Git provider
Gitea (plain git)
Container Registry provider
No response
Additional context
flux get ks -A reveals:
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system flux-system False False SealedSecret/cert-manager/k3s.ca dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: bitnami.com/v1alpha1: the server could not find the requested resource
flux logs -A --level error reveals:
2023-08-20T16:08:16.946Z error Kustomization/flux-system.flux-system - Reconciliation failed after 948.112357ms, next try in 10m0s SealedSecret/cert-manager/k3s.ca dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: bitnami.com/v1alpha1: the server could not find the requested resource
CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io configured
CustomResourceDefinition/buckets.source.toolkit.fluxcd.io configured
CustomResourceDefinition/gitrepositories.source.toolkit.fluxcd.io configured
CustomResourceDefinition/helmcharts.source.toolkit.fluxcd.io configured
CustomResourceDefinition/helmreleases.helm.toolkit.fluxcd.io configured
CustomResourceDefinition/helmrepositories.source.toolkit.fluxcd.io configured
CustomResourceDefinition/kustomizations.kustomize.toolkit.fluxcd.io configured
CustomResourceDefinition/ocirepositories.source.toolkit.fluxcd.io configured
CustomResourceDefinition/providers.notification.toolkit.fluxcd.io configured
CustomResourceDefinition/receivers.notification.toolkit.fluxcd.io configured
Namespace/cert-manager created
Namespace/flux-system configured
and...
2023-08-20T16:18:13.626Z error Kustomization/flux-system.flux-system - Reconciliation failed after 468.302081ms, next try in 10m0s SealedSecret/cert-manager/k3s.ca dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: bitnami.com/v1alpha1: the server could not find the requested resource
Here is the Sealed Secret:
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
creationTimestamp: null
name: k3s.ca
namespace: cert-manager
spec:
encryptedData: [redacted]
tls.key: [redacted]
template:
metadata:
creationTimestamp: null
name: k3s.ca
namespace: cert-manager
type: kubernetes.io/tls
Here is the kustomization that contains all of the cert-manager resources (Issuer, Sealed Secret, Namespace, HelmRelease, and HelmRepository):
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cert-manager
namespace: flux-system
spec:
interval: 1h
sourceRef:
kind: GitRepository
name: flux-system
dependsOn:
- name: kube-prometheus-stack
- name: sealed-secrets
path: ./cluster/infrastructure/cert-manager
prune: true
wait: true
Here is the Sealed Secrets kustomization:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: sealed-secrets
namespace: flux-system
spec:
sourceRef:
kind: GitRepository
name: flux-system
path: ./cluster/infrastructure/sealed-secrets
interval: 1h
prune: true
wait: true
kubectl get pods -A:
NAMESPACE NAME READY STATUS RESTARTS AGE
flux-system helm-controller-74b5f87d94-ppwtp 1/1 Running 0 70m
flux-system kustomize-controller-59d4cb8bc6-zjkpp 1/1 Running 0 70m
flux-system notification-controller-b7d8566b7-8mkk9 1/1 Running 0 70m
flux-system source-controller-645656595b-nldph 1/1 Running 0 70m
kube-system coredns-77ccd57875-cjw6w 1/1 Running 0 71m
kube-system local-path-provisioner-957fdf8bc-p65rs 1/1 Running 0 71m
kube-system metrics-server-648b5df564-v2h2t 1/1 Running 0 71m
kubectl get events -n flux-system --field-selector type=Warning:
LAST SEEN TYPE REASON OBJECT MESSAGE
39s Warning ReconciliationFailed kustomization/flux-system SealedSecret/cert-manager/k3s.ca dry-run failed: failed to get API group resources: unable to retrieve the complete list of server APIs: bitnami.com/v1alpha1: the server could not find the requested resource
kubectl get gitrepositories.source.toolkit.fluxcd.io -A:
NAMESPACE NAME URL AGE READY STATUS
flux-system flux-system ssh://[email protected]/k3s/k3s 72m True stored artifact for revision 'master@sha1:3d4fdf8d3ce535f59d6b5fce7fd51514b80e4d52'
flux get sources all -A:
NAMESPACE NAME REVISION SUSPENDED READY MESSAGE
flux-system gitrepository/flux-system master@sha1:3d4fdf8d False True stored artifact for revision 'master@sha1:3d4fdf8d'
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Ok I found the issue - I forgot SealedSecrets are tied to a specific deployment and after I recreated the cluster the new cluster didn't have the old SealedSecrets keys.
Re-opening this since I don't think it's related to the sealed secret resource not being present. To mitigate the latter I save the sealed-secrets keys and re-apply them prior to bootstrapping flux (per https://github.com/bitnami-labs/sealed-secrets#how-can-i-do-a-backup-of-my-sealedsecrets). I also changed my dependencies a bit to be sure they were correct and then re-created the cluster by separating the config from the controller as suggested by https://github.com/fluxcd/flux2/issues/1980#issuecomment-949307972 . This seems related to #1980 .
Here is the cert-manager-config kustomization:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cert-manager-config
namespace: flux-system
spec:
interval: 1h
retryInterval: 1m
sourceRef:
kind: GitRepository
name: flux-system
dependsOn:
- name: cert-manager-controller
namespace: flux-system
- name: sealed-secrets
namespace: flux-system
path: ./cluster/infrastructure/cert-manager/config
prune: true
wait: true
Here is the cert-manager-controller kustomization:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: cert-manager-controller
namespace: flux-system
spec:
sourceRef:
kind: GitRepository
name: flux-system
path: ./cluster/infrastructure/cert-manager/controller
dependsOn:
- name: kube-prometheus-stack
interval: 1h
retryInterval: 1m
prune: true
wait: true
If I delete the cert-manager-config kustomization, commit, push my changes, the flux-system kustomization eventualy becomes ready. If I add back the cert-manager-config kustomization after that, it becomes ready as well.
Therefore, it seems like the flux-system kustomization isn't respecting the dependencies described. The only work-around I can think of is to delete all kustomizations that have sealed secrets from the repository, push changes, and re-apply them. This seems pretty painful whenever anyone will have to rebuild their cluster.
The issue likely is that you're making Flux try to create a SealedSecret while the CRD isn't there, yet. How are you deploying the SealedSecret CRDs? As part of a HelmRelease? If that's so, you need to create a Kustomization for that HelmRelease and make your other Kustomization (the one that creates SealedSecret resources) depend on that.
@makkes Thanks for your reply. I do already have a Flux Kustomization for Sealed Secrets:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: sealed-secrets
namespace: flux-system
spec:
sourceRef:
kind: GitRepository
name: flux-system
path: ./cluster/infrastructure/sealed-secrets
interval: 1h
retryInterval: 1m
prune: true
wait: true
./cluster/infrastructure/sealed-secrets/repository.yaml:
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
name: sealed-secrets
namespace: flux-system
spec:
url: https://bitnami-labs.github.io/sealed-secrets
interval: 24h
./cluster/infrastructure/sealed-secrets/release.yaml:
apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
name: sealed-secrets
namespace: flux-system
spec:
chart:
spec:
chart: sealed-secrets
reconcileStrategy: ChartVersion
sourceRef:
kind: HelmRepository
name: sealed-secrets
targetNamespace: kube-system
interval: 24h
values:
fullnameOverride: sealed-secrets-controller
The cert-manager-config kustomization depends on the sealed-secrets kustomization already.
I think I might have found the problem though. In ./cluster, where I bootstrap flux, I have one file at the moment: infrastructure.yaml:
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: infrastrucure
namespace: flux-system
spec:
sourceRef:
kind: GitRepository
name: flux-system
path: ./cluster/infrastructure
interval: 1h
retryInterval: 1m
prune: true
wait: true
I think the problem with this is since there is no kustomization.yaml file defined here, the kustomize controller will automatically generate one and include all of my yaml files inside infrastructure as one Flux kustomization.
I added a kustomization.yaml file at ./cluster/kustomization.yaml:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- infrastructure.yaml
This makes the kustomize build only consider infrastructure.yaml as part of the flux-system kustomization, which makes the build succeed and the rest of the dependencies are built correctly.
I think this gotchya should be more clearly documented in the getting started guides perhaps. The only thing that led me to think about this problem was this small blurb buried in the FAQ: https://fluxcd.io/flux/faq/#can-i-use-repositories-with-plain-yamls .
I started off by using the example repository on how to structure my own repository but I wanted to vary it slightly because I have other source at the root, and want to define all of my cluster resources (including flux) at ./cluster .
The problem is resolved by adding that kustomization.yaml file as I described above. Should I close this or keep it open as a documentation issue?
Please see the recommend repo structure here: https://github.com/fluxcd/flux2-kustomize-helm-example#infrastructure
As explained in the docs, you should have at least two Flux Kustomizations, one for controllers (CRDs) and one for configs (CRs) with a dependsOn relationship.
@stefanprodan I do indeed have that (one named cert-manager-controller and another named cert-manager-configs). The problem still manifested regardless, because of what I mentioned about the kustomization.yaml file at ./cluster . In the example repository, the infrastructure directory is separated from the cluster directory. I think if the infrastructure directory were to move underneath the cluster directory the same problem would manifest.
In the clusters/my-cluster dir you should only have Flux Kustomizations without a kustomization.yaml. In the Flux Kustomizations you should refer overlays from the top root level. This is the the structure we recommend to users but you are free to do whatever.
@stefanprodan I realize that's what works for this kind of structure. However I thought one of the benefits to flux was that the developers are free to create a more loose structure. Instead of placing the cluster dir as a separate directory, my cluster dir is at the Flux repository root (not the actual root, since I'm using --path ./cluster with flux bootstrap). If one wants to have this kind of structure, then a kustomization.yaml file in the cluster dir seems like a requirement. It was pretty tedious to track this problem down since the way flux looks at directories without a kustomization.yaml file is buried in the FAQ.
I'm happy since I got it to work, however I wonder if others might benefit from something in the getting started guide about how flux looks at the initial directory it's bootstrapped with. If there are any yaml files in other directories in the tree of the directory that was bootstrapped (other than flux-system), they will get included into the flux-system kustomization which is probably not what the end-user wants.