source-controller
source-controller copied to clipboard
fluxcd
invalid memory address or nil pointer dereference in GCP bucket provider (v.0.29.0)
Hiho,
we try to fetch some helm charts from a GCP bucket using Workload Identity, but the source-controller v.0.29.0 runs into an invalid memory address or nil pointer dereference error.
Bucket definition:
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: Bucket
metadata:
name: my-bucket
namespace: flux-system
spec:
interval: 1m
bucketName: my-bucket
endpoint: storage.googleapis.com
provider: gcp
Error:
│ E0922 07:16:25.547477 1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) │
│ goroutine 373 [running]: │
│ k8s.io/apimachinery/pkg/util/runtime.logPanic({0x4f08a0?, 0x3fd8ea0}) │
│ k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x99 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1() │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:110 +0xb9 │
│ panic({0x4f08a0, 0x3fd8ea0}) │
│ runtime/panic.go:838 +0x207 │
│ cloud.google.com/go/storage.getAgeCondition({0x32ca00?, 0x0?}) │
│ cloud.google.com/go/[email protected]/bucket.go:1539 +0xc5 │
│ cloud.google.com/go/storage.toLifecycle(0xc002d758c0?) │
│ cloud.google.com/go/[email protected]/bucket.go:1565 +0x1d1 │
│ cloud.google.com/go/storage.newBucket(0xc00113fa40) │
│ cloud.google.com/go/[email protected]/bucket.go:726 +0x77d │
│ cloud.google.com/go/storage.(*httpStorageClient).GetBucket(0xc002d7d600, {0xe23b28, 0xc002e3ad50}, {0xc002b76570, 0x13}, 0x50?, {0xc0053256e0?, 0xe00000000032c940?, 0xc001727568?}) │
│ cloud.google.com/go/[email protected]/http_client.go:279 +0x518 │
│ cloud.google.com/go/storage.(*BucketHandle).Attrs(0xc0017275f0, {0xe23af0?, 0xc002e13380?}) │
│ cloud.google.com/go/[email protected]/bucket.go:148 +0x344 │
│ github.com/fluxcd/source-controller/pkg/gcp.(*GCSClient).BucketExists(0xe23b28?, {0xe23af0?, 0xc002e13380?}, {0xc002b76570?, 0xc000945118?}) │
│ github.com/fluxcd/source-controller/pkg/gcp/gcp.go:86 +0x1bc │
│ github.com/fluxcd/source-controller/controllers.fetchEtagIndex({0xe23b28?, 0xc002e3a000?}, {0xe26200, 0xc00012cdf0}, 0xc00071cb40, 0xc002d15500, {0xc002d65d80, 0x36}) │
│ github.com/fluxcd/source-controller/controllers/bucket_controller.go:750 +0xf1 │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).reconcileSource(0xc0004f24b0?, {0xe23b28, 0xc002e3a000}, 0xc00071cb40, 0xc002d15500, {0xc002d65d80, 0x36}) │
│ github.com/fluxcd/source-controller/controllers/bucket_controller.go:503 +0x6bb │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).reconcile(0xe36030?, {0xe23b28, 0xc002e3a000}, 0xc00071cb40, {0xc001727c48, 0x3, 0x7f2386c97e00?}) │
│ github.com/fluxcd/source-controller/controllers/bucket_controller.go:358 +0x519 │
│ github.com/fluxcd/source-controller/controllers.(*BucketReconciler).Reconcile(0xc0004f24b0, {0xe23b28, 0xc002e3a000}, {{{0xc000a4ccb0?, 0x10?}, {0xc000a5e4c8?, 0x2160327?}}}) │
│ github.com/fluxcd/source-controller/controllers/bucket_controller.go:318 +0x55c │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xe23a80?, {0xe23b28?, 0xc002e3a000?}, {{{0xc000a4ccb0?, 0x7775a0?}, {0xc000a5e4c8?, 0x2155854?}}}) │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xc8 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000b06e0, {0xe23a80, 0xc000902140}, {0x5bd3a0?, 0xc002dc2880?}) │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:320 +0x33c │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000b06e0, {0xe23a80, 0xc000902140}) │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x1d9 │
│ sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2() │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85 │
│ created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 │
│ sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:230 +0x325 │
│ {"level":"error","ts":"2022-09-22T07:16:25.547Z","msg":"Reconciler error","controller":"bucket","controllerGroup":"source.toolkit.fluxcd.io","controllerKind":"Bucket","bucket":{"name":"my-bucket","namespace":"flux-system"}, │
│ "namespace":"flux-system","name":"my-bucket","reconcileID":"4c2ffc10-417f-4794-88f9-5f5cc5fdb6c5","error":"panic: runtime error: invalid memory address or nil pointer dereference [recovered]"}
The method getAgeCondition has been removed in the latest Storage API client v0.26.0:
(https://github.com/googleapis/google-cloud-go/compare/storage/v1.25.0...storage/v1.26.0#diff-0c42e3f240ee8d43e271bb6004c8857419ae170063a1d69227cb9811325bd2de)
same panic for source-controller 0.28.0 but 0.27.0 works fine without panics
The crash seems to be from the gcp library
@kunzese I am trying to recreate this. I have tried w a non-existent bucket, and an identity that has no access (and one that does)
@somtochiama our bucket is existing and we confirmed that everything around the workload identity thingy from GKE is working correctly with
kubectl run debug -n flux-system -i --tty --restart=Never --rm --overrides='{ "spec": { "serviceAccount": "source-controller" } }' --image gcr.io/cloud-builders/gsutil ls gs://my-bucket
The content of the bucket is listed without any issues.