[efs-controller] Removing backupPolicy from filesystem puts controller into crashloop

[gnadaban]

Describe the bug Removing the spec.backupPolicy field from a FileSystem resource after setting it to anything puts the controller into a constant crashloop.

{"level":"info","ts":"2024-10-03T14:29:57.019Z","msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"filesystem","controllerGroup":"efs.services.k8s.aws","controllerKind":"FileSystem","FileSystem":{"name":"dummy","namespace":"dummy"},"namespace":"dummy","name":"dummy","reconcileID":"656fc509-e305-40b1-87af-825ecc35a2fb"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1a31837]

goroutine 273 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:111 +0x1e5
panic({0x1c48080?, 0x34fe5a0?})
        /usr/local/go/src/runtime/panic.go:770 +0x132
github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system.(*resourceManager).syncBackupPolicy(0xc000b76e00, {0x2590848, 0xc000f815f0}, 0xc00007c3a8)
        /github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system/hooks.go:245 +0x137
github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system.(*resourceManager).sdkUpdate(0xc000b76e00, {0x2590848, 0xc000f815f0}, 0xc00007c3a8, 0xc000734648, 0xc000e8db90)
        /github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system/sdk.go:491 +0x39a
github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system.(*resourceManager).Update(0xc000b76e00, {0x2590848?, 0xc000f815f0?}, {0x259a078?, 0xc00007c3a8?}, {0x259a078?, 0xc000734648}, 0x1?)
        /github.com/aws-controllers-k8s/efs-controller/pkg/resource/file_system/manager.go:157 +0x77
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).updateResource(0xc000408780, {0x2590848, 0xc000f815f0}, {0x259a010, 0xc000b76e00}, {0x259a078, 0xc00007c3a8}, {0x259a078, 0xc000734648})
        /go/pkg/mod/github.com/aws-controllers-k8s/[email protected]/pkg/runtime/reconciler.go:647 +0x3e3
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Sync(0xc000408780, {0x2590848, 0xc000f815f0}, {0x259a010, 0xc000b76e00}, {0x259a078, 0xc00007c348})
        /go/pkg/mod/github.com/aws-controllers-k8s/[email protected]/pkg/runtime/reconciler.go:390 +0x7de
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).reconcile(0xc000408780, {0x2590848, 0xc000f815f0}, {0x259a010, 0xc000b76e00}, {0x259a078, 0xc00007c348})
        /go/pkg/mod/github.com/aws-controllers-k8s/[email protected]/pkg/runtime/reconciler.go:326 +0x1cd
github.com/aws-controllers-k8s/runtime/pkg/runtime.(*resourceReconciler).Reconcile(0xc000408780, {0x2590848, 0xc000f810b0}, {{{0xc000bf6446?, 0x5?}, {0xc000b163f0?, 0xc001095d10?}}})
        /go/pkg/mod/github.com/aws-controllers-k8s/[email protected]/pkg/runtime/reconciler.go:283 +0x95b
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x2595190?, {0x2590848?, 0xc000f810b0?}, {{{0xc000bf6446?, 0xb?}, {0xc000b163f0?, 0x0?}}})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0xb7
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006562c0, {0x2590880, 0xc0000ad2c0}, {0x1d102a0, 0xc000a0c1e0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x3bc
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006562c0, {0x2590880, 0xc0000ad2c0})
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:261 +0x1be
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:222 +0x79
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2 in goroutine 121
        /go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:218 +0x486

Steps to reproduce

1. Create a FileSystem resource with backupPolicy set to disabled

   cat <<EOF | kubectl apply -f -
   apiVersion: efs.services.k8s.aws/v1alpha1
   kind: FileSystem
   metadata:
     name: dummy
   spec:
     backupPolicy:
       status: DISABLED
     policy: |
       {
         "Statement": [
           {
             "Action": [
               "elasticfilesystem:ClientMount",
               "elasticfilesystem:ClientWrite",
               "elasticfilesystem:ClientRootAccess"
             ],
             "Effect": "Allow",
             "Principal": {
               "AWS": "*"
             },
             "Resource": [
               "*"
             ]
           }
         ],
         "Version": "2012-10-17"
       }
     tags:
     - key: Name
       value: dummy
   EOF
   

2. Wait for the FileSystem to be created by the controller
3. Edit the FileSystem resource and delete the backupPolicy field

   cat <<EOF | kubectl apply -f -
   apiVersion: efs.services.k8s.aws/v1alpha1
   kind: FileSystem
   metadata:
     name: dummy
   spec:
     policy: |
       {
         "Statement": [
           {
             "Action": [
               "elasticfilesystem:ClientMount",
               "elasticfilesystem:ClientWrite",
               "elasticfilesystem:ClientRootAccess"
             ],
             "Effect": "Allow",
             "Principal": {
               "AWS": "*"
             },
             "Resource": [
               "*"
             ]
           }
         ],
         "Version": "2012-10-17"
       }
     tags:
     - key: Name
       value: dummy
   EOF
   

4. EFS controller keeps crashing

Expected outcome Controller reconciles without crashing.

Environment

• Kubernetes version: v1.30.4-eks-a737599
• Using EKS (yes/no), if so version? yes
• AWS service targeted (S3, RDS, etc.) EFS