Segmentation fault in cron job checkin

[matt-genie]

Environment

Currently the latest docker image: ghcr.io/getsentry/sentry-kubernetes:ff2386f6b1176a36ebfcdb96222ae5840dac8cf1 AWS EKS, version: 1.24

Steps to Reproduce

1. Run the sentry-kubernetes image in a Kubernetes cluster with cron job monitoring enabled
2. The container goes into CrashLoopBackOff state

I am confident the pod is configured correctly. We are running the same helm chart in other clusters, with exactly the same configuration, and k8s events are uploaded to sentry.

If I turn off cron job monitoring, the pod starts working.

apiVersion: v1
kind: Pod
metadata:
  name: sentry-agent-b789f68c8-c9k9h
spec:
  containers:
  - env:
    - name: SENTRY_DSN
      valueFrom:
        secretKeyRef:
          key: kubernetes
          name: sentry
    - name: SENTRY_K8S_MONITOR_CRONJOBS
      value: "1"
    - name: SENTRY_K8S_WATCH_NAMESPACES
      value: __all__
    - name: SENTRY_ENVIRONMENT
      value: production
    image: ghcr.io/getsentry/sentry-kubernetes:ff2386f6b1176a36ebfcdb96222ae5840dac8cf1
    imagePullPolicy: IfNotPresent
    name: sentry-agent
    resources: {}
    securityContext: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: sentry-agent
  serviceAccountName: sentry-agent
  terminationGracePeriodSeconds: 30

Expected Result

The pod should not crash with segmentation fault under no circumstances, especially when the service account is allowed to list cron jobs and jobs.

Actual Result

[Sentry] 2024/03/26 16:45:32 Release detection failed: exec: "git": executable file not found in $PATH
[Sentry] 2024/03/26 16:45:32 Some Sentry features will not be available. See https://docs.sentry.io/product/releases/.
[Sentry] 2024/03/26 16:45:32 To stop seeing this message, pass a Release to sentry.Init or set the SENTRY_RELEASE environment variable.
4:45PM INF Auto-detecting cluster configuration...
4:45PM INF Detected in-cluster configuration
4:45PM INF Running integrations...
4:45PM INF Watching events starting from: Tue, 26 Mar 2024 16:45:32 +0000 namespace=__all__ watcher=events
4:45PM INF Add job informer handlers for cronjob monitoring namespace=__all__ watcher=pods
4:45PM INF Add cronjob informer handlers for cronjob monitoring namespace=__all__ watcher=pods
E0326 16:45:32.093041       1 runtime.go:79] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 35 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1643440?, 0x2582f20})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x99
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe?})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x75
panic({0x1643440, 0x2582f20})
	/usr/local/go/src/runtime/panic.go:884 +0x213
main.runSentryCronsCheckin({0x1a629a8, 0xc000385890}, 0xc0007875d0, {0xc00049dd80?, 0xc000397520?})
	/app/crons.go:38 +0xbb
main.createJobInformer.func1({0x181a6c0?, 0xc0007875d0})
	/app/informer_jobs.go:26 +0xcd
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:232
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:816 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000484f38?, {0x1a4d600, 0xc0003fa3f0}, 0x1, 0xc0003947e0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc000484f88?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0003d0080)
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:810 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x85
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x148fa1b]

goroutine 35 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xfffffffe?})
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:56 +0xd7
panic({0x1643440, 0x2582f20})
	/usr/local/go/src/runtime/panic.go:884 +0x213
main.runSentryCronsCheckin({0x1a629a8, 0xc000385890}, 0xc0007875d0, {0xc00049dd80?, 0xc000397520?})
	/app/crons.go:38 +0xbb
main.createJobInformer.func1({0x181a6c0?, 0xc0007875d0})
	/app/informer_jobs.go:26 +0xcd
k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd(...)
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/controller.go:232
k8s.io/client-go/tools/cache.(*processorListener).run.func1()
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:816 +0x134
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:157 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc000484f38?, {0x1a4d600, 0xc0003fa3f0}, 0x1, 0xc0003947e0)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:158 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0xc000484f88?)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:135 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(...)
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:92
k8s.io/client-go/tools/cache.(*processorListener).run(0xc0003d0080)
	/go/pkg/mod/k8s.io/[email protected]/tools/cache/shared_informer.go:810 +0x6b
k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:75 +0x5a
created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start
	/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:73 +0x85

[ben-z]

I'm running into the same issue. Here's a quick and dirty fix: https://github.com/getsentry/sentry-kubernetes/pull/95

ghcr.io/watonomous/sentry-kubernetes@sha256:a9c62340d633b0dd45a4932f8eee6ad17ac892ce7369331278f8ac46872b512a