github.com/libgit2/git2go/v31 panic: runtime error: invalid memory address or nil pointer dereference

[andrew-pickin-epi]

Describe the bug

image-automation-controller crashing.

See below.

Steps to reproduce

Can not reproduce at will. This happens about once per week or so.

Note: Flux actually behaves as expected short of the pod restarting. The update to repo did occur collectly:

rp-testing    	imageupdateautomation/fsa-eks-cluster	True 	no updates made; last commit 0a1aecd at 2022-08-11T15:59:53Z	2022-08-12T14:11:37Z	False 

Expected behavior

Pod doesn't restart unexpectedly.

Kubernetes version / Distro / Cloud provider

AWS EKS 1.22

Flux version

flux: v0.26.3 helm-controller: v0.16.0 image-automation-controller: v0.20.0

Git provider

github

Container Registry provider

AWS ECR

Additional context

$ kubectl -n flux-system logs image-automation-controller-675bff6646-q4jlp --previous 
{"level":"info","ts":"2022-08-11T15:59:08.716Z","logger":"controller-runtime.metrics","msg":"Metrics server is starting to listen","addr":":8080"}
{"level":"info","ts":"2022-08-11T15:59:08.717Z","logger":"setup","msg":"starting manager"}
{"level":"info","ts":"2022-08-11T15:59:08.717Z","msg":"Starting server","path":"/metrics","kind":"metrics","addr":"[::]:8080"}
{"level":"info","ts":"2022-08-11T15:59:08.717Z","msg":"Starting server","kind":"health probe","addr":"[::]:9440"}
I0811 15:59:08.818644       1 leaderelection.go:248] attempting to acquire leader lease flux-system/image-automation-controller-leader-election...
I0811 15:59:50.702150       1 leaderelection.go:258] successfully acquired lease flux-system/image-automation-controller-leader-election
{"level":"info","ts":"2022-08-11T15:59:50.702Z","logger":"controller.imageupdateautomation","msg":"Starting EventSource","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","source":"kind source: *v1beta1.ImageUpdateAutomation"}
{"level":"info","ts":"2022-08-11T15:59:50.702Z","logger":"controller.imageupdateautomation","msg":"Starting EventSource","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","source":"kind source: *v1beta1.GitRepository"}
{"level":"info","ts":"2022-08-11T15:59:50.702Z","logger":"controller.imageupdateautomation","msg":"Starting EventSource","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","source":"kind source: *v1beta1.ImagePolicy"}
{"level":"info","ts":"2022-08-11T15:59:50.702Z","logger":"controller.imageupdateautomation","msg":"Starting Controller","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation"}
{"level":"info","ts":"2022-08-11T15:59:50.803Z","logger":"controller.imageupdateautomation","msg":"Starting workers","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","worker count":4}
{"level":"info","ts":"2022-08-11T16:00:00.137Z","logger":"controller.imageupdateautomation","msg":"pushed commit to origin","reconciler group":"image.toolkit.fluxcd.io","reconciler kind":"ImageUpdateAutomation","name":"fsa-eks-cluster","namespace":"rp-testing","revision":"0a1aecd73ee1ae29cbb8ad1a08c98e7f7cb57c1b","branch":"master"}
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x278384c]

goroutine 343 [running]:
github.com/libgit2/git2go/v31.(*sshSmartSubtransport).Close(0xc0020224b0)
	github.com/libgit2/git2go/[email protected]/ssh.go:178 +0x4c
github.com/libgit2/git2go/v31.smartSubtransportCloseCallback(0x15fce06, 0xc000649040)
	github.com/libgit2/git2go/[email protected]/transport.go:399 +0x6f
github.com/libgit2/git2go/v31._Cfunc_git_clone(0xc0021f4350, 0x7ff07a688a90, 0x7ff07a666bf0, 0xc001d4a4e0)
	_cgo_gotypes.go:3228 +0x4c
github.com/libgit2/git2go/v31.Clone.func3(0xc001d64290, 0x6, 0xc001acad00, 0x279be45)
	github.com/libgit2/git2go/[email protected]/clone.go:44 +0x91
github.com/libgit2/git2go/v31.Clone({0xc00164dbc0, 0xc00204f920}, {0xc002da7080, 0x29}, 0xc0027eacf0)
	github.com/libgit2/git2go/[email protected]/clone.go:44 +0x192
github.com/fluxcd/source-controller/pkg/git/libgit2.(*CheckoutBranch).Checkout(0xc001acacc0, {0x7ed040, 0xc00204f920}, {0xc002da7080, 0x29}, {0xc00164dbc0, 0x2c}, 0x0)
	github.com/fluxcd/[email protected]/pkg/git/libgit2/checkout.go:63 +0x236
github.com/fluxcd/image-automation-controller/controllers.cloneInto({0x7ed040, 0xc00204f920}, {0xc00014f0e0, {0xc00164dbc0, 0x22}}, 0xc00204f860, {0xc002da7080, 0x29})
	github.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:529 +0x1a9
github.com/fluxcd/image-automation-controller/controllers.(*ImageUpdateAutomationReconciler).Reconcile(0xc000194d40, {0x7ed078, 0xc0027eab40}, {{{0xc000348ae0, 0x4bf940}, {0xc000348ad0, 0x30}}})
	github.com/fluxcd/image-automation-controller/controllers/imageupdateautomation_controller.go:257 +0x15ad
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0xc0000d62c0, {0x7ed078, 0xc0027eaa80}, {{{0xc000348ae0, 0x4bf940}, {0xc000348ad0, 0x160ced4}}})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:114 +0x26f
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0000d62c0, {0x7ecfd0, 0xc000194c80}, {0x3df760, 0xc001adaaa0})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:311 +0x33e
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0000d62c0, {0x7ecfd0, 0xc000194c80})
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:223 +0x357

Maintenance Acknowledgement

• [X] I am aware of Flux v1's maintenance status

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[pjbgf]

@andrew-pickin-epi you are using an older version of the image-automation-controller. We have improved the performance and resilience of the controller over the last months, so can you upgrade to the latest version and try again please?

[kingdonb]

Please upgrade to the latest version and let us know if this still affects you.

[kingdonb]

We think these issues are solved with the default addition of the managed git transport in IAC.

Please let us know if this still reproduces on the latest version for you.