keel icon indicating copy to clipboard operation
keel copied to clipboard

Polling issue with helm provider

Open rusenask opened this issue 5 years ago • 11 comments

For some reason user is seeing this https://kubernetes.slack.com/archives/C6JDWE7M2/p1570114712011300?thread_ts=1570108051.007500&cid=C6JDWE7M2.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x15acf77]

goroutine 124 [running]:
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).addJob(0xc000948ae0, 0xc0008a05a0, 0xc002a3ed60, 0xa, 0x371f580, 0xc000a17c00)
    /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:234 +0x257
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).watch(0xc000948ae0, 0xc0008a05a0, 0xc0002997a0, 0x0, 0x0, 0x0)
    /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:192 +0xa3b
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).Watch(0xc000948ae0, 0xc00097fb00, 0x3, 0x4, 0x0, 0x0)
    /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:134 +0xf2
github.com/keel-hq/keel/trigger/poll.(*DefaultManager).scan(0xc0007417c0, 0x20fb020, 0xc000845840, 0x1, 0x1)
    /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:79 +0x78
github.com/keel-hq/keel/trigger/poll.(*DefaultManager).Start(0xc0007417c0, 0x20fb020, 0xc000845840, 0x0, 0x0)
    /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:48 +0xf4
created by main.setupTriggers
    /go/src/github.com/keel-hq/keel/cmd/keel/main.go:423 +0x6ee

Need to refactor that part to have proper error handling

rusenask avatar Oct 14 '19 14:10 rusenask

Also started seeing this, since 0.16.0. While 0.15.1 runs fine.

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1556a85]

goroutine 62 [running]:
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).addJob(0xc000768cf0, 0xc000b80480, 0xc000ed72d0, 0x9, 0x31d3a60, 0xc000dbf900)
        /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:234 +0x255
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).watch(0xc000768cf0, 0xc000b80480, 0x0, 0x0, 0x4000106, 0x0)
        /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:192 +0x8de
github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).Watch(0xc000768cf0, 0xc000e22b20, 0x3, 0x4, 0x0, 0x0)
        /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:134 +0xf2
github.com/keel-hq/keel/trigger/poll.(*DefaultManager).scan(0xc000573fc0, 0x208da60, 0xc000572d00, 0x1, 0x1)
        /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:79 +0x78
github.com/keel-hq/keel/trigger/poll.(*DefaultManager).Start(0xc000573fc0, 0x208da60, 0xc000572d00, 0x0, 0x0)
        /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:48 +0xf2
created by main.setupTriggers
        /go/src/github.com/keel-hq/keel/cmd/keel/main.go:436 +0x6ed

Chart 0.8.16

Install command:

helm upgrade -i keel keel/keel --namespace=kube-system \
    --set gcr.enabled=true \
    --set gcr.projectId=$PROJECT \
    --set gcr.pubsub.enabled=true \
    --set gcr.clusterName=$PROJECT-$CLUSTER \
    --set secret.name=keel-google \
    --set secret.create=false \
    --set googleApplicationCredentials=notused

amq avatar Mar 16 '20 18:03 amq

could be related to #501

martianoff avatar May 12 '20 23:05 martianoff

Hi,

Unfortunately, the new release still has this issue:

` time="2020-06-09T13:53:23Z" level=info msg=started context=buffer time="2020-06-09T13:53:23Z" level=info msg=started context=watch resource=deployments time="2020-06-09T13:53:23Z" level=info msg=started context=watch resource=statefulsets time="2020-06-09T13:53:23Z" level=info msg=started context=watch resource=daemonsets time="2020-06-09T13:53:23Z" level=info msg=started context=watch resource=cronjobs time="2020-06-09T13:53:24Z" level=debug msg="provider.helm: failed to get config for release" error="policy not specified" namespace=default release=nginx-ingress time="2020-06-09T13:53:24Z" level=debug msg="extension.credentialshelper: helper doesn't support this registry" error="unsupported registry" helper=aws tracked_image="namespace:kube-system,image:index.docker.io/keelhq/keel:latest,provider:helm,trigger:poll,sched:@every 3m,secrets:[]" panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1556a85]

goroutine 43 [running]: github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).addJob(0xc000482fc0, 0xc000401b90, 0xc000bc4270, 0x9, 0x31d3a60, 0xc000c96600) /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:234 +0x255 github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).watch(0xc000482fc0, 0xc000401b90, 0x0, 0x0, 0x3000107, 0x0) /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:192 +0x8de github.com/keel-hq/keel/trigger/poll.(*RepositoryWatcher).Watch(0xc000482fc0, 0xc000ca93b0, 0x2, 0x2, 0x0, 0x0) /go/src/github.com/keel-hq/keel/trigger/poll/watcher.go:134 +0xf2 github.com/keel-hq/keel/trigger/poll.(*DefaultManager).scan(0xc0006f0300, 0x208da60, 0xc0004462c0, 0x1, 0x1) /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:79 +0x78 github.com/keel-hq/keel/trigger/poll.(*DefaultManager).Start(0xc0006f0300, 0x208da60, 0xc0004462c0, 0x0, 0x0) /go/src/github.com/keel-hq/keel/trigger/poll/manager.go:48 +0xf2 created by main.setupTriggers /go/src/github.com/keel-hq/keel/cmd/keel/main.go:436 +0x6ed `

vinfry avatar Jun 09 '20 14:06 vinfry

ah, that seems helm part, need to check that one

rusenask avatar Jun 09 '20 14:06 rusenask

I also noticed a panic when adding a poll trigger with an image in a private registry without matching credentials. #519 should at least fix the panic and output a more meaningful log entry.

hlubek avatar Jul 14 '20 18:07 hlubek

thank you! :)

rusenask avatar Jul 14 '20 18:07 rusenask

Still happening when keel is deployed via helm. How can I get polling from azure private registry?

burn2delete avatar Oct 07 '20 03:10 burn2delete

This appears to still be happening

jamesla avatar Dec 12 '20 22:12 jamesla

Does anyone know why this is occuring?

EDIT: Fingered out at least my issue (as mentioned above): https://github.com/keel-hq/keel/issues/532#issuecomment-846048845

We already have webhooks in-place, so enabling polling was a typo on our side.

Silvenga avatar May 21 '21 14:05 Silvenga

This is actually currently happening to me. Runs fine on 2 completely different nodes but a recently migrated one is getting this issue.

unixbird avatar Oct 05 '22 19:10 unixbird

If I put polling on the Daemonset for this helm chart: https://github.com/democratic-csi/democratic-csi#helm-installation

I can consistently trigger the "invalid memory address or nil pointer dereference" crash of keel.

echo-devnull avatar Feb 06 '23 15:02 echo-devnull