kube-consul-register
kube-consul-register copied to clipboard
Observed a panic "index out of range" on first event.
Problem
It seems on the first event emitted when Kubernetes is adding or removing (could be pods, service, deployment configs, not sure yet) when used as a in-cluster Replicaset is crashing kube-consul-register
.
Stack Trace:
Observed a panic: "index out of range" (runtime error: index out of range)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:72
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:65
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:28
/root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180
<autogenerated>:1
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141
/usr/local/go/src/runtime/asm_amd64.s:2337
E0127 01:05:59.699263 1 runtime.go:66] Observed a panic: "index out of range" (runtime error: index out of range)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:72
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:65
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:51
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58
/usr/local/go/src/runtime/asm_amd64.s:509
/usr/local/go/src/runtime/panic.go:491
/usr/local/go/src/runtime/panic.go:28
/root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180
<autogenerated>:1
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141
/usr/local/go/src/runtime/asm_amd64.s:2337
panic: runtime error: index out of range [recovered]
panic: runtime error: index out of range [recovered]
panic: runtime error: index out of range
goroutine 27 [running]:
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111
panic(0x138cb80, 0x1e558c0)
/usr/local/go/src/runtime/panic.go:491 +0x283
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/runtime/runtime.go:58 +0x111
panic(0x138cb80, 0x1e558c0)
/usr/local/go/src/runtime/panic.go:491 +0x283
github.com/tczekajlo/kube-consul-register/utils.HasLabel(0xc420373800, 0xc4201d0337, 0x5, 0x7fdcc782f2d8)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/utils/utils.go:54 +0x1f8
github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch.func3(0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:266 +0x136
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(0x156f428, 0xc4203ae4c0, 0xc4203ae4b0, 0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:180 +0x5d
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate(0xc42021ab60, 0x14e5c60, 0xc4205e2000, 0x14e5c60, 0xc4205e22c0)
<autogenerated>:1 +0x76
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.NewInformer.func1(0x13a7660, 0xc420170560, 0x13a7660, 0xc420170560)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:246 +0x290
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop(0xc4200c4630, 0xc4203b5050, 0x0, 0x0, 0x0, 0x0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:420 +0x202
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).processLoop(0xc4202ec230)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:131 +0x3c
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).(github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x2a
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil.func1(0xc42060eec0)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:96 +0x5e
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.JitterUntil(0xc420805ec0, 0x3b9aca00, 0x0, 0x1303b01, 0xc420088480)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:97 +0xa1
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait.Until(0xc42060eec0, 0x3b9aca00, 0xc420088480)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/pkg/util/wait/wait.go:52 +0x4d
github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache.(*Controller).Run(0xc4202ec230, 0xc420088480)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/vendor/k8s.io/client-go/tools/cache/controller.go:102 +0x1be
github.com/tczekajlo/kube-consul-register/controller/pods.(*Controller).Watch(0xc420064080)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/controller/pods/controller.go:279 +0x2e1
main.main.func3(0x7fdcc7a34c28, 0xc420064080)
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:141 +0x31
created by main.main
/root/gopath/src/github.com/tczekajlo/kube-consul-register/main.go:140 +0x580
Environment
Kubernetes v1.7.4 Consul 1.0.3 kube-consul-register v0.1.6
My Configurations
kube-consul-register
configurations.
configmap:
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-consul-register
data:
consul_address: "consul"
consul_port: "8500"
consul_scheme: "http"
consul_ca_file: ""
consul_cert_file: ""
consul_key_file: ""
consul_insecure_skip_verify: "false"
consul_token: ""
consul_timeout: "2s"
consul_container_name: "consul"
consul_node_selector: "consul=enabled"
pod_label_selector: "stage"
k8s_tag: ""
register_mode: "single"
register_source: "endpoints"
replicaset:
apiVersion: extensions/v1beta1
kind: ReplicaSet
metadata:
name: kube-consul-register
spec:
replicas: 1
template:
metadata:
labels:
app: kube-consul-register
spec:
containers:
- name: kube-consul-register
image: consul-register:0.1.6
imagePullPolicy: Always
resources:
requests:
cpu: 1
memory: 300Mi
args:
- -logtostderr=true
- -configmap=default/kube-consul-register
Sample application configurations.
deployment:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: render-stg-test
spec:
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 2
template:
metadata:
labels:
app: render-stg-test
stage: tag
annotations:
consul.regiser/enabled: "true"
spec:
containers:
- name: render-stg-test
image: render:0.1.6
imagePullPolicy: Always
command:
- "render"
env:
- name: RENDER_SECRET_ENV
valueFrom:
secretKeyRef:
name: render-secret-test
key: RENDER_SECRET_ENV
- name: RENDER_ENV
value: 'foobar'
dnsPolicy: ClusterFirst
service:
apiVersion: v1
kind: Service
metadata:
name: render-test
spec:
ports:
- protocol: TCP
port: 3000
targetPort: 3000
selector:
app: render-stg-test
type: NodePort
Not sure if it matters but this stopped when i added either the label to the service or matchLabel
to deployment.
But still not seeing services being added.
@webbrandon thanks for created the issue. I'll take a look at it as soon as I found some spare time.
I am trying to get down how to configure it to listen for new service pools and can't seem to get things working. I just updated my configs to run register_mode
as single andregister_source
to pods and got this error again.
I also tried running several instances side by side with different configs and that "seemed" to have stopped all event watching across each instance.
Indeed, with your configuration, I can reproduce your issue.
@webbrandon let's take a look at the first post. There are a few mistakes in configuration.
First off, register_source: "endpoints"
should be register_source: "endpoint"
(without "s"). When you use endpoint as a source you should add the annotation to endpoint (kubectl annotate endpoints my-nginx consul.register/enabled=true
).
Second off, in such configuration also pod_label_selector: "stage"
and the annotation consul.regiser/enabled: "true"
in deployment template is useless.
When you fix your configuration everything should be fine.