druid-exporter icon indicating copy to clipboard operation
druid-exporter copied to clipboard

Error: http panic after upgrading from Druid 25.0.0 to Druid 29.0.0

Open frankgrimes97 opened this issue 11 months ago • 1 comments

We're running opstree/druid-exporter:v0.11 and recently noticed the following errors after upgrading Druid (from 25.0.0 to 29.0.0):

2024/03/06 19:17:12 http: panic serving <IP_REDACED>:<PORT_REDACTED>: interface conversion: interface {} is nil, not string
goroutine 1949 [running]:
net/http.(*conn).serve.func1(0xc0006183c0)
	/usr/local/go/src/net/http/server.go:1801 +0x147
panic(0x8c85a0, 0xc00071d380)
	/usr/local/go/src/runtime/panic.go:975 +0x47a
druid-exporter/listener.DruidHTTPEndpoint.func1(0x9e10c0, 0xc00088e000, 0xc000db2200)
	/go/src/druid-exporter/listener/druid_endpoint.go:140 +0x2578
net/http.HandlerFunc.ServeHTTP(0xc00018fc80, 0x9e10c0, 0xc00088e000, 0xc000db2200)
	/usr/local/go/src/net/http/server.go:2042 +0x44
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0001fe000, 0x9e10c0, 0xc00088e000, 0xc000db2000)
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:210 +0xd3
net/http.serverHandler.ServeHTTP(0xc00021c000, 0x9e10c0, 0xc00088e000, 0xc000db2000)
	/usr/local/go/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0006183c0, 0x9e2940, 0xc0006f2040)
	/usr/local/go/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2969 +0x36c

This seems similar to what was previously reported here (and presumably fixed): https://github.com/opstree/druid-exporter/issues/70

It seems some metrics still get scraped successfully though. e.g. we also see these in the logs:

{"level":"info","msg":"Successfully collected data from druid emitter, druid/indexer","time":"2024-03-06T19:17:13Z"}
{"level":"info","msg":"Successfully collected data from druid emitter, druid/indexer","time":"2024-03-06T19:17:14Z"}

frankgrimes97 avatar Mar 06 '24 19:03 frankgrimes97

Seeing the same issue after upgrading from 26.0.0 to 27.0.0

pchang388 avatar Mar 07 '24 18:03 pchang388