Invalid memory address or nil pointer dereference in prometheus/registry.go:591 (v1.10.0)

[alexey-buluy]

trafficstars

Using version v1.10.0 we're periodically observing the issues like that:

2023-05-04T10:41:48.201112+00:00  2023/05/04 10:41:48 http2: panic serving [REDACTED]: runtime error: invalid memory address or nil pointer dereference
2023-05-04T10:41:48.201135+00:00  goroutine 17675201 [running]:
2023-05-04T10:41:48.201145+00:00  net/http.(*http2serverConn).runHandler.func1()
2023-05-04T10:41:48.201150+00:00  	net/http/h2_bundle.go:6042 +0x145
2023-05-04T10:41:48.201153+00:00  panic({0x12340e0, 0x1f248e0})
2023-05-04T10:41:48.201156+00:00  	runtime/panic.go:884 +0x213
2023-05-04T10:41:48.201159+00:00  github.com/prometheus/client_golang/prometheus.processMetric({0x0, 0x0}, 0x0?, 0x0?, 0x0)
2023-05-04T10:41:48.201162+00:00  	github.com/prometheus/client_golang/prometheus/registry.go:591 +0x4b
2023-05-04T10:41:48.201165+00:00  github.com/prometheus/client_golang/prometheus.(*Registry).Gather(0xc000145090)
2023-05-04T10:41:48.201168+00:00  	github.com/prometheus/client_golang/prometheus/registry.go:492 +0x7e7
2023-05-04T10:41:48.201171+00:00  github.com/prometheus/client_golang/prometheus/promhttp.HandlerFor.func1({0x82af21968, 0xc0014fe6c0}, 0xc00099a300)
2023-05-04T10:41:48.201173+00:00  	github.com/prometheus/client_golang/prometheus/promhttp/http.go:126 +0x187
2023-05-04T10:41:48.201179+00:00  net/http.HandlerFunc.ServeHTTP(0xc00022f400?, {0x82af21968?, 0xc0014fe6c0?}, 0xc001464990?)
2023-05-04T10:41:48.201184+00:00  	net/http/server.go:2122 +0x2f
2023-05-04T10:41:48.201187+00:00  github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x82af21968, 0xc0014fe6c0}, 0x82af13c00?)
2023-05-04T10:41:48.201192+00:00  	github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:40 +0xd4
2023-05-04T10:41:48.201195+00:00  net/http.HandlerFunc.ServeHTTP(0x82af13c98?, {0x82af21968?, 0xc0014fe6c0?}, 0x104e627?)
2023-05-04T10:41:48.201197+00:00  	net/http/server.go:2122 +0x2f
2023-05-04T10:41:48.201200+00:00  github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x82af13c98?, 0xc000ef2100?}, 0xc00099a300)
2023-05-04T10:41:48.201202+00:00  	github.com/prometheus/client_golang/prometheus/promhttp/instrument_server.go:101 +0x94
2023-05-04T10:41:48.201211+00:00  net/http.HandlerFunc.ServeHTTP(0xc00013e630?, {0x82af13c98?, 0xc000ef2100?}, 0x0?)
2023-05-04T10:41:48.201214+00:00  	net/http/server.go:2122 +0x2f
2023-05-04T10:41:48.201216+00:00  szs-op/metrics.Init.func1(0xc000ef2100)
2023-05-04T10:41:48.201219+00:00  	szs-op/metrics/metrics.go:29 +0x6d
2023-05-04T10:41:48.201221+00:00  github.com/gin-gonic/gin.(*Context).Next(...)
2023-05-04T10:41:48.201223+00:00  	github.com/gin-gonic/gin/context.go:165
2023-05-04T10:41:48.201227+00:00  github.com/gin-gonic/gin.(*Engine).handleHTTPRequest(0xc000582340, 0xc000ef2100)
2023-05-04T10:41:48.201229+00:00  	github.com/gin-gonic/gin/gin.go:489 +0x63e
2023-05-04T10:41:48.201232+00:00  github.com/gin-gonic/gin.(*Engine).ServeHTTP(0xc000582340, {0x16bdf30?, 0xc00081c0f8}, 0xc00099a300)
2023-05-04T10:41:48.201235+00:00  	github.com/gin-gonic/gin/gin.go:445 +0x1c5
2023-05-04T10:41:48.201237+00:00  net/http.serverHandler.ServeHTTP({0x443091?}, {0x16bdf30, 0xc00081c0f8}, 0xc00099a300)
2023-05-04T10:41:48.201239+00:00  	net/http/server.go:2936 +0x316
2023-05-04T10:41:48.201242+00:00  net/http.initALPNRequest.ServeHTTP({{0x16beee8?, 0xc001215a70?}, 0xc0008ab500?, {0xc000410000?}}, {0x16bdf30, 0xc00081c0f8}, 0xc00099a300)
2023-05-04T10:41:48.201244+00:00  	net/http/server.go:3545 +0x245
2023-05-04T10:41:48.201246+00:00  net/http.(*http2serverConn).runHandler(0x0?, 0xc000a6a7d0?, 0x6f40e6?, 0xc00071e510?)
2023-05-04T10:41:48.201248+00:00  	net/http/h2_bundle.go:6049 +0x83
2023-05-04T10:41:48.201251+00:00  created by net/http.(*http2serverConn).processHeaders
2023-05-04T10:41:48.201253+00:00  	net/http/h2_bundle.go:5762 +0x68a

The exact place where the panic happens is here: https://github.com/prometheus/client_golang/blob/27f0506d6ebbb117b6b697d0552ee5be2502c5f2/prometheus/registry.go#L625 (line 591 in v1.10.0)

The issue is pretty rare and I wasn't been able to reproduce it yet, but it happens periodically. Seems like sometimes an invalid metric object is passed to processMetric which is causing it to panic in an attempt to call Desc() for it.

[imalreadytaken]

same problem. Latest version has not fix it. Maybe add a nil pointer check?

[NullHypothesis]

We just observed this error in v1.17.0. I can confirm that this is a rare issue that's not easy to reproduce.

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for the last 3 months. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity in the next 4 weeks, this issue will be closed (we can always reopen an issue if we need!).