atlantis icon indicating copy to clipboard operation
atlantis copied to clipboard

Published 20 hours ago • verified publisher icon runatlantis

Panic when closing a pull request and using prometheus metrics

Open neinkob15 opened this issue 3 years ago • 3 comments

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Atlantis server does panic when closing a pull-request in Bitbucket and prometheus metrics activated

Reproduction Steps

  1. setup atlantis (using Version v0.19.7) for Bitbucket using this repo-config:

      ATLANTIS_REPO_CONFIG_JSON='... 
      "metrics": {
          "prometheus": {
              "endpoint": "/metrics"
           }
      }'

  2. close an open pull-request in Bitbucket

Logs

Logs 
{"level":"info","ts":"2022-07-29T07:34:17.203Z","caller":"server/server.go:853","msg":"Atlantis started - listening on port 4141","json":{}}
{"level":"info","ts":"2022-07-29T07:34:17.203Z","caller":"scheduled/executor_service.go:46","msg":"Scheduled Executor Service started","json":{}}
{"level":"info","ts":"2022-07-29T07:34:53.133Z","caller":"events/events_controller.go:379","msg":"identified event as type \"closed\"","json":{}}
2022/07/29 07:34:53 PANIC: descriptor Desc{fqName: "atlantis_pullclosed.cleanup_execution_success", help: "atlantis_pullclosed.cleanup_execution_success counter", constLabels: {}, variableLabels: []} is invalid: "atlantis_pullclosed.cleanup_execution_success" is not a valid metric name
goroutine 137 [running]:
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
	github.com/urfave/[email protected]/recovery.go:159 +0xc7
panic({0x10d5fe0, 0xc0001212a0})
	runtime/panic.go:1038 +0x215
github.com/uber-go/tally/prometheus.NewReporter.func1({0x1535d60, 0xc0001212a0})
	github.com/uber-go/[email protected]+incompatible/prometheus/reporter.go:306 +0x68
github.com/uber-go/tally/prometheus.(*reporter).AllocateCounter(0xc000a5e6c0, {0xc000061290, 0x2d}, 0x1518030)
	github.com/uber-go/[email protected]+incompatible/prometheus/reporter.go:366 +0x11f
github.com/uber-go/tally.(*scope).Counter(0xc0001da780, {0x1310133, 0x40c66d})
	github.com/uber-go/[email protected]+incompatible/scope.go:280 +0x173
github.com/runatlantis/atlantis/server/events.(*InstrumentedPullClosedExecutor).CleanUpPull(_, {{0xc000061200, 0x2b}, {0xc000061200, 0x19}, {0xc00006121a, 0x11}, {0xc000164900, 0x77}, {0xc000132180, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/instrumented_pull_closed_executor.go:35 +0x1a3
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handlePullRequestEvent(_, {_, _}, {{0xc000061200, 0x2b}, {0xc000061200, 0x19}, {0xc00006121a, 0x11}, {0xc000164900, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:452 +0x5f5
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleBitbucketServerPullRequestEvent(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, {0xc0004c2030, 0xa}, {0xc0004a7000, 0xc0004d14b8, 0x83048f}, {0xc0000610e0, 0x24})
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:380 +0x590
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleBitbucketServerPost(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524a00)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:243 +0x669
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524a00)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:126 +0x375
net/http.HandlerFunc.ServeHTTP(0xc000524900, {0x7efdf55252b8, 0xc00000e2f0}, 0x416fc6)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000166c00, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524800)
	github.com/gorilla/[email protected]/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7efdf55252b8, 0xc00000e2f0}, 0x11831a0, 0xc0000303e0)
	github.com/urfave/[email protected]/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x8, {0x7efdf55252b8, 0xc00000e2f0}, 0x7efdf5846108, 0x8)
	github.com/urfave/[email protected]/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1539c20, 0xc00000d0f8}, 0xc00000d140}, {0x7efdf55252b8, 0xc00000e2f0}, 0x3)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP(0xc000128e40, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524800, 0xc0000303c0)
	github.com/runatlantis/atlantis/server/middleware.go:68 +0x448
github.com/urfave/negroni.middleware.ServeHTTP({{0x15367e0, 0xc000128e40}, 0xc00000d128}, {0x7efdf55252b8, 0xc00000e2f0}, 0x20)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000686400, {0x7efdf55252b8, 0xc00000e2f0}, 0xc0005a2a48, 0x40ac8a)
	github.com/urfave/[email protected]/recovery.go:193 +0x8f
github.com/urfave/negroni.middleware.ServeHTTP({{0x1537240, 0xc000223db0}, 0xc00000d110}, {0x7efdf55252b8, 0xc00000e2f0}, 0x0)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc00083c8a0, {0x154a398, 0xc0001de1c0}, 0x10)
	github.com/urfave/[email protected]/negroni.go:96 +0x110
net/http.serverHandler.ServeHTTP({0x1547c50}, {0x154a398, 0xc0001de1c0}, 0xc000524800)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc0004f0000, {0x15558f0, 0xc00077a180})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8
2022/07/29 07:34:53 http: panic serving 100.64.33.1:36968: runtime error: invalid memory address or nil pointer dereference
goroutine 137 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1802 +0xb9
panic({0x110f480, 0x1d9c420})
	runtime/panic.go:1047 +0x266
github.com/urfave/negroni.(*Recovery).ServeHTTP.func1()
	github.com/urfave/[email protected]/recovery.go:166 +0x23c
panic({0x10d5fe0, 0xc0001212a0})
	runtime/panic.go:1038 +0x215
github.com/uber-go/tally/prometheus.NewReporter.func1({0x1535d60, 0xc0001212a0})
	github.com/uber-go/[email protected]+incompatible/prometheus/reporter.go:306 +0x68
github.com/uber-go/tally/prometheus.(*reporter).AllocateCounter(0xc000a5e6c0, {0xc000061290, 0x2d}, 0x1518030)
	github.com/uber-go/[email protected]+incompatible/prometheus/reporter.go:366 +0x11f
github.com/uber-go/tally.(*scope).Counter(0xc0001da780, {0x1310133, 0x40c66d})
	github.com/uber-go/[email protected]+incompatible/scope.go:280 +0x173
github.com/runatlantis/atlantis/server/events.(*InstrumentedPullClosedExecutor).CleanUpPull(_, {{0xc000061200, 0x2b}, {0xc000061200, 0x19}, {0xc00006121a, 0x11}, {0xc000164900, 0x77}, {0xc000132180, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/events/instrumented_pull_closed_executor.go:35 +0x1a3
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handlePullRequestEvent(_, {_, _}, {{0xc000061200, 0x2b}, {0xc000061200, 0x19}, {0xc00006121a, 0x11}, {0xc000164900, ...}, ...}, ...)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:452 +0x5f5
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleBitbucketServerPullRequestEvent(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, {0xc0004c2030, 0xa}, {0xc0004a7000, 0xc0004d14b8, 0x83048f}, {0xc0000610e0, 0x24})
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:380 +0x590
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).handleBitbucketServerPost(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524a00)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:243 +0x669
github.com/runatlantis/atlantis/server/controllers/events.(*VCSEventsController).Post(0xc000534160, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524a00)
	github.com/runatlantis/atlantis/server/controllers/events/events_controller.go:126 +0x375
net/http.HandlerFunc.ServeHTTP(0xc000524900, {0x7efdf55252b8, 0xc00000e2f0}, 0x416fc6)
	net/http/server.go:2047 +0x2f
github.com/gorilla/mux.(*Router).ServeHTTP(0xc000166c00, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524800)
	github.com/gorilla/[email protected]/mux.go:210 +0x1cf
github.com/urfave/negroni.Wrap.func1({0x7efdf55252b8, 0xc00000e2f0}, 0x11831a0, 0xc0000303e0)
	github.com/urfave/[email protected]/negroni.go:46 +0x4b
github.com/urfave/negroni.HandlerFunc.ServeHTTP(0x8, {0x7efdf55252b8, 0xc00000e2f0}, 0x7efdf5846108, 0x8)
	github.com/urfave/[email protected]/negroni.go:29 +0x33
github.com/urfave/negroni.middleware.ServeHTTP({{0x1539c20, 0xc00000d0f8}, 0xc00000d140}, {0x7efdf55252b8, 0xc00000e2f0}, 0x3)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/runatlantis/atlantis/server.(*RequestLogger).ServeHTTP(0xc000128e40, {0x7efdf55252b8, 0xc00000e2f0}, 0xc000524800, 0xc0000303c0)
	github.com/runatlantis/atlantis/server/middleware.go:68 +0x448
github.com/urfave/negroni.middleware.ServeHTTP({{0x15367e0, 0xc000128e40}, 0xc00000d128}, {0x7efdf55252b8, 0xc00000e2f0}, 0x20)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Recovery).ServeHTTP(0xc000686400, {0x7efdf55252b8, 0xc00000e2f0}, 0xc0005a2a48, 0x40ac8a)
	github.com/urfave/[email protected]/recovery.go:193 +0x8f
github.com/urfave/negroni.middleware.ServeHTTP({{0x1537240, 0xc000223db0}, 0xc00000d110}, {0x7efdf55252b8, 0xc00000e2f0}, 0x0)
	github.com/urfave/[email protected]/negroni.go:38 +0xb6
github.com/urfave/negroni.(*Negroni).ServeHTTP(0xc00083c8a0, {0x154a398, 0xc0001de1c0}, 0x10)
	github.com/urfave/[email protected]/negroni.go:96 +0x110
net/http.serverHandler.ServeHTTP({0x1547c50}, {0x154a398, 0xc0001de1c0}, 0xc000524800)
	net/http/server.go:2879 +0x43b
net/http.(*conn).serve(0xc0004f0000, {0x15558f0, 0xc00077a180})
	net/http/server.go:1930 +0xb08
created by net/http.(*Server).Serve
	net/http/server.go:3034 +0x4e8

Environment details

Additional Context

neinkob15 avatar Jul 29 '22 07:07 neinkob15

is this still happening with v0.19.8?

jamengual avatar Aug 26 '22 04:08 jamengual

Hi, yes I just upgraded my Atlantis to v0.19.8, closed an open pull-request (by deleting) and the same error panic appears:

{"level":"info","ts":"2022-08-26T08:11:19.782Z","caller":"events/events_controller.go:379","msg":"identified event as type "closed"","json":{}} 2022/08/26 08:11:19 PANIC: descriptor Desc{fqName: "atlantis_pullclosed.cleanup_execution_success", help: "atlantis_pullclosed.cleanup_execution_success counter", constLabels: {}, variableLabels: []} is invalid: "atlantis_pullclosed.cleanup_execution_success" is not a valid metric name

neinkob15 avatar Aug 26 '22 08:08 neinkob15

Duplicated https://github.com/runatlantis/atlantis/issues/2379

zer0beat avatar Sep 20 '22 21:09 zer0beat

We also faced the same issue, after updating to atlantis v0.19.9, it was sorted.

Pardeep009 avatar Feb 25 '23 19:02 Pardeep009

Thanks for confirming @Pardeep009 . I'll close this then since upgrading fixes it.

Please use the latest version if possible.

nitrocode avatar Feb 25 '23 20:02 nitrocode