thanos
thanos copied to clipboard
Published
20 hours ago •
thanos-io
thanos-io
query: regression in v0.36.0 - concurrent_gate_queries periodically reaching limit
trafficstars
Thanos, Prometheus and Golang version used:
Prometheus: v2.53.1 Thanos: v0.36.0
Object Storage Provider:
S3
What happened:
Moving from v0.35.1 to v0.36.0 repeatedly resulted in our query maxing out concurrent_gate_queries, and with 0 available queries would hang firing our heartbeat alerts.
You can observe running v0.36.0 - we constantly have multiple concurrent queries running. After some period of time (this frequently happened on the hour, but not every hour) there are 0 available concurrent "workers".
Deploying v0.35.0 and then v0.35.1 results in a flat graph with 0 - 1 inflight queries running, which is the usual behavior for us, 30d graph:
What you expected to happen:
thanos_query_concurrent_gate_queries_in_flight to remain roughly the same between the version.
How to reproduce it (as minimally and precisely as possible):
Running v0.36.0 with some load, more load makes the issue more apparent. We observed this in dev, but it wasn't as obvious and we never capped out:
Full logs to relevant components:
Anything else we need to know:
Worth a mention - because of https://github.com/thanos-io/thanos/issues/7400 we ran main-2024-06-11-8ec0c24 from 2024-08-02. We then deployed v0.36.0 on 2024-08-06. And rolled back to v0.35.1 few hours ago today (2024-08-09).
This isn't full blown 0 available, but running in dev with 17/20 or 15/20 available:
goroutine profile: total 55
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x48a265 0x488098 0x4e760e 0x4e75f6 0x4e738d 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x48a264 syscall.Syscall+0x24 /usr/local/go/src/syscall/syscall_linux.go:69
# 0x488097 syscall.read+0x37 /usr/local/go/src/syscall/zsyscall_linux_amd64.go:721
# 0x4e760d syscall.Read+0x2ad /usr/local/go/src/syscall/syscall_unix.go:181
# 0x4e75f5 internal/poll.ignoringEINTRIO+0x295 /usr/local/go/src/internal/poll/fd_unix.go:736
# 0x4e738c internal/poll.(*FD).Read+0x2c /usr/local/go/src/internal/poll/fd_unix.go:160
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
Another replica:
goroutine profile: total 79
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
5 @ 0x43e6ee 0x44eb65 0xf6be1c 0xf9fd49 0xf9fd36 0xf9efbf 0xf9dd1a 0xf9ee25 0x1715f34 0x155d442 0x16c63e6 0x1dbee16 0x1dbebf7 0x4719a1
# 0xf6be1b google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader+0x7b /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:330
# 0xf9fd48 google.golang.org/grpc/internal/transport.(*Stream).RecvCompress+0xc8 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:345
# 0xf9fd35 google.golang.org/grpc.(*csAttempt).recvMsg+0xb5 /go/pkg/mod/google.golang.org/[email protected]/stream.go:1071
# 0xf9efbe google.golang.org/grpc.(*clientStream).RecvMsg.func1+0x1e /go/pkg/mod/google.golang.org/[email protected]/stream.go:928
# 0xf9dd19 google.golang.org/grpc.(*clientStream).withRetry+0x139 /go/pkg/mod/google.golang.org/[email protected]/stream.go:777
# 0xf9ee24 google.golang.org/grpc.(*clientStream).RecvMsg+0x124 /go/pkg/mod/google.golang.org/[email protected]/stream.go:927
# 0x1715f33 github.com/thanos-io/thanos/pkg/tracing/interceptors.(*monitoredClientStream).RecvMsg+0x53 /app/pkg/tracing/interceptors/client.go:76
# 0x155d441 github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredClientStream).RecvMsg+0x61 /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/client.go:62
# 0x16c63e5 github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesClient).Recv+0x45 /app/pkg/store/storepb/rpc.pb.go:1015
# 0x1dbee15 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0xf5 /app/pkg/store/proxy_merge.go:608
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
5 @ 0x43e6ee 0x44eb65 0xf9c04c 0x4719a1
# 0xf9c04b google.golang.org/grpc.newClientStreamWithParams.func4+0x8b /go/pkg/mod/google.golang.org/[email protected]/stream.go:392
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
4 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
3 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
3 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fceea6 0x1fceae8 0x47d19f 0x1fce9ad 0x1fce8ad 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fceea5 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).init+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:170
# 0x1fceae7 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fce9ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fce8ac github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd9d24 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd9d23 github.com/thanos-io/promql-engine/execution/binary.(*vectorOperator).init.func1+0x43 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/vector.go:181
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
Another:
goroutine profile: total 71
12 @ 0x43e6ee 0x44eb65 0xf2bf75 0x4719a1
# 0xf2bf74 google.golang.org/grpc/internal/grpcsync.(*CallbackSerializer).run+0x114 /go/pkg/mod/google.golang.org/[email protected]/internal/grpcsync/callback_serializer.go:76
4 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x5646f7 0x4dfb70 0x9b4ae5 0x9b4ab4 0x9b5225 0xf5d045 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x5646f6 bufio.(*Reader).Read+0x196 /usr/local/go/src/bufio/bufio.go:244
# 0x4dfb6f io.ReadAtLeast+0x8f /usr/local/go/src/io/io.go:335
# 0x9b4ae4 io.ReadFull+0x64 /usr/local/go/src/io/io.go:354
# 0x9b4ab3 golang.org/x/net/http2.readFrameHeader+0x33 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:237
# 0x9b5224 golang.org/x/net/http2.(*Framer).ReadFrame+0x84 /go/pkg/mod/golang.org/x/[email protected]/http2/frame.go:501
# 0xf5d044 google.golang.org/grpc/internal/transport.(*http2Client).reader+0x244 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1602
4 @ 0x43e6ee 0x44eb65 0xf49273 0xf49986 0xf53065 0x4719a1
# 0xf49272 google.golang.org/grpc/internal/transport.(*controlBuffer).get+0x112 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:418
# 0xf49985 google.golang.org/grpc/internal/transport.(*loopyWriter).run+0x85 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/controlbuf.go:551
# 0xf53064 google.golang.org/grpc/internal/transport.newHTTP2Client.func6+0x84 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:463
4 @ 0x43e6ee 0x44eb65 0xf5d811 0x4719a1
# 0xf5d810 google.golang.org/grpc/internal/transport.(*http2Client).keepalive+0x150 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/http2_client.go:1676
4 @ 0x43e6ee 0x44eb65 0xf6be1c 0xf9fd49 0xf9fd36 0xf9efbf 0xf9dd1a 0xf9ee25 0x1715f34 0x155d442 0x16c63e6 0x1dbee16 0x1dbebf7 0x4719a1
# 0xf6be1b google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader+0x7b /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:330
# 0xf9fd48 google.golang.org/grpc/internal/transport.(*Stream).RecvCompress+0xc8 /go/pkg/mod/google.golang.org/[email protected]/internal/transport/transport.go:345
# 0xf9fd35 google.golang.org/grpc.(*csAttempt).recvMsg+0xb5 /go/pkg/mod/google.golang.org/[email protected]/stream.go:1071
# 0xf9efbe google.golang.org/grpc.(*clientStream).RecvMsg.func1+0x1e /go/pkg/mod/google.golang.org/[email protected]/stream.go:928
# 0xf9dd19 google.golang.org/grpc.(*clientStream).withRetry+0x139 /go/pkg/mod/google.golang.org/[email protected]/stream.go:777
# 0xf9ee24 google.golang.org/grpc.(*clientStream).RecvMsg+0x124 /go/pkg/mod/google.golang.org/[email protected]/stream.go:927
# 0x1715f33 github.com/thanos-io/thanos/pkg/tracing/interceptors.(*monitoredClientStream).RecvMsg+0x53 /app/pkg/tracing/interceptors/client.go:76
# 0x155d441 github.com/grpc-ecosystem/go-grpc-middleware/v2/interceptors.(*monitoredClientStream).RecvMsg+0x61 /go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware/[email protected]/interceptors/client.go:62
# 0x16c63e5 github.com/thanos-io/thanos/pkg/store/storepb.(*storeSeriesClient).Recv+0x45 /app/pkg/store/storepb/rpc.pb.go:1015
# 0x1dbee15 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0xf5 /app/pkg/store/proxy_merge.go:608
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
4 @ 0x43e6ee 0x44eb65 0xf7108e 0x4719a1
# 0xf7108d google.golang.org/grpc/internal/resolver/dns.(*dnsResolver).watcher+0x1ed /go/pkg/mod/google.golang.org/[email protected]/internal/resolver/dns/dns_resolver.go:212
4 @ 0x43e6ee 0x44eb65 0xf9c04c 0x4719a1
# 0xf9c04b google.golang.org/grpc.newClientStreamWithParams.func4+0x8b /go/pkg/mod/google.golang.org/[email protected]/stream.go:392
3 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a110b 0x5641c3 0x5642f3 0x7a6fbc 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a110a net/http.(*connReader).Read+0x14a /usr/local/go/src/net/http/server.go:791
# 0x5641c2 bufio.(*Reader).fill+0x102 /usr/local/go/src/bufio/bufio.go:113
# 0x5642f2 bufio.(*Reader).Peek+0x52 /usr/local/go/src/bufio/bufio.go:151
# 0x7a6fbb net/http.(*conn).serve+0x75b /usr/local/go/src/net/http/server.go:2049
3 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1dbfe3f 0x1dc690f 0x1dc6e32 0x1dbbdf2 0x1db6abf 0x1de7628 0x1de6f5d 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1dbfe3e github.com/thanos-io/thanos/pkg/store.(*eagerRespSet).Next+0x1e /app/pkg/store/proxy_merge.go:721
# 0x1dc690e github.com/thanos-io/thanos/pkg/losertree.(*Tree[...]).moveNext+0x4e /app/pkg/losertree/tree.go:62
# 0x1dc6e31 github.com/thanos-io/thanos/pkg/losertree.New[...]+0x151 /app/pkg/losertree/tree.go:25
# 0x1dbbdf1 github.com/thanos-io/thanos/pkg/store.NewProxyResponseLoserTree+0xd1 /app/pkg/store/proxy_merge.go:188
# 0x1db6abe github.com/thanos-io/thanos/pkg/store.(*ProxyStore).Series+0x11be /app/pkg/store/proxy.go:365
# 0x1de7627 github.com/thanos-io/thanos/pkg/query.(*querier).selectFn+0x327 /app/pkg/query/querier.go:346
# 0x1de6f5c github.com/thanos-io/thanos/pkg/query.(*querier).Select.func1+0x21c /app/pkg/query/querier.go:293
2 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
2 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x589da5 0x59c3c5 0x7a0cd7 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x589da4 net.(*netFD).Read+0x24 /usr/local/go/src/net/fd_posix.go:55
# 0x59c3c4 net.(*conn).Read+0x44 /usr/local/go/src/net/net.go:185
# 0x7a0cd6 net/http.(*connReader).backgroundRead+0x36 /usr/local/go/src/net/http/server.go:683
2 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1ff0fba 0x47d19f 0x1ff0f0c 0x1ff0ebf 0x1ff04f1 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1ff0fb9 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:182
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1ff0f0b sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1ff0ebe github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:181
# 0x1ff04f0 github.com/thanos-io/promql-engine/storage/prometheus.(*vectorSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/vector_selector.go:108
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x40f309 0x46dfa9 0x6266d3 0x4719a1
# 0x46dfa8 os/signal.signal_recv+0x28 /usr/local/go/src/runtime/sigqueue.go:152
# 0x6266d2 os/signal.loop+0x12 /usr/local/go/src/os/signal/signal_unix.go:23
1 @ 0x4336b1 0x46ba1d 0x1047f71 0x1047da5 0x1044846 0x157d788 0x157e285 0x7a8289 0x7a9ba2 0x7aaf6e 0x7a6e54 0x4719a1
# 0x46ba1c runtime/pprof.runtime_goroutineProfileWithLabels+0x1c /usr/local/go/src/runtime/mprof.go:844
# 0x1047f70 runtime/pprof.writeRuntimeProfile+0xb0 /usr/local/go/src/runtime/pprof/pprof.go:734
# 0x1047da4 runtime/pprof.writeGoroutine+0x44 /usr/local/go/src/runtime/pprof/pprof.go:694
# 0x1044845 runtime/pprof.(*Profile).WriteTo+0x145 /usr/local/go/src/runtime/pprof/pprof.go:329
# 0x157d787 net/http/pprof.handler.ServeHTTP+0x4a7 /usr/local/go/src/net/http/pprof/pprof.go:267
# 0x157e284 net/http/pprof.Index+0xe4 /usr/local/go/src/net/http/pprof/pprof.go:384
# 0x7a8288 net/http.HandlerFunc.ServeHTTP+0x28 /usr/local/go/src/net/http/server.go:2141
# 0x7a9ba1 net/http.(*ServeMux).ServeHTTP+0x141 /usr/local/go/src/net/http/server.go:2519
# 0x7aaf6d net/http.serverHandler.ServeHTTP+0x8d /usr/local/go/src/net/http/server.go:2943
# 0x7a6e53 net/http.(*conn).serve+0x5f3 /usr/local/go/src/net/http/server.go:2014
1 @ 0x43e6ee 0x4099d9 0x409852 0x1dbeef2 0x1dbebf7 0x4719a1
# 0x1dbeef1 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2.3+0x1d1 /app/pkg/store/proxy_merge.go:617
# 0x1dbebf6 github.com/thanos-io/thanos/pkg/store.newEagerRespSet.func2+0x256 /app/pkg/store/proxy_merge.go:650
1 @ 0x43e6ee 0x409c4d 0x409852 0x215d70d 0x626b49 0x4719a1
# 0x215d70c main.main.func2+0x2c /app/cmd/thanos/main.go:129
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x409c4d 0x409852 0x6269d5 0x215d138 0x43e27b 0x4719a1
# 0x6269d4 github.com/oklog/run.(*Group).Run+0x154 /go/pkg/mod/github.com/oklog/[email protected]/group.go:43
# 0x215d137 main.main+0x19f7 /app/cmd/thanos/main.go:169
# 0x43e27a runtime.main+0x2ba /usr/local/go/src/runtime/proc.go:267
1 @ 0x43e6ee 0x409c4d 0x409872 0x1de6b73 0x1de5364 0x1fefcdc 0x1fefb68 0x47d19f 0x1fef9b3 0x1fef946 0x1feb405 0x1feb388 0x47d19f 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x1de6b72 github.com/thanos-io/thanos/pkg/query.(*querier).Select.func2+0x92 /app/pkg/query/querier.go:308
# 0x1de5363 github.com/thanos-io/thanos/pkg/query.(*lazySeriesSet).Next+0x23 /app/pkg/query/iter.go:298
# 0x1fefcdb github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).loadSeries+0x13b /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:72
# 0x1fefb67 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fef9b2 sync.(*Once).Do+0x92 /usr/local/go/src/sync/once.go:65
# 0x1fef945 github.com/thanos-io/promql-engine/storage/prometheus.(*seriesSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/series_selector.go:55
# 0x1feb404 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).loadSeries+0x44 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:43
# 0x1feb387 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries.func1+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4e75da 0x4e75c8 0x4f2f92 0x4f2f8a 0x1d4fcb2 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4e75d9 internal/poll.(*pollDesc).waitRead+0x279 /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4e75c7 internal/poll.(*FD).Read+0x267 /usr/local/go/src/internal/poll/fd_unix.go:164
# 0x4f2f91 os.(*File).read+0x51 /usr/local/go/src/os/file_posix.go:29
# 0x4f2f89 os.(*File).Read+0x49 /usr/local/go/src/os/file.go:118
# 0x1d4fcb1 github.com/fsnotify/fsnotify.(*Watcher).readEvents+0xd1 /go/pkg/mod/github.com/fsnotify/[email protected]/backend_inotify.go:483
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0x7ab3c4 0x159fc8e 0x159f40b 0x97f596 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0x7ab3c3 net/http.(*Server).Serve+0x363 /usr/local/go/src/net/http/server.go:3061
# 0x159fc8d github.com/prometheus/exporter-toolkit/web.Serve+0x34d /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:317
# 0x159f40a github.com/prometheus/exporter-toolkit/web.ServeMultiple.func1+0x2a /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:271
# 0x97f595 golang.org/x/sync/errgroup.(*Group).Go.func1+0x55 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:78
1 @ 0x43e6ee 0x4370d7 0x46be45 0x4e62e7 0x4eb7cc 0x4eb7ba 0x58bdc9 0x5a53de 0x5a4590 0xf8ebf4 0x2033dd6 0x2168b4f 0x626b49 0x4719a1
# 0x46be44 internal/poll.runtime_pollWait+0x84 /usr/local/go/src/runtime/netpoll.go:343
# 0x4e62e6 internal/poll.(*pollDesc).wait+0x26 /usr/local/go/src/internal/poll/fd_poll_runtime.go:84
# 0x4eb7cb internal/poll.(*pollDesc).waitRead+0x2ab /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
# 0x4eb7b9 internal/poll.(*FD).Accept+0x299 /usr/local/go/src/internal/poll/fd_unix.go:611
# 0x58bdc8 net.(*netFD).accept+0x28 /usr/local/go/src/net/fd_unix.go:172
# 0x5a53dd net.(*TCPListener).accept+0x1d /usr/local/go/src/net/tcpsock_posix.go:152
# 0x5a458f net.(*TCPListener).Accept+0x2f /usr/local/go/src/net/tcpsock.go:315
# 0xf8ebf3 google.golang.org/grpc.(*Server).Serve+0x4b3 /go/pkg/mod/google.golang.org/[email protected]/server.go:875
# 0x2033dd5 github.com/thanos-io/thanos/pkg/server/grpc.(*Server).ListenAndServe+0x195 /app/pkg/server/grpc/grpc.go:168
# 0x2168b4e main.runQuery.func14+0x2e /app/cmd/thanos/query.go:825
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x105ad5f 0x4719a1
# 0x105ad5e go.opencensus.io/stats/view.(*worker).start+0x9e /go/pkg/mod/[email protected]/stats/view/worker.go:292
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x2169e2e 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x2169e2d main.runQuery.func5+0x3ad /app/cmd/thanos/query.go:611
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x17207ab 0x216b8dc 0x626b49 0x4719a1
# 0x17207aa github.com/thanos-io/thanos/pkg/runutil.Repeat+0xca /app/pkg/runutil/runutil.go:94
# 0x216b8db main.prepareEndpointSet.func2+0x7b /app/cmd/thanos/query.go:917
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x1d51f05 0x216ad85 0x626b49 0x4719a1
# 0x1d51f04 github.com/prometheus/prometheus/discovery/file.(*Discovery).Run+0x364 /go/pkg/mod/github.com/prometheus/[email protected]/discovery/file/file.go:256
# 0x216ad84 main.runQuery.func1+0x24 /app/cmd/thanos/query.go:576
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215dbc5 0x215d53f 0x626b49 0x4719a1
# 0x215dbc4 main.interrupt+0x104 /app/cmd/thanos/main.go:180
# 0x215d53e main.main.func4+0x1e /app/cmd/thanos/main.go:153
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x215de92 0x215d4c5 0x626b49 0x4719a1
# 0x215de91 main.reload+0xf1 /app/cmd/thanos/main.go:193
# 0x215d4c4 main.main.func6+0x24 /app/cmd/thanos/main.go:163
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0x216aaee 0x626b49 0x4719a1
# 0x216aaed main.runQuery.func3+0x10d /app/cmd/thanos/query.go:585
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44eb65 0xe100c8 0x4719a1
# 0xe100c7 github.com/baidubce/bce-sdk-go/util/log.NewLogger.func1+0xa7 /go/pkg/mod/github.com/baidubce/[email protected]/util/log/logger.go:375
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fceea6 0x1fceae8 0x47d19f 0x1fce9ad 0x1fce8ad 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fceea5 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).init+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:170
# 0x1fceae7 github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fce9ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fce8ac github.com/thanos-io/promql-engine/execution/aggregate.(*kAggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/khashaggregate.go:146
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fd9d24 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fd9d23 github.com/thanos-io/promql-engine/execution/binary.(*vectorOperator).init.func1+0x43 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/vector.go:181
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x1fe0377 0x1fdeee8 0x47d19f 0x1fdedad 0x1fdecad 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fcd306 0x1fcd0da 0x1fcc0c8 0x47d19f 0x1fcbf75 0x1fcbe6d 0x1fe0b1d 0x1fe308b 0x47d19f 0x1fe2fec 0x1fe2f9f 0x1fe2d31 0x1fd71d8 0x1fd61e8 0x47d19f 0x1fd60ad 0x1fd5fad
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x1fe0376 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries+0x236 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:214
# 0x1fdeee7 github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fdedac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fdecac github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:84
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fcd305 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeScalarTables+0x45 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:221
# 0x1fcd0d9 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).initializeTables+0x39 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:195
# 0x1fcc0c7 github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fcbf74 sync.(*Once).Do+0x134 /usr/local/go/src/sync/once.go:65
# 0x1fcbe6c github.com/thanos-io/promql-engine/execution/aggregate.(*aggregate).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/aggregate/hashaggregate.go:100
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe308a github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init.func1+0x4a /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:110
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fe2feb sync.(*Once).Do+0x6b /usr/local/go/src/sync/once.go:65
# 0x1fe2f9e github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).init+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:109
# 0x1fe2d30 github.com/thanos-io/promql-engine/execution/exchange.(*duplicateLabelCheckOperator).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/duplicate_label.go:89
# 0x1fd71d7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).loadSeries+0x37 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:190
# 0x1fd61e7 github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series.func2+0x27 /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fd60ac sync.(*Once).Do+0x12c /usr/local/go/src/sync/once.go:65
# 0x1fd5fac github.com/thanos-io/promql-engine/execution/binary.(*scalarOperator).Series+0x2c /go/pkg/mod/github.com/thanos-io/[email protected]/execution/binary/scalar.go:102
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d845 0x47e748 0x97f425 0x159f385 0x159f876 0x16e522d 0x216976f 0x626b49 0x4719a1
# 0x46d844 sync.runtime_Semacquire+0x24 /usr/local/go/src/runtime/sema.go:62
# 0x47e747 sync.(*WaitGroup).Wait+0x47 /usr/local/go/src/sync/waitgroup.go:116
# 0x97f424 golang.org/x/sync/errgroup.(*Group).Wait+0x24 /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:56
# 0x159f384 github.com/prometheus/exporter-toolkit/web.ServeMultiple+0x144 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:274
# 0x159f875 github.com/prometheus/exporter-toolkit/web.ListenAndServe+0x455 /go/pkg/mod/github.com/prometheus/[email protected]/web/tls_config.go:307
# 0x16e522c github.com/thanos-io/thanos/pkg/server/http.(*Server).ListenAndServe+0x22c /app/pkg/server/http/http.go:85
# 0x216976e main.runQuery.func10+0x2e /app/cmd/thanos/query.go:768
# 0x626b48 github.com/oklog/run.(*Group).Run.func1+0x28 /go/pkg/mod/github.com/oklog/[email protected]/group.go:38
1 @ 0x43e6ee 0x44fb98 0x44fb6f 0x46d905 0x47ce3d 0x47d130 0x47d0fe 0x1feb1ed 0x1feb186 0x1fec8ba 0x47d19f 0x1fec814 0x1fec7bf 0x1febd11 0x1fe0b1d 0x1fe076e 0x4719a1
# 0x46d904 sync.runtime_SemacquireMutex+0x24 /usr/local/go/src/runtime/sema.go:77
# 0x47ce3c sync.(*Mutex).lockSlow+0x15c /usr/local/go/src/sync/mutex.go:171
# 0x47d12f sync.(*Mutex).Lock+0x4f /usr/local/go/src/sync/mutex.go:90
# 0x47d0fd sync.(*Once).doSlow+0x1d /usr/local/go/src/sync/once.go:70
# 0x1feb1ec sync.(*Once).Do+0x8c /usr/local/go/src/sync/once.go:65
# 0x1feb185 github.com/thanos-io/promql-engine/storage/prometheus.(*filteredSelector).GetSeries+0x25 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/filtered_selector.go:34
# 0x1fec8b9 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries.func1+0x59 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:223
# 0x47d19e sync.(*Once).doSlow+0xbe /usr/local/go/src/sync/once.go:74
# 0x1fec813 sync.(*Once).Do+0x73 /usr/local/go/src/sync/once.go:65
# 0x1fec7be github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).loadSeries+0x1e /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:222
# 0x1febd10 github.com/thanos-io/promql-engine/storage/prometheus.(*matrixSelector).Series+0xb0 /go/pkg/mod/github.com/thanos-io/[email protected]/storage/prometheus/matrix_selector.go:130
# 0x1fe0b1c github.com/thanos-io/promql-engine/execution/exchange.(*concurrencyOperator).Series+0xbc /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/concurrent.go:54
# 0x1fe076d github.com/thanos-io/promql-engine/execution/exchange.(*coalesce).loadSeries.func1+0xcd /go/pkg/mod/github.com/thanos-io/[email protected]/execution/exchange/coalesce.go:204
1 @ 0x4719a1
Likely https://github.com/thanos-io/thanos/pull/7618 I guess! Can you try 0.36.1?
We "solved" this by changing thanos rule to use GRPC to address thanos query.
Rule flag: --query.config-file=/etc/thanos/query-config.yaml
Config file:
- grpc_config:
endpoint_addresses:
- thanos-query:10901