[Experimental Query] - Query Pushdown - Issue with multiple queriers connected?

[ahurtaud]

trafficstars

Thanos, Prometheus and Golang version used: Thanos v0.27.0

What happened: We enabled the experimental feature query-pushdown on all our thanos-query. We have a thanos-query on top, connected to multiple thanos-query leaf (3 kubernetes, exposed with announced label "stack") which connects to sidecars of each k8s.

When we query max by, and we filter using an anounced labelset. The query returns no data. If we request the same without filtering, we have the correct metrics.

What you expected to happen: Return the correct metrics set.

How to reproduce it (as minimally and precisely as possible):

• Architecture: query1 (with query-pushdown exp feature) ---> query2 (with query-pushdown and announced labelset stack="query2") --> a sidecar.
• query: max by (stack,version) (prometheus_build_info{stack="query2"}) and max by (stack,version) (prometheus_build_info)

[GiedriusS]

Thanks for trying it out and the repro! Should be easy to reproduce in e2e tests. Help wanted.

[ahurtaud]

Ok so we had a bit more look into it and the way to reproduce was a bit false. Actually it happens only if the announced label used is given by the sidecar and not set on the query as selector-label

edit: to reproduce:

• Architecture: query1 (with query-pushdown exp feature) ---> query2 (with query-pushdown with no selector-label) --> a sidecar connected to a prometheus with external labels (and so announced labelset by sidecar).

• query: max by (extlabel,version) (prometheus_build_info{extlabel="extvalue"}) and max by (extlabel,version) (prometheus_build_info)

We will continue to investigate to hopefully find the code issue and propsoe a fix. (newbie here, we will keep you up-to-date if we succeed or not. Help wanted anyway :) )

[ahurtaud]

Edit 2: Actually only a single query to sidecar with prometheus externalLabels is enough! executing a query using the externalLabel give no data.

• Architecture: query (with query-pushdown and no selector-label flags) --> a sidecar connected to a prometheus with external labels (and so announced labelset by sidecar).

[ahurtaud]

We would need help on this one, with @Gladorme, we tried but it seemsto be not as easy as it looks ... :/ @GiedriusS as you have implemented most of that feature, would you mind having a look at this one when you have time please? Thank you :)