cortexproject
Support Ruler to query Query Frontend
What this PR does:
Add ruler.frontend-address to allow query to query frontends instead of ingesters. If -ruler.frontend-address is set, rulers query to query frontends for evaluating rules. It can support query frontend features like vertical query sharding.
Which issue(s) this PR fixes: Fixes # 5105
Checklist
- [x] Tests updated
- [x] Documentation added
- [x]
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]
@yeya24 I have reflected the review you left. Thank you for the review!
We are in the process of releasing 1.18 Please rebase PR and change the changelog to master
@yeya24 I think we need to quey to QFE via gRPC. I'm thinking the step as follows:
- Define query request, response (including floating histogram) proto in cortex.pb
- Ruler has grpc client and query to QFE via the client.
Make sense. I am thinking about the same thing. Query frontend already supports grpc as long as you wrap your request with httpgrpc. We don't need to define a new rpc in cortex.pb
For the protobuf change, we don't need to do it in this PR.
@SungJin1212 This is how query frontend converts a http request to httpgrpc request https://github.com/cortexproject/cortex/blob/master/pkg/frontend/transport/roundtripper.go#L39.
We can do something similar. But from Ruler to Query Frontend.
@yeya24 what do you think how to fetch the float histogram? For my understanding, querier have to send response (including floating histogram) to QFE.
We have another ongoing PR which should deal with this.
Let's skip it here and no need to worry about this in this pr
That requires more changes. As you said the Querier needs to return float histograms in protobuf format first. Actually We already have another PR that is trying to do this so let's not duplicate the work here.
I think I am good once we switch to gRPC to query QFE
@yeya24 Ah.. ok I got it. What I have to do is convert http to gRPC. Thank you.
@SungJin1212 As @friedrichg said, let's mark this feature as experimental in cortexmetrics.io/docs/configuration/v1guarantees#experimental-features.
Architecture diagram needs update cortexmetrics.io/docs/architecture
I think we can skip this in this pr. We need to update it anyway.
Why is it better? I can see frontend caching is one is reason. You could add some a guide page similar to cortexmetrics.io/docs/guides/ruler-sharding
We will add a doc but will create another issue to track that. This item doesn't need to be owned by @SungJin1212
@yeya24 @friedrichg I have added experimental flags and got the rebase. I think caching and using vertical sharding for evaluating rules is a main contribution of this pr.