tidb-operator

tidb-operator copied to clipboard

Feature Request

Is your feature request related to a problem? Please describe: I have a prometheus operator installation with thanos and I would rather reuse this than run a separate prometheus and grafana deployment for each cluster.

Describe the feature you'd like: Service monitor and json dashboards for doing this.

Describe alternatives you've considered: Using the current tidb monitor is a possibility but in my opinion it is wastes resources in this scenario.

@sstubbs Do you mean that you would like a central Prometheus and Grafana to collect the metrics for all of the clusters and do not want to install any TidbMonitor object?

We have supported one prometheus monitor multiple tidbclusters in #3155, and they have been merged into the master branch. You can use thanos example to integrate into your prometheus operator. https://github.com/pingcap/tidb-operator/tree/master/examples/monitor-with-thanos

Great thank you!

@mikechengwei Thanks for the response! #3155 is mainly for the heterogeneous cluster which is actually the same cluster with the original cluster, if we want to collect metrics for different normal TidbClusters (especially with TLS enabled) in different namespaces with one TiDBMonitor, this would require a different solution.

@sstubbs Could you please describe your requirement in more detail?

I see. I just find it easier to manage one prometheus than a few and it's nice to be able to persist data to thanos when needed and monitor everything in the cluster from a single grafana dashboard. If this issue hasn't come up before perhaps there aren't many people with this use case.

ok, @mikechengwei may work on a solution that one Prometheus for one TidbCluster and aggregate all of the metrics data with Thanos and one single Grafana for Dashboard. There is a draft PR already https://github.com/pingcap/docs-tidb-operator/pull/640 but Mike may update it later.

I see thanks a lot. Not a completely needed feature from my side but it would definitely be nice to have. A few of the other charts I'm using such as itio are also going this way of providing service monitors and dashboards rather than bundling them. They have deprecated the bundled prometheus as of 1.7.

I understand for a lot of users though that quickly being able to deploy bundled tidb monitors are preferred.

OK. We evaluate this.

Now, we provide prometheus operator example https://github.com/pingcap/tidb-operator/tree/master/examples/prometheus-operator and thanos example https://github.com/pingcap/tidb-operator/tree/master/examples/monitor-with-thanos. But it is not elegant.

I think we can do

1. optimize the configuration of thanos in tidbmonitor,add thanos spec like in prometheus operator.
2. Based on thanos requirement, provide an elegant promethes operator example.

@mikechengwei Could you please help improve this example to support TiFlash, TiCDC, and Pump? Thanks!

@mikechengwei Could you please help improve this example to support TiFlash, TiCDC, and Pump? Thanks!

ok, one component one tidbmonitor, then thanos aggreate all tidbmonitor, right?

I do not recommend user-defined servicemonitor.