Disable consumer name in Prometheus metrics (because it blows up with high cardinality

[frankkoornstra]

Description

I love that Protmetheus has such good support in Apisix, it really contributes to the observability. The only thing I wish for is to be able to disable the consumer label. Prometheus isn't great with high cardinality labels and consumer is one of those. We're not really interested in the consumer anyway - we log the requests another way - but we are interested in the metrics without that level of detail. A simple disable_consumer_name: true in the settings would do the trick :pray:

[tao12345666333]

Do you want to disable it globally, or a prometheus plugin on a certain route?

[frankkoornstra]

For my use case, on the plugin on the route. Globally would be nice but I'd rather have more granular options

[acuteaura]

you should be able to do this inside prometheus with

writeRelabelConfigs:
- action: labeldrop
  regex: consumer

in Prometheus Operator, most CRDs should also have a relabelings key that takes the same format.

In general, I'd agree that cardinality is too high for this metric, but some use cases only involve authenticating a small set of well-known third parties, and then it's reasonable again.

[frankkoornstra]

Thanks for the tip. I'm deploying Apisix through the official Helm chart however and I don't think it has any facilities for this. Only thing I can find is this:

# -- Observability configuration.
metrics:
  serviceMonitor:
    # -- Enable or disable Apache APISIX serviceMonitor
    enabled: false
    # -- namespace where the serviceMonitor is deployed, by default, it is the same as the namespace of the apisix
    namespace: ""
    # -- name of the serviceMonitor, by default, it is the same as the apisix fullname
    name: ""
    # -- interval at which metrics should be scraped
    interval: 15s
    # -- @param serviceMonitor.labels ServiceMonitor extra labels
    labels: {}
    # -- @param serviceMonitor.annotations ServiceMonitor annotations
    annotations: {}

I guess I'll need to create my own ServiceMonitor instead and do some relabeling as suggested and not rely on the ServiceMonitor of the official Helm chart.

[acuteaura]

It looks like the Helm chart does support this:

https://github.com/apache/apisix-helm-chart/blob/5d6dfbe512583816a8380619174e1df844f6965e/charts/apisix-ingress-controller/templates/servicemonitor.yaml#L38

https://github.com/apache/apisix-helm-chart/blob/5d6dfbe512583816a8380619174e1df844f6965e/charts/apisix/templates/service-monitor.yaml#L44

[frankkoornstra]

Thanks for that. I'll wait for the next release to include it since Helm chart 2.1.0 does not contain that just yet but good to know that it's coming :+1: