charts [bitnami/keycloak] Keycloak Version 25.x.x and Metrics Exposure

Name and Version

bitnami/keycloak-22.x.x

What is the problem this feature will solve?

Starting with Keycloak version 25.x.x, a management interface has been introduced that exposes the /metrics endpoint. When the main Keycloak HTTP server is configured with tls.enabled, the HTTP server on the management interface also operates in TLS mode.

However, there appears to be a potential bug in Keycloak: while the management interface exposes /metrics over HTTPS, the realm-specific metrics endpoints (e.g., realms/{realm}/metrics) are still exposed via the main Keycloak HTTP server, which might run without TLS, i.e., on plain HTTP.

In the meantime, the ServiceMonitor configuration template only supports the default HTTP scheme, with no option to configure scrapers to collect metrics via an https-based endpoint. This limitation means that metrics from the management interface, which are served over HTTPS, cannot be gathered directly using the existing ServiceMonitor configuration.

What is the feature you are proposing to solve the problem?

servicemonitor.yaml

spec:
  {{- if .Values.metrics.serviceMonitor.jobLabel }}
  jobLabel: {{ .Values.metrics.serviceMonitor.jobLabel }}
  {{- end }}
  endpoints:
    {{- $defaultEndpoint := pick .Values.metrics.serviceMonitor "port" "interval" "scrapeTimeout" "relabelings" "metricRelabelings" "honorLabels" "scheme" }}
    {{- $defaultScheme := "http" }}
    {{- $endpoints := ternary (.Values.metrics.serviceMonitor.endpoints) (list (dict "path" .Values.metrics.serviceMonitor.path)) (empty .Values.metrics.serviceMonitor.path) }}
    {{- range $endpoints }}
    {{- $endpoint := merge . $defaultEndpoint }}
    - port: {{ $endpoint.port | quote }}
      scheme: {{ default $defaultScheme $endpoint.scheme | quote }}
      {{- if eq (default $defaultScheme $endpoint.scheme) "https" }}
      tlsConfig:
        insecureSkipVerify: {{ default true $endpoint.insecureSkipVerify }}
      {{- end }}
      path: {{ include "common.tplvalues.render" ( dict "value" $endpoint.path "context" $) }}
      {{- if $endpoint.interval }}
      interval: {{ $endpoint.interval }}
      {{- end }}
      {{- if $endpoint.scrapeTimeout }}
      scrapeTimeout: {{ $endpoint.scrapeTimeout }}
      {{- end }}
      {{- if $endpoint.relabelings }}
      relabelings: {{- include "common.tplvalues.render" ( dict "value" $endpoint.relabelings "context" $) | nindent 6 }}
      {{- end }}
      {{- if $endpoint.metricRelabelings }}
      metricRelabelings: {{- include "common.tplvalues.render" ( dict "value" $endpoint.metricRelabelings "context" $) | nindent 6 }}
      {{- end }}
      {{- if $endpoint.honorLabels }}
      honorLabels: {{ $endpoint.honorLabels }}
      {{- end }}
    {{- end }}
  namespaceSelector:
    matchNames:
      - {{ include "common.names.namespace" . | quote }}
  selector:
    matchLabels: {{- include "common.labels.matchLabels" ( dict "customLabels" .Values.commonLabels "context" $ ) | nindent 6 }}
      {{- if .Values.metrics.serviceMonitor.selector }}
      {{- include "common.tplvalues.render" (dict "value" .Values.metrics.serviceMonitor.selector "context" $) | nindent 6 }}
      {{- end }}
      app.kubernetes.io/component: metrics
{{- end }}

values.yaml

serviceMonitor:
     enabled: true
    endpoints:
      - path: '{{ include "keycloak.httpPath" . }}metrics'
        scheme: https
        insecureSkipVerify: true
        port: metrics
      - path: '{{ include "keycloak.httpPath" . }}realms/{{ .Values.adminRealm }}/metrics'
        scheme: http
        port: http
    interval: 30s
    scrapeTimeout: ""
    labels: {}
    selector: {}
    relabelings: []
    metricRelabelings: []
    honorLabels: false
    jobLabel: ""

Aug 22 '24 14:08 nix-power

The following template (custom) has been tested in our Keycloak K8S Kubernetes clusters and found working, all metrics https and http endpoints appear in Grafana.

servicemonitorcustom.yaml

{{- /*
  This is a temporary solution, as current servicemonitor.yaml template in Bitnami Helm chart
  does not support different schemes for multiple endpoints
  https://github.com/bitnami/charts/issues/28975
*/}}

{{- define "common.tplvalues.render" -}}
  {{- $value := typeIs "string" .value | ternary .value (.value | toYaml) }}
  {{- if contains "{{" (toJson .value) }}
    {{- if .scope }}
      {{- tpl (cat "{{- with $.RelativeScope -}}" $value "{{- end }}") (merge (dict "RelativeScope" .scope) .context) }}
    {{- else }}
      {{- tpl $value .context }}
    {{- end }}
  {{- else }}
    {{- $value }}
  {{- end }}
  {{- end -}}

{{- if .Values.idp.enabled }}
  {{- if .Values.idp.serviceMonitor.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ .Values.idp.serviceMonitor.name }}
  namespace: {{ .Values.idp.serviceMonitor.namespace }}
  {{- $labels := .Values.idp.serviceMonitor.labels }}
  labels:
    app.kubernetes.io/component: keycloak
spec:
  {{- if .Values.idp.serviceMonitor.jobLabel }}
  jobLabel: {{ .Values.serviceMonitor.jobLabel }}
  {{- end }}
  endpoints:
    {{- $defaultEndpoint := pick .Values.idp.serviceMonitor "port" "interval" "scrapeTimeout" "relabelings" "metricRelabelings" "honorLabels" "scheme" }}
    {{- $defaultScheme := "http" }}
    {{- $endpoints := ternary (.Values.idp.serviceMonitor.endpoints) (list (dict "path" .Values.idp.serviceMonitor.path)) (empty .Values.idp.serviceMonitor.path) }}
    {{- range $endpoints }}
    {{- $endpoint := merge . $defaultEndpoint }}
    - port: {{ $endpoint.port | quote }}
      scheme: {{ default $defaultScheme $endpoint.scheme | quote }}
      {{- if eq (default $defaultScheme $endpoint.scheme) "https" }}
      tlsConfig:
        insecureSkipVerify: {{ default true $endpoint.insecureSkipVerify }}
      {{- end }}
      path: {{ include "common.tplvalues.render" ( dict "value" $endpoint.path "context" $) }}
      {{- if $endpoint.interval }}
      interval: {{ $endpoint.interval }}
      {{- end }}
      {{- if $endpoint.scrapeTimeout }}
      scrapeTimeout: {{ $endpoint.scrapeTimeout }}
      {{- end }}
      {{- if $endpoint.relabelings }}
      relabelings: {{- include "common.tplvalues.render" ( dict "value" $endpoint.relabelings "context" $) | nindent 6 }}
      {{- end }}
      {{- if $endpoint.metricRelabelings }}
      metricRelabelings: {{- include "common.tplvalues.render" ( dict "value" $endpoint.metricRelabelings "context" $) | nindent 6 }}
      {{- end }}
      {{- if $endpoint.honorLabels }}
      honorLabels: {{ $endpoint.honorLabels }}
      {{- end }}
    {{- end }}
  namespaceSelector:
    matchNames:
      - {{ .Values.idp.serviceMonitor.namespace }}
  selector:
    matchLabels:
      {{- if .Values.idp.serviceMonitor.selector }}
      {{- include "common.tplvalues.render" (dict "value" .Values.idp.serviceMonitor.selector "context" $) | nindent 6 }}
      {{- end }}
      app.kubernetes.io/component: metrics
{{- end }}
{{- end }}

Corresponding section of values.yaml

idp:
  enabled: true
  serviceMonitor:
      enabled: true
      namespace: idp
      name: idp-keycloak
      selector:
        app.kubernetes.io/instance: idp
        app.kubernetes.io/name: keycloak
      endpoints:
        - path: '/management/metrics'
          scheme: https
          port: metrics
          insecureSkipVerify: true
          interval: 30s
        - path: '/realms/master/metrics'
          scheme: http
          interval: 30s
          port: http
        - path: '/realms/application/metrics'
          scheme: http
          interval: 30s
          port: http
keycloak:
  metrics:
      enabled: true
      service:
        annotations: null
        extraPorts:
          - name: metrics
            port: 9000
            protocol: TCP
            targetPort: metrics
      # Workaround for bug: https://github.com/bitnami/charts/issues/28975
      # handling serviceMonitor resurce in the local idp template [servicemonitorcustom.yaml] and values of idp.serviceMonitor
      #serviceMonitor:
      #  enabled: true
      #  interval: 15s
      #  port: http
      #  endpoints:
      #    - path: '{{ include "keycloak.httpPath" . }}realms/master/metrics'
      #    - path: '{{ include "keycloak.httpPath" . }}realms/application/metrics'

Aug 22 '24 20:08 nix-power

Hi,

Thank you for sharing your solution. So, if I understand correctly, we should add a tls section in the ServiceMonitor object. I will forward it to the engineering team but as it is not a critical feature we cannot guarantee an ETA. However, if you want to speed up the process, you can submit a PR and the team will check it.

Aug 23 '24 07:08 javsalgar

This is primarily about adding the scheme attribute to the ServiceMonitor object, as the tls configuration for the scraper application should be managed separately, not related to this chart

Aug 23 '24 20:08 nix-power

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

Sep 08 '24 01:09 github-actions[bot]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.

Sep 14 '24 01:09 github-actions[bot]

Can confirm that in latest 26.0.7 this issue has been resolved

Dec 22 '24 13:12 nix-power