clickhouse-operator Feature request - Allow AnnotationsTobeSkipped to be configurable

We use internal tooling to update the ClickHouseInstallation when we would like to kick off a reconcile task. This internal tooling inserts a number of custom build annotations into the resource for internal tracking purposes which includes unique hashes and timestamps.

An unfortunate consequence of this is that a new prometheus series is created by the exporter for each reconcile that the operator initiates since these annotations are read by the exporter from the CHI and added as prometheus labels: https://github.com/Altinity/clickhouse-operator/blob/4bf622e07361f72b89568a121c0f7b8198252be0/pkg/metrics/metrics.go#L124

Digging deeper, looks like some annotations can be skipped but this is currently a hardcoded list: https://github.com/Altinity/clickhouse-operator/blob/4bf622e07361f72b89568a121c0f7b8198252be0/pkg/util/k8s.go#L30

We would appreciate the ability to configure custom annotations to ignore.

Oct 10 '24 22:10 nidhik

/assign

Oct 11 '24 07:10 chengjoey

hi @nidhik ,try modify etc-clickhouse-operator-files configmap, and add excludeFromPropagationAnnotations, like this:

apiVersion: v1
data:
  config.yaml: |
    excludeFromPropagationAnnotations: ["skipped-a", "skipped-b"]

this should help

Oct 12 '24 05:10 chengjoey

Hello! I work with @nidhik.

Unfortunately that does not seem to solve our problem as it only ignores propagating annotations to the Kubernetes resources. The annotations are still scraped as part of the ClickHouse operator metrics.

Although I add

################################################
##
## Annotations management section
##
################################################
annotation:
  # Applied when:
  #  1. Propagating annotations from the CHI's `metadata.annotations` to child objects' `metadata.annotations`,
  #  2. Propagating annotations from the CHI Template's `metadata.annotations` to CHI's `metadata.annotations`,
  # Include annotations from the following list:
  # Applied only when not empty. Empty list means "include all, no selection"
  include: []
  # Exclude annotations from the following list:
  exclude: [mux.com/styri-bazel-label,
          mux.com/styri-bazel-short-label,
          mux.com/styri-build-host,
          mux.com/styri-build-timestamp,
          mux.com/styri-build-user,
          mux.com/styri-buildkite,
          mux.com/styri-git-branch,
          mux.com/styri-git-commit,
          mux.com/styri-git-timestamp,
          mux.com/styri-update-timestamp,
          mux.com/styri-version
          ]

I also tried with excludeFromPropagationAnnotations. annotation.exclude seems equivalent.

Resulting kubernetes resources

kubectl describe chi realtime-secon-install
Name:         realtime-secon-install
Namespace:    aws-us-east-1-dos1
Labels:       app=realtime-secon
              mux.com/ch=realtime-secon
Annotations:  mux.com/styri-argo: false
              mux.com/styri-bazel-label: @//data/servers/ch-realtime-operator/secon:secon_styri.aws-us-east-1-dos1.update
              mux.com/styri-bazel-short-label: @//data/servers/ch-realtime-operator/secon:secon_styri
              mux.com/styri-build-host: stephens-mbp.lan
              mux.com/styri-build-timestamp: 2024-10-10 14:33:27 -0700
              mux.com/styri-build-user: scrowe
              mux.com/styri-buildkite: false
              mux.com/styri-git-branch: master
              mux.com/styri-git-commit: f6828cacdd8a2447a16d11199c3ffae7a47969ae
              mux.com/styri-git-timestamp: 2024-10-10 18:01:00 +0000
              mux.com/styri-update-timestamp: 2024-10-10 14:33:31 -0700
              mux.com/styri-version: 1728583260-local-f6828cacdd
API Version:  clickhouse.altinity.com/v1
Kind:         ClickHouseInstallation
...

k describe pod ch-realtime-secon-0-0-0
Name:             ch-realtime-secon-0-0-0
Namespace:        aws-us-east-1-dos1
Priority:         0
Service Account:  default
Node:             redacted
Start Time:       Tue, 15 Oct 2024 10:15:18 -0700
Labels:           app=realtime-secon
                  clickhouse.altinity.com/app=chop
                  clickhouse.altinity.com/chi=realtime-secon-install
                  clickhouse.altinity.com/cluster=realtime-secon
                  clickhouse.altinity.com/namespace=aws-us-east-1-dos1
                  clickhouse.altinity.com/ready=yes
                  clickhouse.altinity.com/replica=0
                  clickhouse.altinity.com/shard=0
                  controller-revision-hash=ch-realtime-secon-0-0-594b868f5b
                  mux.com/ch=realtime-secon
                  statefulset.kubernetes.io/pod-name=ch-realtime-secon-0-0-0
Annotations:      mux.com/styri-argo: false
Status:           Running
...

Snippet of scraped clickhouse operator metrics

# HELP clickhouse_operator_chi_reconciles_completed number of CHI reconciles completed successfully
# TYPE clickhouse_operator_chi_reconciles_completed counter
clickhouse_operator_chi_reconciles_completed{app="realtime-secon",chi="realtime-secon-install",mux_com_ch="realtime-secon",mux_com_styri_argo="false",mux_com_styri_bazel_label="@//data/servers/ch-realtime-operator/secon:secon_styri.aws-us-east-1-dos1.update",mux_com_styri_bazel_short_label="@//data/servers/ch-realtime-operator/secon:secon_styri",mux_com_styri_build_host="stephens-mbp.lan",mux_com_styri_build_timestamp="2024-10-10 14:33:27 -0700",mux_com_styri_build_user="scrowe",mux_com_styri_buildkite="false",mux_com_styri_git_branch="master",mux_com_styri_git_commit="f6828cacdd8a2447a16d11199c3ffae7a47969ae",mux_com_styri_git_timestamp="2024-10-10 18:01:00 +0000",mux_com_styri_update_timestamp="2024-10-10 14:33:31 -0700",mux_com_styri_version="1728583260-local-f6828cacdd",namespace="aws-us-east-1-dos1"} 1
# HELP clickhouse_operator_chi_reconciles_started number of CHI reconciles started
# TYPE clickhouse_operator_chi_reconciles_started counter
clickhouse_operator_chi_reconciles_started{app="realtime-secon",chi="realtime-secon-install",mux_com_ch="realtime-secon",mux_com_styri_argo="false",mux_com_styri_bazel_label="@//data/servers/ch-realtime-operator/secon:secon_styri.aws-us-east-1-dos1.update",mux_com_styri_bazel_short_label="@//data/servers/ch-realtime-operator/secon:secon_styri",mux_com_styri_build_host="stephens-mbp.lan",mux_com_styri_build_timestamp="2024-10-10 14:33:27 -0700",mux_com_styri_build_user="scrowe",mux_com_styri_buildkite="false",mux_com_styri_git_branch="master",mux_com_styri_git_commit="f6828cacdd8a2447a16d11199c3ffae7a47969ae",mux_com_styri_git_timestamp="2024-10-10 18:01:00 +0000",mux_com_styri_update_timestamp="2024-10-10 14:33:31 -0700",mux_com_styri_version="1728583260-local-f6828cacdd",namespace="aws-us-east-1-dos1"} 1

Oct 15 '24 18:10 stephenmcrowe

@nidhik , fixed in 0.24.1 (not released yet). This is a part of operator configuration now:

spec:
  metrics:
    labels:
      exclude:
      - exclude_this_label
      - exclude_this_annotation
` ``

Nov 04 '24 14:11 alex-zaitsev

Released https://github.com/Altinity/clickhouse-operator/releases/tag/release-0.24.1

Dec 02 '24 16:12 alex-zaitsev