Issues with metricrelabeling for podmonitor using TA

[laxmansoma22]

Component(s)

No response

What happened?

Description

I am trying to keep only certain metrics for kube-state-metrics and drop everything , we started using otel collector target allocator recently instead of kube prometheus stack as its working with same template...

Steps to Reproduce

Expected Result

we need only this metrics and labels

metricRelabelings:

• action: labeldrop regex: '(endpoint|instance|location|prometheus|prometheus_replica|service|uid)'
• sourceLabels: [name] action: keep regex: '(kube_pod_container_resource_requests|kube_pod_container_status_restarts_total|kube_pod_status_phase|kube_pod_info)'

Actual Result

instead , its not able to scrape them...

Kubernetes Version

1.31.3

Operator version

OS: (e.g., "Ubuntu 20.04")

Collector version

ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.120.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") Compiler(if manually compiled): (e.g., "go 14.2")

Log output

Additional context

No response

[swiatekm]

Could you post your OpenTelemetryCollector CR? From your description, what you're trying to do should work, but it's hard to say without more details.

[laxmansoma22]

apiVersion: opentelemetry.io/v1beta1 kind: OpenTelemetryCollector metadata: name: otelc-ta namespace: opentelemetry-operator spec: env:

name: GOOGLE_APPLICATION_LOGIN valueFrom: secretKeyRef: name: rke-otel-sa key: otel-sa.json name: GOOGLE_APPLICATION_CREDENTIALS value: /otel/otel-sa.json name: OTEL_RESOURCE_ATTRIBUTES value: "cluster={{ .Values.clusterName }},namespace=opentelemetry-operator,location={{ .Values.region }},project_id= {{ .Values.projectName }}" volumeMounts: name: rke-otel-sa mountPath: /otel readOnly: true volumes: name: rke-otel-sa secret: secretName: rke-otel-sa defaultMode: 0777 mode: statefulset serviceAccount: opentelemetry-operator targetAllocator: enabled: true serviceAccount: opentelemetry-operator prometheusCR: enabled: true podMonitorSelector: {} serviceMonitorSelector: {} resources: requests: memory: "200Mi" cpu: "100m" config: receivers: prometheus: config: scrape_configs:

• job_name: 'kubernetes-nodes-cadvisor' scheme: https tls_config: ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt insecure_skip_verify: true bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token kubernetes_sd_configs:
• role: node relabel_configs:
• target_label: address replacement: kubernetes.default.svc:443
• source_labels: [_meta_kubernetes_node_name] regex: (.+) target_label: metrics_path replacement: /api/v1/nodes/$$1/proxy/metrics/cadvisor

metric_relabel_configs:

# - action: labeldrop

# regex: '(image|id|endpoint|instance|job|location|prometheus|prometheus_replica|service|container|name)'

- sourceLabels: ["name"]

action: keep

regex: "container(memory_working_set_bytes|container_cpu_usage_total)"

replacement: "$$1"

separator: ";"

kubeletstats: collection_interval: 10s auth_type: 'serviceAccount' endpoint: '${env:K8S_NODE_NAME}:10250' insecure_skip_verify: true metric_groups:

• node
• pod
• container extra_metadata_labels:
• container.id processors: filter/drop_metrics_by_name: metrics: exclude: match_type: strict metric_names:
• container.memory.rss

filter/keep_metrics_by_name:

metrics:

exclude:

match_type: strict

metric_names:

- istio_requests_total

- istio_request_duration_milliseconds_bucket

resourcedetection: detectors: [env] batch: send_batch_size: 1000 timeout: 5s memory_limiter: check_interval: 5s limit_percentage: 80 spike_limit_percentage: 25 k8sattributes: extract: metadata: - k8s.node.name - k8s.namespace.name - k8s.pod.name passthrough: false pod_association: - sources: - from: resource_attribute name: k8s.pod.ip - sources: - from: resource_attribute name: k8s.pod.uid - sources: - from: connection exporters: debug: verbosity: detailed googlemanagedprometheus: metric: compression: gzip add_metric_suffixes: false extra_metrics_config: enable_target_info: false project: {{ .Values.projectName }} service: pipelines: metrics: receivers: [prometheus] processors: [memory_limiter, batch, resourcedetection, filter/drop_metrics_by_name,k8sattributes] exporters: [googlemanagedprometheus] laxmansoma22 laxmansoma22 commented 5 days ago laxmansoma22 5 days ago Author apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: envoy-stats-monitor namespace: opentelemetry-operator labels: monitoring: prometheus release: opentelemetry-operator spec: selector: matchExpressions:

• {key: istio-prometheus-ignore, operator: DoesNotExist} namespaceSelector: any: true jobLabel: envoy-stats podMetricsEndpoints:

path: /stats/prometheus interval: 30s relabelings: action: keep sourceLabels: [app] action: keep sourceLabels: [__meta_kubernetes_pod_container_name] regex: "istio-proxy" action: keep sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape] sourceLabels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetLabel: address action: labeldrop regex: "_meta_kubernetes_pod_label(.+)" sourceLabels: [__meta_kubernetes_namespace] action: replace targetLabel: namespace sourceLabels: [__meta_kubernetes_pod_name] action: replace targetLabel: pod_name metricRelabelings: sourceLabels: ["name"] action: keep regex: '(istio_requests_total|istio_request_duration_milliseconds_bucket)' ArthurSens ArthurSens commented 4 days ago ArthurSens 4 days ago Member I believe you'll receive better support by reaching out to the target allocator maintainers :)

The repository is https://github.com/open-telemetry/opentelemetry-operator

ArthurSens added question Further information is requested and removed waiting for author

receiver/prometheus Prometheus receiver 4 days ago

github-actions mentioned this 3 days ago Weekly Report: 2025-05-13 - 2025-05-20 #40138 laxmansoma22 laxmansoma22 commented 2 days ago laxmansoma22 2 days ago Author thanks , will post it there

laxmansoma22 Add a comment new Comment Markdown input: edit mode selected. Write Preview Use Markdown to format your comment Remember, contributions to this repository should follow its contributing guidelines, security policy and code of conduct. Metadata Assignees No one assigned Labels bug Something isn't working question Further information is requested Type No type Projects No projects Milestone No milestone Relationships None yet Development No branches or pull requests NotificationsCustomize You're receiving notifications because you're subscribed to this thread. Participants

[laxmansoma22]

I am trying to keep only certain metrics , but its allowing me to do that as part of the podmonitor....

[laxmansoma22]

apiVersion: monitoring.coreos.com/v1 kind: PodMonitor metadata: name: envoy-stats-monitor namespace: opentelemetry-operator labels: monitoring: prometheus release: opentelemetry-operator spec: selector: matchExpressions:

• {key: istio-prometheus-ignore, operator: DoesNotExist} namespaceSelector: any: true jobLabel: envoy-stats podMetricsEndpoints:

path: /stats/prometheus interval: 30s relabelings: action: keep sourceLabels: [app] action: keep sourceLabels: [__meta_kubernetes_pod_container_name] regex: "istio-proxy" action: keep sourceLabels: [__meta_kubernetes_pod_annotationpresent_prometheus_io_scrape] sourceLabels: [address, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 targetLabel: address action: labeldrop regex: "_meta_kubernetes_pod_label(.+)" sourceLabels: [__meta_kubernetes_namespace] action: replace targetLabel: namespace sourceLabels: [__meta_kubernetes_pod_name] action: replace targetLabel: pod_name metricRelabelings: sourceLabels: ["name"] action: keep regex: '(istio_requests_total|istio_request_duration_milliseconds_bucket)'

[swiatekm]

Could you please reformat these into a more readable form? Using code blocks tagged yaml should help.

```yaml
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
```