Support headless services in kubernetes

[akram]

Hi team, and great work on parca

I didn't find anything about: Supporting headless services in kubernetes

Would this be possible with Parca?

here is what a headless service looks like:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/component: metrics
    app.kubernetes.io/name: argocd-metrics
    app.kubernetes.io/part-of: argocd
  name: my-headless-service
spec:
  clusterIP: None
  ports:
  - name: metrics
    port: 8082
    protocol: TCP
    targetPort: 8082
  selector:
    app.kubernetes.io/name: argocd-application-controller
  sessionAffinity: None
  type: ClusterIP

When making a DNS resolution to a headless service, it will return the A records for all the pods matching the headless service selector. for example:

root@my-pod:/data# getent hosts my-headless-service
10.131.0.245    my-headless-service.argocd.svc.cluster.local
10.131.1.12     my-headless-service.argocd.svc.cluster.local

That could be nice that when parca receives

      static_configs:
      - targets: [ 'my-headless-service:8082'  ]

It automatically resolve that into multiple addresses and add the relevant targets.

edit: strangely I have done the test within the parca pod and it seems that DNS resolution for headless services is not working properly. So, maybe the base image of parca has an issue with that.

[brancz]

Scrape configs are identical to those of Prometheus, so yes, you can do this with kubernetes_sd_configs: https://prometheus.io/docs/prometheus/latest/configuration/configuration/

[akram]

In my case, the metrics url also exposes the /debug/pprof urls which we are interested in. Would that work with kubernetes_sd_configs ?

do you have a configuration example for parca ?

[brancz]

Here's the config we use in production:

scrape_configs:
  - job_name: pods
    scrape_interval: "3s"
    profiling_config:
      pprof_config:
        memory:
          keep_sample_type:
          - type: inuse_space
            unit: bytes
        process_cpu:
          enabled: false
    kubernetes_sd_configs:
      - role: pod
        selectors:
          - role: pod
            label: parca-scraping=true
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_pod_label_app_kubernetes_io_(.+)
        replacement: "app_kubernetes_io_$1"
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: replace
        target_label: container

This ignores pprof's CPU profiling since we already capture profiling data at lower overhead and higher resolution with Parca Agent as well as only keeps heap profiling data from memory profiles (everything else tends to be too expensive to ingest). For the keep_sample_type configuration you will need to run a currently not yet released version like the container image ghcr.io/parca-dev/parca:main-1691672302-414b9271.

[akram]

oh ok. I forgot to mention that I am not using the agent using because of lack of permissions on my cluster. In similar cases as mine, do you think that it could make sense to have a support for headless services? Or do you think that it is too much of an edge case ?

I would be interested in working on a PR with some guidance if the use case seems useful.

[brancz]

By using role: endpoints instead of role: pods you can already discover targets via endpoints objects (which is what headless services create under the hood). Let me know if you need help putting together a config! Also always feel free to drop by on our Discord.