Configure Cluster Checks dynamically

[PSanetra]

It would be nice to have a ClusterCheck CRD, which could be deployed independently from the Cluster Agent deployment. Currently it is only possible to configure such a check via a ConfigMap and reference this ConfigMap in the DatadogAgent resource of the Cluster Agent.

This way the Cluster Agent could be deployed centrally without knowing about application specific cluster checks (such as checks on applications, running outside of the cluster).

[clamoriniere]

Hi @PSanetra,

Thanks for opening this issue. Indeed we are evaluating the possibility to configure cluster check with a dedicated CRD.

For now, what you describe is already possible thanks to Auto-discovery Annotations attached to a dedicated Kubernetes Service.

For example to target a service running outside a kubernetes cluster. you can create an empty K8s Service with the check configuration in a Datadog auto-discovery annotations.

apiVersion: v1
kind: Service
metadata:
  name: my-check-exposer
  namespace: my-app
  annotations:
    ad.datadoghq.com/endpoints.check_names: '["my-check"]'
    ad.datadoghq.com/endpoints.init_configs: '[{}]'
    ad.datadoghq.com/endpoints.instances: |
      [
        {
          "host": "https://my-check.com/metrics",
          "port": "6379"
        }
      ]
spec:
  clusterIP: "None"

It is less native than a CRD, but you can already create, update cluster check configurations without redeploying the cluster-agent.

Let me know if my answer has helped you to improve your monitoring configuration.

[PSanetra]

Nice this is exactly what we were looking for.

For some reason this worked only with the ad.datadoghq.com/service.instances annotation instead of ad.datadoghq.com/endpoints.instances. I am not sure what we missed in our cluster agent configuration.

I think this approach still has some drawback as it is sometimes necessary to put service credentials into the cluster check configuration. It would be nice if one could put these credentials into a Kubernetes secret.

[clamoriniere]

for endpoints you need to add the list of target IPs manually in the K8s Endpoints resource that correspond to the K8s Service. Let me know if it solved the your issue.