karpenter-provider-aws feat: allow setting service annotations via helm chart

Fixes #N/A

Description Allows setting annotations on the karpenter service. We need this so that we can add annotations to setup datadog to monitor karpenter. There are probably other use cases.

How was this change tested?

Generated a helm chart locally with the additional values:

service:
  annotations:
    ad.datadoghq.com/service.checks: |
      {
        "karpenter": {
          "cluster_check": true,
          "init_config": {},
          "instances": [
            {
              "openmetrics_endpoint": "http://%%host%%:8000/metrics",
              "exclude_metrics": [
                "^karpenter\\.go\\..*$"
              ]
            }
          ]
        }
      }

Does this change impact docs?

[ ] Yes, PR includes docs updates
[ ] Yes, issue opened: #
[x] No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Aug 28 '24 21:08 jamesduffy

Deploy Preview for karpenter-docs-prod canceled.

Name	Link
Latest commit	e8752b6999334f3ff864d6e11bf3c7ba01e81186
Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/6716c0bc3445cf000831eac8

Aug 28 '24 21:08 netlify[bot]

What's preventing you from using the existing additionalAnnotations field? Unless absolutely necessary, we'd like to avoid additional configuration in the helm chart.

Sep 05 '24 17:09 jmdeal

What's preventing you from using the existing additionalAnnotations field? Unless absolutely necessary, we'd like to avoid additional configuration in the helm chart.

additionalAnnotations adds the annotations to a lot of different things. This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected. Therefore, we would need a way to add the annotation in a single place. Datadog annotations are often on the service.

Sep 13 '24 17:09 jamesduffy

This PR has been inactive for 14 days. StaleBot will close this stale PR after 14 more days of inactivity.

Sep 28 '24 12:09 github-actions[bot]

@jmdeal I would like to move forward with this if my explanation makes sense.

Oct 02 '24 17:10 jamesduffy

Sorry this has slipped, could you elaborate on what the consequences of this are?

This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected.

Unless this is strictly blocking, and no other workarounds exist, we don't want to expand the configuration surface for the helm chart we ship.

Oct 14 '24 22:10 jmdeal

Sorry this has slipped, could you elaborate on what the consequences of this are?

This would be picked up from the Datadog agent as separate checks and instead of running just one check for Karpenter it would run duplicate checks for each place additionalAnnotations is injected.

Unless this is strictly blocking, and no other workarounds exist, we don't want to expand the configuration surface for the helm chart we ship.

The datadog agent watches for annotations to configure it's checks/metric collection. Using the existing additionalAnnotations would create annotations on more than one resource. This would configure the datadog agent to run multiple collections of the same metrics making it unusable.

If anyone wants to use datadog to collect metrics using datadog's autodiscovery they will also have this problem.

Oct 14 '24 23:10 jamesduffy

Got it, I assume for most resources this would be a no-op, but you can get duplicate metrics since it would scrape both the Karpenter pods and the service?

Oct 15 '24 00:10 jmdeal

Correct. In summary, using additionalAnnotations would duplicate metrics sent to Datadog. Adding service.annotations on the service would allow us to create annotations to create only one Datadog metric collection.

Oct 15 '24 08:10 jamesduffy