[CECO-1109] Allow label overrides for ds for DDA updates

[khewonc]

What does this PR do?

Allows node label overrides for daemonsets when updating DDAs. After creating a DDA, subsequent changes to the node agent label overrides weren't being propagated to the agent pods.

Motivation

CECO-1109

Additional Notes

Fixes https://github.com/DataDog/datadog-operator/issues/1156

Minimum Agent Versions

Are there minimum versions of the Datadog Agent and/or Cluster Agent required?

• Agent: n/a
• Cluster Agent: n/a

Describe your test plan

1. Testing label overrides with 1.7.0+:

• Create a DDA with or without node agent label overrides
• Add a label to modify the value of one of the label selectors on the daemonset

  override:
    nodeAgent:
      labels:
        "agent.datadoghq.com/name": "bar"

• Check that the label on the agent pod remains the same as the label selector (agent.datadoghq.com/name=datadog instead of agent.datadoghq.com/name=bar) and that this log is in the operator logs:

{"level":"INFO","ts":"2024-05-08T18:00:52Z","logger":"controllers.DatadogAgent","msg":"Selector value does not match template labels, modifying template labels","datadogagent":"default/datadog","component":"nodeAgent","daemonset.Namespace":"default","daemonset.Name":"datadog-agent","selector label":"agent.datadoghq.com/name: datadog","template label":"agent.datadoghq.com/name: bar"}

• Add (or change) the node agent label override

  override:
    nodeAgent:
      labels:
        "foo": "bar"

• Check that the new label was added to the agent pod
• Remove the override. The label should not be present in the new agent pod

2. Testing daemonset restarts from previous operator version:

• Create a DDA with operator <1.7.0 with a node label override

  override:
    nodeAgent:
      labels:
        "foo": "bar"

• The daemonset label should be the same as the pod template label. Also note the agent.datadoghq.com/agentspechash annotation on the daemonset to use for comparison in later steps
• Change the label override

  override:
    nodeAgent:
      labels:
        "foo": "bar123"

• The agent.datadoghq.com/agentspechash annotation value should change, the daemonset label should change to foo:bar123, but the pod template spec should still have the old foo:bar label and the node agent pod shouldn't have restarted
• Update the operator to 1.7.0+
• The node agent pod should have restarted. The agent.datadoghq.com/agentspechash annotation value should be the same and both the daemonset label and pod template spec label should say foo:bar123

Checklist

• [x] PR has at least one valid label: bug, enhancement, refactoring, documentation, tooling, and/or dependencies
• [x] PR has a milestone or the qa/skip-qa label

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 39.13043% with 28 lines in your changes are missing coverage. Please review.

Project coverage is 59.53%. Comparing base (6b0c7bd) to head (336944b). Report is 3 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1173      +/-   ##
==========================================
- Coverage   59.55%   59.53%   -0.02%     
==========================================
  Files         174      174              
  Lines       21559    21592      +33     
==========================================
+ Hits        12839    12855      +16     
- Misses       7941     7958      +17     
  Partials      779      779              

Flag	Coverage Δ
unittests	59.53% <39.13%> (-0.02%)	:arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
...ers/datadogagent/controller_reconcile_v2_common.go	28.36% <39.13%> (+2.74%)	:arrow_up:

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6b0c7bd...336944b. Read the comment docs.