Poll agent integration configuration files for changes

[lattwood]

What does this PR do?

This removes the restart requirement for the agent to pick up new integration configurations. The agent will poll on the same interval as other autodiscovery checks.

Motivation

• It closes #11160.
• It allows generating integration configuration from a sidecar and doesn't require restarting the DataDog Agent to pick up the new changes.

Possible Drawbacks / Trade-offs

• The Agent will have more read IO operations due to polling the filesystem for changes. This impact is slightly mitigated by the 5 minute TTL on the configuration cache.

Describe how to test/QA your changes

• Change integration configuration, wait 5 minutes for the cache entry to expire and get picked up.
• Beyond that, would appreciate some guidance on unit tests for this, unless DataDog employees are of the opinion this doesn't warrant it.

Reviewer's Checklist

• [ ] If known, an appropriate milestone has been selected; otherwise the Triage milestone is set.
• [ ] Use the major_change label if your change either has a major impact on the code base, is impacting multiple teams or is changing important well-established internals of the Agent. This label will be use during QA to make sure each team pay extra attention to the changed behavior. For any customer facing change use a releasenote.
• [ ] A release note has been added or the changelog/no-changelog label has been applied.
• [ ] Changed code has automated tests for its functionality.
• [ ] Adequate QA/testing plan information is provided if the qa/skip-qa label is not applied.
• [ ] At least one team/.. label has been applied, indicating the team(s) that should QA this change.
• [ ] If applicable, docs team has been notified or an issue has been opened on the documentation repo.
• [ ] If applicable, the need-change/operator and need-change/helm labels have been applied.
• [ ] If applicable, the config template has been updated.

[djmitche]

Thanks for this! This has the potential to cause issues with components that have quietly assumed on-disk configs don't change dynamically. An example that comes to mind because I've been staring at it lately is that a number of logs-agent launchers do not handle sources being removed. It might be worth looking for similar ramifications.

This may also change behavior that other users are implicitly relying on -- being able to "stage" config changes and only make them active on agent restart. The 5-minute poll, in particular, may be surprising here: imagine making a change to a config file, double-checking that behavior didn't change, grabbing a cup of coffee, and returning to find that the change had deployed without you. So, perhaps this should be behind a default-off configuration option? Alternately, perhaps the reload should be triggered not by a timer but by an agent CLI command (agent reload-integrations or something)?

I'll mark this as major-change, since it might have wide-reaching impact. And, despite what CODEOWNERS decided this is probably a question for the @DataDog/container-integrations team.

[lattwood]

@djmitche good points, I will ponder on potential solutions around it.

[lattwood]

Also, not sure containers is the right team? The intent is to autoconfigure the agent for AWS managed resource/services

[djmitche]

That team owns autodiscovery :)

[lattwood]

@djmitche suggested changes made

[vboulineau]

Hello @lattwood,

I've tested the PR and with some minor changes it works fine for checks, and as noted by Dustin, deleting some logs configs may not work (will reword the changelog).

I see valid use cases for this feature so I plan to merge for 7.41. If you're happy with that, you can close this PR in favor of https://github.com/DataDog/datadog-agent/pull/13777

[lattwood]

Sure, will go ahead and close in support of #13777.

Was kinda hoping to land another change in the agent though! 😂 (https://github.com/DataDog/datadog-agent/commits?author=lattwood)