Change Agent token storage path to always writable folder

[vboulineau]

What does this PR do?

Agent token is currently stored alongside dataog.yaml in /etc/datadog-agent. However /etc/datadog-agent is perhaps not suited for this:

• Content of /etc/datadog-agent is read by the Agent but not written to or modified.
• There is no guarantee /etc/datadog-agent to be writable (actually, the Agent should probably not have the permissions to write there).

However, we do have a run path, dedicated to file being written by the Agent: /opt/datadog-agent/run, which has to be writable (sys probe socket, log pointer, etc.). Storing the token there seems more suited as the token cannot be passed in and is always generated at runtime.

Motivation

Fix issues with read-only filesystem in Cluster Agent.

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

QA-ing this change means ensuring Agent commands requiring API access (like agent status) still work properly and cross-agent communication works correctly as well. Agent commands (like agent status) must also be tested in the Cluster Agent.

This has some E2E coverage as some Agent commands or API calls are made in some tests, but I let owners decide the parts that could warrant manual QA.

In containerized environments, this has been already validated to be working for Agent and Cluster Agent with Helm and Operator.

[pr-commenter[bot]]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=38161757 --os-family=ubuntu

Note: This applies to commit 3d4a9a30

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.

Project coverage is 51.65%. Comparing base (87f3610) to head (270f643). Report is 9 commits behind head on main.

:exclamation: Current head 270f643 differs from pull request most recent head 3d4a9a3

Please upload reports for the commit 3d4a9a3 to get more accurate results.

Files	Patch %	Lines
pkg/api/security/security.go	50.00%	1 Missing :warning:

Additional details and impacted files

@@             Coverage Diff             @@
##             main   #26541       +/-   ##
===========================================
+ Coverage   36.22%   51.65%   +15.42%     
===========================================
  Files        3358     1866     -1492     
  Lines      363690   161042   -202648     
===========================================
- Hits       131740    83182    -48558     
+ Misses     221766    73065   -148701     
+ Partials    10184     4795     -5389     

Flag	Coverage Δ
amzn_aarch64	51.85% <66.66%> (+14.87%)	:arrow_up:
centos_x86_64	51.74% <66.66%> (+14.85%)	:arrow_up:
ubuntu_aarch64	51.85% <66.66%> (+14.88%)	:arrow_up:
ubuntu_x86_64	51.85% <66.66%> (+14.88%)	:arrow_up:
windows_amd64	55.03% <100.00%> (+16.09%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[pr-commenter[bot]]

Regression Detector

Regression Detector Results

Run ID: 6c54cb01-ffb5-4508-92a3-b5faff152845 Metrics dashboard Target profiles

Baseline: 87f361073054a1a912c42f5ad16152c05b536f50 Comparison: 3d4a9a30d945d262814fd703601b518a8c02d54a

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	links
➖	tcp_syslog_to_blackhole	ingress throughput	+7.52	[-5.72, +20.75]	Logs
➖	pycheck_1000_100byte_tags	% cpu utilization	+2.03	[-2.91, +6.97]	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+1.23	[+0.34, +2.11]	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	Logs
➖	uds_dogstatsd_to_api	ingress throughput	-0.00	[-0.00, +0.00]	Logs
➖	idle	memory utilization	-0.42	[-0.46, -0.38]	Logs
➖	otel_to_otel_logs	ingress throughput	-0.92	[-1.72, -0.11]	Logs
➖	basic_py_check	% cpu utilization	-1.47	[-4.15, +1.22]	Logs
➖	file_tree	memory utilization	-2.13	[-2.25, -2.01]	Logs

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".