What does this PR do?

This integrates the process and core agent with the process-agent component created in https://github.com/DataDog/datadog-agent/pull/22450

Motivation

This allows the process checks to be run in either the process agent or the core agent in a linux environment.

PROCS-3723 and PROCS-3724

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

The process agent and core agent will need to be tested in the following configurations

Tests (Linux Only)

Test Case	Configuration	Core Agent	Process Agent
1	run_in_core_agent :white_check_mark: process, container, or process-discovery collection :white_check_mark:	:white_check_mark:	:x:
2	run_in_core_agent :x: process collection :white_check_mark:	:x:	:white_check_mark:
3	run_in_core_agent :white_check_mark: process collection :white_check_mark: NPM :white_check_mark:	:x:	:white_check_mark:
4	run_in_core_agent :x: process collection :white_check_mark: langauge detection: :white_check_mark:	:x:	:white_check_mark:

Test Case 1 - Running the process checks in the core agent enabled

Setup Config

datadog.yaml

process_config:
  process_collection:
    enabled: true
  run_in_core_agent:
    enabled: true

system-probe.yaml

network_config:
    enabled: false

Start datadog agent
sudo service datadog-agent start
Verify the process agent does not start and exits by checking logs under /var/log/datadog/process-agent.log

2024-02-14 10:45:23 PST | PROCESS | INFO | (comp/process/agent/agentimpl/agent_linux.go:44 in agentEnabled) | The process checks will run in the core agent
2024-02-14 10:45:23 PST | PROCESS | INFO | (command/main_common.go:193 in runApp) | process-agent is not enabled, exiting...

Verify the process agent component starts in the core agent under /var/log/datadog/agent.log

2024-02-14 10:33:29 PST | CORE | INFO | (pkg/process/runner/runner.go:276 in Run) | Starting process-agent with enabled checks=[process rtprocess]
...
2024-02-14 10:33:29 PST | CORE | INFO | (pkg/process/runner/runner.go:233 in logCheckDuration) | Finished process check #1 in 9.37683ms

Verify process and container data are visible in Live Processes
Verify only the agent process is running and process-agent is not running in Live Processes

Test Case 2 - Running the process checks in the core agent disabled

Setup Config

datadog.yaml

process_config:
  process_collection:
    enabled: true
  run_in_core_agent:
    enabled: false

system-probe.yaml

network_config:
    enabled: false

Start datadog agent
sudo service datadog-agent start
Verify the process-agent starts logs under /var/log/datadog/process-agent.log

2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:276 in Run) | Starting process-agent with enabled checks=[process rtprocess]
2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:233 in logCheckDuration) | Finished process check #1 in 9.249009ms

Verify the process agent component does not start in the core agent under /var/log/datadog/agent.log. There should be no process checks running or starting.
Verify process and container data are visible in Live Processes
Verify the process-agent is running in Live Processes

Test Case 3 - Running the process checks in the core agent enabled, but NPM is also enabled

This test that having NPM enabled blocks the process checks from running in the core agent and the checks will continue in the process-agent.

Setup Config

datadog.yaml

process_config:
  process_collection:
    enabled: true
  run_in_core_agent:
    enabled: true

system-probe.yaml

network_config:
    enabled: true

Start datadog agent
sudo service datadog-agent start
Verify the process-agent starts logs under /var/log/datadog/process-agent.log

2024-02-14 13:44:38 PST | PROCESS | WARN | (comp/process/agent/agentimpl/agent_linux.go:37 in agentEnabled) | Network Performance Monitoring is not supported in the core agent. The process-agent will be enabled as a standalone agent
....
2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:276 in Run) | Starting process-agent with enabled checks=[process rtprocess]
2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:233 in logCheckDuration) | Finished process check #1 in 9.249009ms

Verify the process agent component does not start in the core agent under /var/log/datadog/agent.log.

2024-02-14 13:44:38 PST | CORE | ERROR | (comp/process/agent/agentimpl/agent_linux.go:51 in agentEnabled) | Network Performance Monitoring is not supported in the core agent. Please ensure the process-agent is enabled as a standalone agent to collect process, container and network performance metrics.

Verify process and container data are visible in Live Processes
Verify the process-agent is running in Live Processes

Test Case 4 - Running the process checks in the core agent disabled and language detection is enabled

Setup Config

datadog.yaml

process_config:
  process_collection:
    enabled: true
  run_in_core_agent:
    enabled: false
language_detection:
  enabled: true

Start datadog agent
sudo service datadog-agent start
Verify the process-agent starts logs under /var/log/datadog/process-agent.log

2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:276 in Run) | Starting process-agent with enabled checks=[process rtprocess]
2024-02-14 12:37:42 PST | PROCESS | INFO | (pkg/process/runner/runner.go:233 in logCheckDuration) | Finished process check #1 in 9.249009ms

Verify the process agent component does not start in the core agent under /var/log/datadog/agent.log. There should be no process checks running or starting.
Verify process and container data are visible in Live Processes
Verify the process-agent is running in Live Processes

Mar 01 '24 01:03 wiyu

Bloop Bleep... Dogbot Here

Regression Detector Results

Run ID: 86ce9ab2-4b85-4b2f-8c4b-6f7a6bc93a9b Baseline: c221f839802de34c89f5b6454a7e2ac675863ea5 Comparison: c1073a0d6d73b45e52ce206f7e274ef8638d927c

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	-1.14	[-7.64, +5.36]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	basic_py_check	% cpu utilization	+1.33	[-0.95, +3.61]
➖	file_tree	memory utilization	+0.18	[+0.11, +0.25]
➖	otel_to_otel_logs	ingress throughput	+0.18	[-0.43, +0.78]
➖	trace_agent_json	ingress throughput	+0.02	[-0.02, +0.05]
➖	trace_agent_msgpack	ingress throughput	+0.00	[-0.01, +0.02]
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.00, +0.00]
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.00, +0.00]
➖	process_agent_standard_check_with_stats	memory utilization	-0.01	[-0.03, +0.02]
➖	idle	memory utilization	-0.03	[-0.06, +0.01]
➖	process_agent_standard_check	memory utilization	-0.24	[-0.27, -0.21]
➖	process_agent_real_time_mode	memory utilization	-0.47	[-0.50, -0.43]
➖	tcp_syslog_to_blackhole	ingress throughput	-0.53	[-0.59, -0.47]
➖	file_to_blackhole	% cpu utilization	-1.14	[-7.64, +5.36]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-1.53	[-2.96, -0.11]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".