What does this PR do?

Adds support for the failing over of logs when High Availability mode is enabled and failover is triggered.

Motivation

This work is happening as part of the larger HAMR effort.

Additional Notes

The implementation follows closely with what we did for metrics (#21644), which seemed to fit best when placed down at the level of each DestinationSender.

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

As in #21644, no Agent behavior changes unless High Availability mode is enabled and failover is triggered.

configure your Agent to tail a temporary file, such as /tmp/test-logs.log (follow these instructions, specifically the Tail files use case, to configure a tailer for the log file)
create a second API key to use for failover (best done in another org in another datacenter than your existing API key; easier to watch Logs Live Tail for both orgs)
start the Agent with High Availability mode and logs collection enabled, and the failover site and API key configured: DD_LOGS_ENABLED=true DD_HA_ENABLED=true DD_HA_SITE=<failover site i.e. us5.datadoghq.com> DD_HA_API_KEY=xxxxxxxxxxx
open another shell and run something like this: while true; do sleep 1; echo "fake log line - the time is now $(date +%s)" >> /tmp/test-logs.log; done
make sure you're seeing the logs come into the primary org
trigger failover by setting ha.failover to true: datadog-agent config set ha.failover true
see a log message in the Agent's output about "Forwarder for domain ... has been been failed over to, enabling it for HA."
check that logs are flowing not only into the primary org, but also the failover org
disable failover by setting ha.failover to false: datadog-agent config set ha.failover false
logs should stop flowing to the failover org

Mar 06 '24 20:03 tobz

We may want to consider adding a condition here to force HTTP when HA is enabled.

If an agent comes up during a failover, the connectivity check could fail forcing the agent to fallback to the TCP sender which feels like incorrect behavior in this scenario.

Mar 11 '24 15:03 gh123man

Bloop Bleep... Dogbot Here

Regression Detector Results

Run ID: 5c524fad-130b-40c1-9b41-0e1e04bae461 Baseline: 39eea0d6bce393e259882133d83548563da0979e Comparison: f469a95bbedac7a88a0aab108e8d9a7bd5f27d8f

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	file_to_blackhole	% cpu utilization	-0.14	[-6.73, +6.45]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	pycheck_1000_100byte_tags	% cpu utilization	+2.65	[-2.67, +7.98]
➖	process_agent_real_time_mode	memory utilization	+0.67	[+0.63, +0.71]
➖	file_tree	memory utilization	+0.44	[+0.37, +0.50]
➖	otel_to_otel_logs	ingress throughput	+0.42	[-0.23, +1.07]
➖	process_agent_standard_check_with_stats	memory utilization	+0.20	[+0.16, +0.23]
➖	tcp_syslog_to_blackhole	ingress throughput	+0.20	[+0.12, +0.28]
➖	idle	memory utilization	+0.10	[+0.06, +0.14]
➖	basic_py_check	% cpu utilization	+0.08	[-2.36, +2.52]
➖	trace_agent_json	ingress throughput	+0.03	[-0.01, +0.07]
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.05, +0.05]
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.06, +0.06]
➖	trace_agent_msgpack	ingress throughput	-0.02	[-0.03, -0.01]
➖	file_to_blackhole	% cpu utilization	-0.14	[-6.73, +6.45]
➖	process_agent_standard_check	memory utilization	-0.26	[-0.29, -0.22]
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.39	[-3.51, +2.73]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

Mar 12 '24 15:03 pr-commenter[bot]

We may want to consider adding a condition here to force HTTP when HA is enabled.

If an agent comes up during a failover, the connectivity check could fail forcing the agent to fallback to the TCP sender which feels like incorrect behavior in this scenario.

Addressed this in 90e5cf000db12a505a00706d291a5b7db9c4df0d, btw.

Mar 13 '24 13:03 tobz

/merge

Mar 13 '24 16:03 tobz

:steam_locomotive: MergeQueue

This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals. Note: if you pushed new commits since the last approval, you may need additional approval. You can remove it from the waiting list with /remove command.

Use /merge -c to cancel this operation!

Mar 13 '24 16:03 dd-devflow[bot]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=30039846 --os-family=ubuntu

Mar 13 '24 17:03 pr-commenter[bot]

:steam_locomotive: MergeQueue

Added to the queue.

There are 2 builds ahead of this PR! (estimated merge in less than 26m)

Use /merge -c to cancel this operation!

Mar 13 '24 17:03 dd-devflow[bot]