[Backport 7.58.x] auto-instrumentation: increase default memory request to 100Mi

[agent-platform-auto-pr[bot]]

Backport 023ef5945eaed5aea081f20a7788a68c0107f816 from #29392.

---

What does this PR do?

Increases the default memory request for the auto instrumentation container from 20Mi to 100Mi.

100Mi is closer to the recommended minimum memory requirements for Alpine.

Motivation

https://datadoghq.atlassian.net/browse/APMON-1472

Right now the containers may get OOM Killed when copying lib injection files from the lib injection container to the application container. This memory usage comes from the cp command's usage of sendfile and is correlated to the total number of files being copied.

From @levan-m:

From my perspective, making this change in Agent is cleaner. Memory increase will be tied to a specific Agent version upgrade. On the other hand, Operator config may change while Agent version is pinned (or vice versa). This could create a situation where Operator overrides default value in future Agent versions, or sets higher limit on older installations which work fine with 20mb.

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Testing limits manually.

Before:

$ docker run -it --rm  --memory=20Mib --memory-swap=20Mib -v "$(pwd):/out" gcr.io/datadoghq/dd-lib-python-init:2.12 /datadog-init/copy-lib.sh /out/
Killed

After:

$ docker run -it --rm  --memory=100Mib --memory-swap=100Mib -v "$(pwd):/out" gcr.io/datadoghq/dd-lib-python-init:2.12 /datadog-init/copy-lib.sh /out/

This change can also be validated by manually updating the cluster agent configuation:

Manually setting the resource request:

datadog:
 apiKey: <API-KEY>
 site: datadoghq.com
 tags:
      - env:<ENV>
 apm:
   instrumentation:
      enabled: true
      libVersions:
         java: "1"
         dotnet: "3"
         python: "2"
         js: "5"
         ruby: "2"
clusterAgent:
  env:
    - name: DD_ADMISSION_CONTROLLER_AUTO_INSTRUMENTATION_INIT_RESOURCES_MEMORY
      value: 100Mi

Using a custom built image with the changes:

datadog:
 apiKey: <API-KEY>
 site: datadoghq.com
 tags:
      - env:<ENV>
 apm:
   instrumentation:
      enabled: true
      libVersions:
         java: "1"
         dotnet: "3"
         python: "2"
         js: "5"
         ruby: "2"
clusterAgent:
   image:
     name: <user>/cluster_agent
     tag: master
     repository: docker.io/<user>/cluster_agent
     doNotCheckTag: true

helm upgrade datadog-agent -f datadog-values.yml datadog/datadog

Sample app configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: kyle-django-app
  labels:
    app: python-app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: app
        image: ddkverhoog/django-helloworld:v1.0.1
        env:
          - name: DD_TRACE_DEBUG
            value: "true"
        readinessProbe:
          timeoutSeconds: 1
          successThreshold: 1
          failureThreshold: 1
          httpGet:
            host:
            scheme: HTTP
            path: /
            port: 18080
          initialDelaySeconds: 30
          periodSeconds: 1
        ports:
          - containerPort: 18080
            protocol: TCP

[agent-platform-auto-pr[bot]]

[Fast Unit Tests Report]

On pipeline 44545146 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

• tests_flavor_dogstatsd_deb-x64
• tests_flavor_heroku_deb-x64
• tests_flavor_iot_deb-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

[pr-commenter[bot]]

Test changes on VM

Use this command from test-infra-definitions to manually test this PR changes on a VM:

inv create-vm --pipeline-id=44545146 --os-family=ubuntu

Note: This applies to commit a40de3a5

[pr-commenter[bot]]

Regression Detector

Regression Detector Results

Run ID: ede5babe-7973-4048-8f39-15a2302f4eae Metrics dashboard Target profiles

Baseline: 2e1846529cd4e864428c769af79d6454efcead77 Comparison: a40de3a5b9bda3f8847739328fcae9a366c56266

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	file_tree	memory utilization	+0.50	[+0.39, +0.61]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	+0.49	[-0.27, +1.26]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	+0.33	[+0.28, +0.38]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.00, +0.00]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	-0.00	[-0.01, +0.01]	1	Logs
➖	otel_to_otel_logs	ingress throughput	-0.24	[-1.06, +0.59]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	-0.28	[-2.96, +2.41]	1	Logs
➖	idle	memory utilization	-0.39	[-0.43, -0.34]	1	Logs
➖	basic_py_check	% cpu utilization	-0.70	[-3.54, +2.14]	1	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	idle	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[davidor]

/merge

[dd-devflow[bot]]

:steam_locomotive: MergeQueue: pull request added to the queue

The median merge time in 7.58.x is 22m.

Use /merge -c to cancel this operation!