trafficstars

What does this PR do?

Allow omnibus to package and build the agent at a different location than /opt/datadog-agent. Also gives the option to generate config files at a custom path instead of /etc/datadog-agent.

Note: The packaged agent is still meant to work from /opt/datadog-agent and will still try to get config files from /etc/datadog-agent

Motivation

MacOS runners don't support virtualization and have already a datadog-agent present for internal tracking. We need to be able to build the agent on runners without overwriting on the existing agent.

Additional Notes

This PR depends on this omnibus-ruby PR

Currently tested only on the base flavor Capture d’écran 2024-09-06 à 19 28 47 Once the macOS builds uses this PR, we can remove this test job since regression will be handled by the macOS build job.

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Manual testing

export INSTALL_DIR="/some/custom/path" # needs to be exported
inv omnibus.build --install-directory "$INSTALL_DIR" --config-directory "/some/custom/config/path"

CI Currently custom_agent_build_test create an x64 build to /custom/datadog-agent with config files in /custom/etc/datadog-agent the job fails if not correctly generated at these locations.

Sep 06 '24 17:09 Pythyu

Regression Detector

Regression Detector Results

Run ID: 926cc1b6-d43d-4528-9a97-dff5c54aaa6a Metrics dashboard Target profiles

Baseline: eda71a74e309e7b32b2b80c74ed32128dcee4b57 Comparison: 9befe2087205abc82df0cf5cacec5c3c0cb47410

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	otel_to_otel_logs	ingress throughput	+0.69	[-0.14, +1.51]	1	Logs
➖	idle	memory utilization	+0.22	[+0.18, +0.26]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.01, +0.01]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.00	[-0.00, +0.00]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.12	[-0.18, -0.05]	1	Logs
➖	uds_dogstatsd_to_api_cpu	% cpu utilization	-0.32	[-1.08, +0.44]	1	Logs
➖	file_tree	memory utilization	-0.63	[-0.75, -0.52]	1	Logs
➖	basic_py_check	% cpu utilization	-0.75	[-3.56, +2.07]	1	Logs
➖	pycheck_lots_of_tags	% cpu utilization	-1.33	[-3.97, +1.31]	1	Logs

Bounds Checks

perf	experiment	bounds_check_name	replicates_passed
✅	idle	memory_usage	10/10

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

Sep 06 '24 18:09 pr-commenter[bot]

Gitlab CI Configuration Changes

Modified Jobs

.agent_build_common

  .agent_build_common:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    variables:
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi

datadog-agent-6-arm64

  datadog-agent-6-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_6
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 6
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '2,3'

datadog-agent-6-x64

  datadog-agent-6-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_6
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 6
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '2,3'

datadog-agent-7-arm64

  datadog-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '3'

datadog-agent-7-x64

  datadog-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: base
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-arm64

  datadog-ot-agent-7-arm64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-arm64
    - go_deps
    - generate_minimized_btfs_arm64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:arm64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: arm64
      PYTHON_RUNTIMES: '3'

datadog-ot-agent-7-x64

  datadog-ot-agent-7-x64:
    artifacts:
      expire_in: 2 weeks
      paths:
      - $OMNIBUS_PACKAGE_DIR
    before_script:
    - export RELEASE_VERSION=$RELEASE_VERSION_7
    cache:
    - key:
        files:
        - omnibus/Gemfile
        - release.json
        prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
      paths:
      - omnibus/vendor/bundle
    image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
    needs:
    - go_mod_tidy_check
    - build_system-probe-x64
    - go_deps
    - generate_minimized_btfs_x64
    rules:
    - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
      when: never
    - when: on_success
    script:
    - echo "About to build for $RELEASE_VERSION"
    - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
    - rm -f modcache.tar.xz
    - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
    - rm -rf $OMNIBUS_PACKAGE_DIR/*
    - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
    - mkdir -p /tmp/system-probe
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
    - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
    - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
    - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
    - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
      --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
      --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
-     --flavor "$FLAVOR"
+     --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
    - ls -la $OMNIBUS_PACKAGE_DIR
    - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
    stage: package_build
    tags:
    - arch:amd64
    variables:
      AGENT_MAJOR_VERSION: 7
      FLAVOR: ot
      KUBERNETES_CPU_REQUEST: 16
      KUBERNETES_MEMORY_LIMIT: 32Gi
      KUBERNETES_MEMORY_REQUEST: 32Gi
      PACKAGE_ARCH: amd64
      PYTHON_RUNTIMES: '3'

Added Jobs

.agent_build_script

.agent_build_script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
  --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
  --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
  --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json

datadog-agent-7-x64-custom-path-test

datadog-agent-7-x64-custom-path-test:
  artifacts:
    expire_in: 2 weeks
    paths:
    - $OMNIBUS_PACKAGE_DIR
  before_script:
  - export RELEASE_VERSION=$RELEASE_VERSION_7
  cache:
  - key:
      files:
      - omnibus/Gemfile
      - release.json
      prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
    paths:
    - omnibus/vendor/bundle
  image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
  needs:
  - go_mod_tidy_check
  - build_system-probe-x64
  - go_deps
  - generate_minimized_btfs_x64
  rules:
  - if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
    when: never
  - when: on_success
  script:
  - mkdir /custom
  - export CONFIG_DIR="/custom"
  - export INSTALL_DIR="/custom/datadog-agent"
  - echo "About to build for $RELEASE_VERSION"
  - mkdir -p $GOPATH/pkg/mod && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod
  - rm -f modcache.tar.xz
  - pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
  - rm -rf $OMNIBUS_PACKAGE_DIR/*
  - tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
  - mkdir -p /tmp/system-probe
  - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
  - $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
  - cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
  - chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
  - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
    --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR  ${USE_S3_CACHING}
    --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
    --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
  - ls -la $OMNIBUS_PACKAGE_DIR
  - $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
  - ls -la $OMNIBUS_PACKAGE_DIR
  - ls -la $INSTALL_DIR
  - ls -la /custom/etc
  - (ls -la /opt/datadog-agent 2>/dev/null && exit 1) || echo "/opt/datadog-agent
    has correctly not been generated"
  - (ls -la /etc/datadog-agent 2>/dev/null && exit 1) || echo "/etc/datadog-agent
    has correctly not been generated"
  stage: package_build
  tags:
  - arch:amd64
  variables:
    AGENT_MAJOR_VERSION: 7
    FLAVOR: base
    KUBERNETES_CPU_REQUEST: 16
    KUBERNETES_MEMORY_LIMIT: 32Gi
    KUBERNETES_MEMORY_REQUEST: 32Gi
    PACKAGE_ARCH: amd64
    PYTHON_RUNTIMES: '3'

Changes Summary

Removed	Modified	Added	Renamed
0	7	2	0

Sep 09 '24 12:09 pr-commenter[bot]

[Fast Unit Tests Report]

On pipeline 45000364 (CI Visibility). The following jobs did not run any unit tests:

Jobs:

tests_deb-arm64-py3
tests_deb-x64-py3
tests_flavor_dogstatsd_deb-x64
tests_flavor_heroku_deb-x64
tests_flavor_iot_deb-x64
tests_rpm-arm64-py3
tests_rpm-x64-py3
tests_windows-x64

If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help

Sep 09 '24 12:09 agent-platform-auto-pr[bot]

/merge

Sep 25 '24 11:09 Pythyu

:steam_locomotive: MergeQueue: pull request added to the queue

The median merge time in main is 23m.

Use /merge -c to cancel this operation!

Sep 25 '24 11:09 dd-devflow[bot]

datadog-agent
datadog-agent copied to clipboard

[Omnibus] Support packaging of the agent at a different location than /opt/datadog-agent

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

Gitlab CI Configuration Changes

Modified Jobs

Added Jobs

Changes Summary

datadog-agent datadog-agent copied to clipboard

[Omnibus] Support packaging of the agent at a different location than /opt/datadog-agent

What does this PR do?

Motivation

Additional Notes

Possible Drawbacks / Trade-offs

Describe how to test/QA your changes

Regression Detector

Regression Detector Results

No significant changes in experiment optimization goals

Fine details of change detection per experiment

Bounds Checks

Explanation

Gitlab CI Configuration Changes

Modified Jobs

Added Jobs

Changes Summary

datadog-agent
datadog-agent copied to clipboard