datadog-agent
datadog-agent copied to clipboard
Published
20 hours ago •
DataDog
DataDog
Start dropping python2 support
What does this PR do?
Start dropping python2 as we don't ship it with the agent anymore. It handles:
- the CI job parameters
- rtloader
- omnibus
- The MSI package
Motivation
We don't need to clutter the code with various parameters or branches for something we don't support anymore.
There are most likely some missing pieces and a lot of remaining references to python2, but this serves as a good starting point
Describe how to test/QA your changes
I'm not sure this needs dedicated QA as long as the CI still passes since we're only removing code, but since the change is large, I can definitely convinced otherwise
Possible Drawbacks / Trade-offs
Additional Notes
This is probably better review commit by commit. The changes should be very close to atomic, and things can be split out in multiple PRs if we chose to.
Associated PRs:
- https://github.com/DataDog/omnibus-software/pull/549
- https://github.com/DataDog/datadog-agent-macos-build/pull/213
Regression Detector
Regression Detector Results
Run ID: 8fa7ba51-edb2-4636-b05a-46f0c8360bce Metrics dashboard Target profiles
Baseline: 5f34bad6de8bcff8c6d29ff000b2f93e0ef1980f Comparison: 8ce88a9e555ce9604b97a356e78e26c67158eef8
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
No significant changes in experiment optimization goals
Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +3.06 | [+2.32, +3.79] | 1 | Logs |
| ➖ | pycheck_lots_of_tags | % cpu utilization | +0.78 | [-1.75, +3.31] | 1 | Logs |
| ➖ | file_tree | memory utilization | +0.61 | [+0.48, +0.73] | 1 | Logs |
| ➖ | idle | memory utilization | +0.37 | [+0.33, +0.41] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle | memory utilization | +0.26 | [+0.22, +0.31] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | +0.24 | [-0.25, +0.73] | 1 | Logs |
| ➖ | idle_all_features | memory utilization | +0.18 | [+0.08, +0.28] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.06 | [-0.19, +0.30] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.00 | [-0.07, +0.08] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.01, +0.01] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.01 | [-0.34, +0.32] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.01 | [-0.24, +0.21] | 1 | Logs |
| ➖ | otel_to_otel_logs | ingress throughput | -0.05 | [-0.86, +0.76] | 1 | Logs |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | -0.07 | [-0.25, +0.11] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.12 | [-0.20, -0.05] | 1 | Logs |
| ➖ | basic_py_check | % cpu utilization | -0.16 | [-2.92, +2.60] | 1 | Logs |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.54 | [-0.64, -0.44] | 1 | Logs bounds checks dashboard |
Bounds Checks
| perf | experiment | bounds_check_name | replicates_passed |
|---|---|---|---|
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 |
| ✅ | idle | memory_usage | 10/10 |
| ✅ | idle_all_features | memory_usage | 10/10 |
| ✅ | quality_gate_idle | memory_usage | 10/10 |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
![cit-pr-commenter[bot] avatar](https://avatars.githubusercontent.com/in/330445?v=4 "Published by a pub.dev verified publisher"){kind=small}
Python compression & decompression in the MSI installer.
I think I addressed this now, but if not I might need a few more pointers/help
Gitlab CI Configuration Changes
Modified Jobs
.agent_7_build
.agent_7_build:
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: base
- PYTHON_RUNTIMES: '3'
.agent_build_common
.agent_build_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
.agent_build_common_dmg
.agent_build_common_dmg:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
script:
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- mkdir -p $OMNIBUS_PACKAGE_DIR
- "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 1\"\nelse\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 2\"\nfi\n"
- $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
- export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
- python3 -m pip install -r tasks/libs/requirements-github.txt
- inv -e github.trigger-macos --workflow-type "build" --datadog-agent-ref "$CI_COMMIT_SHA"
- --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --python-runtimes
? ^^ ^ ---------
+ --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --destination
? ^^^ ^^^^^
- "$PYTHON_RUNTIMES" --destination "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT"
- --integrations-core-ref "$INTEGRATIONS_CORE_VERSION"
+ "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT" --integrations-core-ref
+ "$INTEGRATIONS_CORE_VERSION"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
timeout: 3h
.agent_build_script
.agent_build_script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
.common_build_oci
.common_build_oci:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
script:
- echo "About to build for $RELEASE_VERSION"
- AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
- export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --host-distribution=ociru --install-directory="$INSTALL_DIR"
+ --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
? +++++++++++++++++++++++++++++++++++++
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
.dogstatsd_build_common
.dogstatsd_build_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
? ^^^^^^ -------
+ - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
? ^^^^ ++
- 3 --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
? -------------
+ $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--target-project dogstatsd
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
.heroku_build_base
.heroku_build_base:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor heroku
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor heroku
- ls -la $OMNIBUS_PACKAGE_DIR
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent-dbg_*_${PACKAGE_ARCH}.deb
$S3_ARTIFACTS_URI/$DESTINATION_DBG_DEB
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
DD_PKG_ARCH: x86_64
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
.integration_tests_windows_base
.integration_tests_windows_base:
before_script:
- $tmpfile = [System.IO.Path]::GetTempFileName()
- (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
-tempFile "$tmpfile")
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- $vcpkgBlobSaSUrl=$(cat "$tmpfile")
- Remove-Item "$tmpfile"
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
-e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+ -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
- -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
- -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\integrationtests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: integration_test
tags:
- runner:windows-docker
- windowsversion:1809
.lint_macos_gitlab
.lint_macos_gitlab:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
- variables:
- PYTHON_RUNTIMES: '3'
.lint_windows_base
.lint_windows_base:
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
? ----------------------------------
+ -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
c:\mnt\tasks\winbuildscripts\lint.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: lint
tags:
- runner:windows-docker
- windowsversion:1809
.linux_lint
.linux_lint:
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
variables:
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
.linux_tests
.linux_tests:
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit: '**/junit-out-*.xml'
when: always
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
variables:
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
TEST_OUTPUT_FILE: test_output.json
.macos_gitlab
.macos_gitlab:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
- variables:
- PYTHON_RUNTIMES: '3'
.ot_agent_7_build
.ot_agent_7_build:
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: ot
- PYTHON_RUNTIMES: '3'
.rtloader_tests
.rtloader_tests:
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
needs:
- go_deps
script:
- '# Skipping go tests'
stage: source_test
.tests_macos_gitlab
.tests_macos_gitlab:
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit: '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
+ - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
- --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
- $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
.tests_windows_base
.tests_windows_base:
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
junit: '**/junit-out-*.xml'
when: always
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- $FAST_TESTS_FLAG=""
- If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
-e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
-e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
-e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
- -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
- -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
- PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
- EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+ -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+ -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+ TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
- -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
? ----------------------------------
+ -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
-e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\unittests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
TEST_OUTPUT_FILE: test_output.json
.tests_windows_secagent
.tests_windows_secagent:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\secagent.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
.tests_windows_sysprobe
.tests_windows_sysprobe:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\sysprobe.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
.windows_main_agent_base
.windows_main_agent_base:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
needs:
- go_mod_tidy_check
- go_deps
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
variables:
OMNIBUS_TARGET: main
.windows_msi_base
.windows_msi_base:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
needs:
- go_mod_tidy_check
- go_deps
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
agent_dmg-x64-a7
agent_dmg-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: main
paths:
- comp/core/gui/guiimpl/systray/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: main
paths:
- omnibus/**/*
- .gitlab-ci.yml
- release.json
- .gitlab/package_build/**/*
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- mkdir -p $OMNIBUS_PACKAGE_DIR
- "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 1\"\nelse\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 2\"\nfi\n"
- $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
- export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
- python3 -m pip install -r tasks/libs/requirements-github.txt
- inv -e github.trigger-macos --workflow-type "build" --datadog-agent-ref "$CI_COMMIT_SHA"
- --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --python-runtimes
? ^^ ^ ---------
+ --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --destination
? ^^^ ^^^^^
- "$PYTHON_RUNTIMES" --destination "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT"
- --integrations-core-ref "$INTEGRATIONS_CORE_VERSION"
+ "$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT" --integrations-core-ref
+ "$INTEGRATIONS_CORE_VERSION"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
timeout: 6h
variables:
AGENT_MAJOR_VERSION: 7
- PYTHON_RUNTIMES: '3'
agent_heroku_deb-x64-a7
agent_heroku_deb-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor heroku
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor heroku
- ls -la $OMNIBUS_PACKAGE_DIR
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-heroku-agent-dbg_*_${PACKAGE_ARCH}.deb
$S3_ARTIFACTS_URI/$DESTINATION_DBG_DEB
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
DD_PKG_ARCH: x86_64
DESTINATION_DBG_DEB: datadog-heroku-agent-dbg_7_amd64.deb
DESTINATION_DEB: datadog-heroku-agent_7_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
RELEASE_VERSION: $RELEASE_VERSION_7
datadog-agent-7-arm64
datadog-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-agent-7-x64
datadog-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-agent-7-x64-custom-path-test
datadog-agent-7-x64-custom-path-test:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir /custom
- export CONFIG_DIR="/custom"
- export INSTALL_DIR="/custom/datadog-agent"
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
- ls -la $OMNIBUS_PACKAGE_DIR
- ls -la $INSTALL_DIR
- ls -la /custom/etc
- (ls -la /opt/datadog-agent 2>/dev/null && exit 1) || echo "/opt/datadog-agent
has correctly not been generated"
- (ls -la /etc/datadog-agent 2>/dev/null && exit 1) || echo "/etc/datadog-agent
has correctly not been generated"
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-agent-oci-arm64-a7
datadog-agent-oci-arm64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
- export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --host-distribution=ociru --install-directory="$INSTALL_DIR"
+ --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
? +++++++++++++++++++++++++++++++++++++
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
DESTINATION_OCI: datadog-agent-7-remote-updater-arm64.tar.xz
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-agent-oci-x64-a7
datadog-agent-oci-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
- export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --host-distribution=ociru --install-directory="$INSTALL_DIR"
+ --system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
? +++++++++++++++++++++++++++++++++++++
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
DESTINATION_OCI: datadog-agent-7-remote-updater-amd64.tar.xz
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-arm64
datadog-ot-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-x64
datadog-ot-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
- --skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
- --flavor "$FLAVOR" --config-directory "$CONFIG_DIR" --install-directory "$INSTALL_DIR"
+ --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
+ --system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
+ --install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
dogstatsd-arm64
dogstatsd-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_dogstatsd-binary_arm64
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
? ^^^^^^ -------
+ - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
? ^^^^ ++
- 3 --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
? -------------
+ $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--target-project dogstatsd
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CXX: aarch64-unknown-linux-gnu-g++
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
dogstatsd-x64
dogstatsd-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_dogstatsd-binary_x64
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --python-runtimes
? ^^^^^^ -------
+ - inv -e omnibus.build --release-version $RELEASE_VERSION_7 --major-version 7 --base-dir
? ^^^^ ++
- 3 --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
? -------------
+ $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--target-project dogstatsd
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CXX: x86_64-unknown-linux-gnu-g++
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
integration_tests_windows-x64
integration_tests_windows-x64:
before_script:
- $tmpfile = [System.IO.Path]::GetTempFileName()
- (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
-tempFile "$tmpfile")
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- $vcpkgBlobSaSUrl=$(cat "$tmpfile")
- Remove-Item "$tmpfile"
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
-e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+ -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
- -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
- -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\integrationtests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: integration_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
lint_flavor_dogstatsd_linux-x64
lint_flavor_dogstatsd_linux-x64:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
tags:
- arch:amd64
variables:
FLAVORS: --flavor dogstatsd
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
lint_flavor_heroku_linux-x64
lint_flavor_heroku_linux-x64:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
tags:
- arch:amd64
variables:
FLAVORS: --flavor heroku
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
lint_flavor_iot_linux-x64
lint_flavor_iot_linux-x64:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
tags:
- arch:amd64
variables:
FLAVORS: --flavor iot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
lint_linux-arm64
lint_linux-arm64:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
tags:
- arch:arm64
variables:
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
lint_linux-x64
lint_linux-x64:
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "3"
? ----------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e install-tools
- inv -e linter.go --cpus $KUBERNETES_CPU_REQUEST --debug $FLAVORS $EXTRA_OPTS
stage: lint
tags:
- arch:amd64
variables:
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
lint_macos_gitlab_amd64
lint_macos_gitlab_amd64:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
tags:
- macos:monterey-amd64
- specific:true
- variables:
- PYTHON_RUNTIMES: '3'
lint_macos_gitlab_arm64
lint_macos_gitlab_arm64:
allow_failure: true
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
tags:
- macos:monterey-arm64
- specific:true
- variables:
- PYTHON_RUNTIMES: '3'
lint_windows-x64
lint_windows-x64:
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
? ----------------------------------
+ -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
c:\mnt\tasks\winbuildscripts\lint.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: lint
tags:
- runner:windows-docker
- windowsversion:1809
timeout: 1h30m
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
tests_deb-arm64-py3
tests_deb-arm64-py3:
after_script:
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:arm64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_deb-x64-py3
tests_deb-x64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_dogstatsd_deb-x64
tests_flavor_dogstatsd_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor dogstatsd
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_heroku_deb-x64
tests_flavor_heroku_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor heroku
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_iot_deb-x64
tests_flavor_iot_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor iot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_macos
tests_macos:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh "junit-*-repacked.tgz"
artifacts:
expire_in: 2 weeks
paths:
- test_output.json
- junit-*-repacked.tgz
reports:
junit:
- '**/junit-out-*.xml'
when: always
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- setup_agent_version
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $RUN_UNIT_TESTS == "on"
- changes:
compare_to: main
paths:
- pkg/fleet/**/*
variables:
FAST_TESTS: 'true'
- allow_failure: true
when: manual
script:
- "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 1\"\nelse\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 2\"\nfi\n"
- $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
- export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
- python3 -m pip install -r tasks/libs/requirements-github.txt
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" = "true" ]]; then FAST_TESTS_FLAG="--fast-tests true"; fi
- inv -e github.trigger-macos --workflow-type "test" --datadog-agent-ref "$CI_COMMIT_SHA"
+ --version-cache "$VERSION_CACHE_CONTENT" $FAST_TESTS_FLAG --test-washer $COVERAGE_CACHE_FLAG
- --python-runtimes "$PYTHON_RUNTIMES" --version-cache "$VERSION_CACHE_CONTENT"
- $FAST_TESTS_FLAG --test-washer $COVERAGE_CACHE_FLAG
stage: source_test
tags:
- arch:amd64
timeout: 6h
- variables:
- PYTHON_RUNTIMES: '3'
tests_macos_gitlab_amd64
tests_macos_gitlab_amd64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
+ - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
- --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
- $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
tags:
- macos:monterey-amd64
- specific:true
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_macos_gitlab_arm64
tests_macos_gitlab_arm64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- - inv -e rtloader.make --python-runtimes $PYTHON_RUNTIMES
+ - inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
+ - inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- - inv -e test --rerun-fails=2 --python-runtimes $PYTHON_RUNTIMES --race --profile
- --cpus 12 --save-result-json $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz"
- $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
tags:
- macos:monterey-arm64
- specific:true
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_rpm-arm64-py3
tests_rpm-arm64-py3:
after_script:
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:arm64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_rpm-x64-py3
tests_rpm-x64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev --python-runtimes "$PYTHON_RUNTIMES"
? -------------------------------------
+ - inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --python-runtimes
? ^^^^ ^^ ^^^^^ -
+ - inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
? ^ ^^ ^^
- "$PYTHON_RUNTIMES" --coverage --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json
+ --cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
- $TEST_OUTPUT_FILE --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG
? ------------------
+ --junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
? ++++++++++++++
- --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
EXTRA_OPTS: --build-exclude=systemd
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_windows-x64
tests_windows-x64:
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
junit:
- '**/junit-out-*.xml'
when: always
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- $FAST_TESTS_FLAG=""
- If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
-e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
-e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
-e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
- -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
- -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
- PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
- EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+ -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+ -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+ TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
- -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
? ----------------------------------
+ -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
-e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\unittests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
TEST_OUTPUT_FILE: test_output.json
tests_windows_secagent_x64
tests_windows_secagent_x64:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\secagent.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
tests_windows_sysprobe_x64
tests_windows_sysprobe_x64:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\sysprobe.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
windows_msi_and_bosh_zip_x64-a7
windows_msi_and_bosh_zip_x64-a7:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
before_script:
- set RELEASE_VERSION $RELEASE_VERSION_7
needs:
- go_mod_tidy_check
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
timeout: 2h
variables:
AGENT_MAJOR_VERSION: 7
ARCH: x64
OMNIBUS_TARGET: main
- PYTHON_RUNTIMES: '3'
windows_zip_agent_binaries_x64-a7
windows_zip_agent_binaries_x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
before_script:
- set RELEASE_VERSION $RELEASE_VERSION_7
needs:
- go_mod_tidy_check
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG}
-e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME} -e OMNIBUS_TARGET=${OMNIBUS_TARGET}
-e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION" -e MAJOR_VERSION="$AGENT_MAJOR_VERSION"
- -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES"
? ^^^^^^^^ - ^^^^^^^^^^^^^^^^
+ -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e GOMODCACHE="c:\modcache"
? ^^ ++++++ ^^^^^^^^^^^
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e BUCKET_BRANCH="$BUCKET_BRANCH"
+ -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET"
+ -e USE_S3_CACHING="$USE_S3_CACHING" -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG}
- -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
- -e BUCKET_BRANCH="$BUCKET_BRANCH" -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET"
- -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
- -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
- -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
variables:
AGENT_MAJOR_VERSION: 7
ARCH: x64
OMNIBUS_TARGET: agent_binaries
Changes Summary
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 56 | 0 | 0 |
:information_source: Diff available in the job log.
![agent-platform-auto-pr[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}
Python compression & decompression in the MSI installer.
I think I addressed this now, but if not I might need a few more pointers/help
There is still code that perform decompression of Python 2 left in the installer, see https://github.com/DataDog/datadog-agent/blob/main/tools/windows/DatadogAgentInstaller/CustomActions/PythonDistributionCustomAction.cs#L137 It doesn't fail the pipeline, but that doesn't mean we should leave dead code around, this should work:
private static ActionResult DecompressPythonDistribution(
ISession session,
string outputDirectoryName,
string compressedDistributionFile,
string pythonDistributionName,
int pythonDistributionSize)
{
...
if (File.Exists(embedded))
{
....
}
else
{
throw new InvalidOperationException($"The file {embedded} doesn't exist, but it should");
}
}
private static ActionResult DecompressPythonDistributions(ISession session)
{
var size = 0;
var embedded3Size = session.Property("embedded3_SIZE");
if (!string.IsNullOrEmpty(embedded3Size))
{
size = int.Parse(embedded3Size);
}
return DecompressPythonDistribution(session, "embedded3", "embedded3.COMPRESSED", "Python 3", size);
}
private static ActionResult PrepareDecompressPythonDistributions(ISession session)
{
try
{
var total = 0;
var embedded3Size = session.Property("embedded3_SIZE");
if (!string.IsNullOrEmpty(embedded3Size))
{
total += int.Parse(embedded3Size);
}
// Add embedded Python size to the progress bar size
// Even though we can't record accurate progress, it will look like it's
// moving every time we decompress a Python distribution.
using var record = new Record(MessageRecordFields.ProgressAddition, total);
if (session.Message(InstallMessage.Progress, record) != MessageResult.OK)
{
session.Log("Could not set the progress bar size");
return ActionResult.Failure;
}
}
catch (Exception e)
{
session.Log($"Error settings the progress bar size: {e}");
return ActionResult.Failure;
}
return ActionResult.Success;
}
Python compression & decompression in the MSI installer.
I think I addressed this now, but if not I might need a few more pointers/help
Ok that seems to be consistent with what I pushed a bit earlier
To claritfy things though, I don't clame to remove all traces of python2 with this PR, this is a starting point that removes the most obvious occurrences. There are still some Python2_Home traces throughout the code that I plan to remove in a later PR, but I fear doing it all at once is a bit risky
Test changes on VM
Use this command from test-infra-definitions to manually test this PR changes on a VM:
inv create-vm --pipeline-id=46518099 --os-family=ubuntu
Note: This applies to commit aa96c86e
:steam_locomotive: MergeQueue: waiting for PR to be ready
This merge request is not mergeable yet, because of pending checks/missing approvals. It will be added to the queue as soon as checks pass and/or get approvals.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.
Use /merge -c to cancel this operation!
[Fast Unit Tests Report]
On pipeline 47595959 (CI Visibility). The following jobs did not run any unit tests:
Jobs:
- tests_deb-arm64-py3
- tests_deb-x64-py3
- tests_flavor_dogstatsd_deb-x64
- tests_flavor_heroku_deb-x64
- tests_flavor_iot_deb-x64
- tests_rpm-arm64-py3
- tests_rpm-x64-py3
- tests_windows-x64
If you modified Go files and expected unit tests to run in these jobs, please double check the job logs. If you think tests should have been executed reach out to #agent-devx-help
:steam_locomotive: MergeQueue: pull request added to the queue
The median merge time in main is 22m.
Use /merge -c to cancel this operation!