datadog-agent
datadog-agent copied to clipboard
Published
20 hours ago •
DataDog
DataDog
[BARX-645] tasks: remove --python-runtimes parameters
What does this PR do?
Remove --python-runtimes invoke tasks parameters.
Motivation
Dropping python2 support. There parameters are now never passed by the CI, and never read by the build system
Describe how to test/QA your changes
Possible Drawbacks / Trade-offs
Additional Notes
Gitlab CI Configuration Changes
Modified Jobs
.agent_7_build
.agent_7_build:
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: base
- PYTHON_RUNTIMES: '3'
.integration_tests_windows_base
.integration_tests_windows_base:
before_script:
- $tmpfile = [System.IO.Path]::GetTempFileName()
- (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
-tempFile "$tmpfile")
- If ($lastExitCode -ne "0") { exit "$lastExitCode" }
- $vcpkgBlobSaSUrl=$(cat "$tmpfile")
- Remove-Item "$tmpfile"
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
-e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+ -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
- -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
- -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\integrationtests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: integration_test
tags:
- runner:windows-docker
- windowsversion:1809
.lint_macos_gitlab
.lint_macos_gitlab:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
- variables:
- PYTHON_RUNTIMES: '3'
.lint_windows_base
.lint_windows_base:
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
? ----------------------------------
+ -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
c:\mnt\tasks\winbuildscripts\lint.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: lint
tags:
- runner:windows-docker
- windowsversion:1809
.macos_gitlab
.macos_gitlab:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
- variables:
- PYTHON_RUNTIMES: '3'
.ot_agent_7_build
.ot_agent_7_build:
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: ot
- PYTHON_RUNTIMES: '3'
.tests_macos_gitlab
.tests_macos_gitlab:
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit: '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
.tests_windows_base
.tests_windows_base:
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
junit: '**/junit-out-*.xml'
when: always
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- $FAST_TESTS_FLAG=""
- If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
-e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
-e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
-e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
- -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
- -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
- PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
- EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+ -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+ -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+ TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
- -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
? ----------------------------------
+ -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
-e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\unittests.bat
'
- If ($lastExitCode -ne "0") { exit "$lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
TEST_OUTPUT_FILE: test_output.json
.tests_windows_secagent
.tests_windows_secagent:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\secagent.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
.tests_windows_sysprobe
.tests_windows_sysprobe:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\sysprobe.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
.windows_main_agent_base
.windows_main_agent_base:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
needs:
- go_mod_tidy_check
- go_deps
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
variables:
OMNIBUS_TARGET: main
.windows_msi_base
.windows_msi_base:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
needs:
- go_mod_tidy_check
- go_deps
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
agent_dmg-x64-a7
agent_dmg-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: main
paths:
- comp/core/gui/guiimpl/systray/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- changes:
compare_to: main
paths:
- omnibus/**/*
- .gitlab-ci.yml
- release.json
- .gitlab/package_build/**/*
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- mkdir -p $OMNIBUS_PACKAGE_DIR
- "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 1\"\nelse\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 2\"\nfi\n"
- $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
- export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
- python3 -m pip install -r tasks/libs/requirements-github.txt
- inv -e github.trigger-macos --workflow-type "build" --datadog-agent-ref "$CI_COMMIT_SHA"
--release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION" --destination
"$OMNIBUS_PACKAGE_DIR" --version-cache "$VERSION_CACHE_CONTENT" --integrations-core-ref
"$INTEGRATIONS_CORE_VERSION"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
timeout: 6h
variables:
AGENT_MAJOR_VERSION: 7
- PYTHON_RUNTIMES: '3'
agent_heroku_deb-x64-a7
agent_heroku_deb-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to build for $RELEASE_VERSION"
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor heroku
- ls -la $OMNIBUS_PACKAGE_DIR
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
DD_PKG_ARCH: x86_64
DESTINATION_DBG_DEB: datadog-heroku-agent-dbg_7_amd64.deb
DESTINATION_DEB: datadog-heroku-agent_7_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
RELEASE_VERSION: $RELEASE_VERSION_7
datadog-agent-7-arm64
datadog-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
--install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-agent-7-x64
datadog-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
--install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-agent-7-x64-custom-path-test
datadog-agent-7-x64-custom-path-test:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir /custom
- export CONFIG_DIR="/custom"
- export INSTALL_DIR="/custom/datadog-agent"
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
--install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
- ls -la $OMNIBUS_PACKAGE_DIR
- ls -la $INSTALL_DIR
- ls -la /custom/etc
- (ls -la /opt/datadog-agent 2>/dev/null && exit 1) || echo "/opt/datadog-agent
has correctly not been generated"
- (ls -la /etc/datadog-agent 2>/dev/null && exit 1) || echo "/etc/datadog-agent
has correctly not been generated"
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-agent-oci-arm64-a7
datadog-agent-oci-arm64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
- export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
DESTINATION_OCI: datadog-agent-7-remote-updater-arm64.tar.xz
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-agent-oci-x64-a7
datadog-agent-oci-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- AGENT_VERSION="$(inv agent.version -u)-1" || exit $?
- export INSTALL_DIR=/opt/datadog-packages/datadog-agent/"$AGENT_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --host-distribution=ociru --install-directory="$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
DESTINATION_OCI: datadog-agent-7-remote-updater-amd64.tar.xz
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-arm64
datadog-ot-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-23-arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
--install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: aarch64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/aarch64-unknown-linux-gnu.toolchain.cmake
DD_CXX: aarch64-unknown-linux-gnu-g++
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
- PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-x64
datadog-ot-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: registry.ddbuild.io/ci/datadog-agent-buildimages/linux-glibc-2-17-x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
--base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING} --skip-deps --go-mod-cache="$GOPATH/pkg/mod"
--system-probe-bin=/tmp/system-probe --flavor "$FLAVOR" --config-directory "$CONFIG_DIR"
--install-directory "$INSTALL_DIR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_CC: x86_64-unknown-linux-gnu-gcc
DD_CMAKE_TOOLCHAIN: /opt/cmake/x86_64-unknown-linux-gnu.toolchain.cmake
DD_CXX: x86_64-unknown-linux-gnu-g++
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
- PYTHON_RUNTIMES: '3'
integration_tests_windows-x64
integration_tests_windows-x64:
before_script:
- $tmpfile = [System.IO.Path]::GetTempFileName()
- (& "$CI_PROJECT_DIR\tools\ci\fetch_secret.ps1" -parameterName "$Env:VCPKG_BLOB_SAS_URL"
-tempFile "$tmpfile")
- If ($lastExitCode -ne "0") { exit "$lastExitCode" }
- $vcpkgBlobSaSUrl=$(cat "$tmpfile")
- Remove-Item "$tmpfile"
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e CI_JOB_URL="${CI_JOB_URL}"
-e CI_JOB_NAME="${CI_JOB_NAME}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PROJECT_NAME="${CI_PROJECT_NAME}"
+ -e AWS_NETWORKING=true -e GOMODCACHE="c:\modcache" -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}"
- -e AWS_NETWORKING=true -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache"
- -e VCPKG_BINARY_SOURCES="clear;x-azblob,${vcpkgBlobSaSUrl}" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\integrationtests.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: integration_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
lint_macos_gitlab_amd64
lint_macos_gitlab_amd64:
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
tags:
- macos:monterey-amd64
- specific:true
- variables:
- PYTHON_RUNTIMES: '3'
lint_macos_gitlab_arm64
lint_macos_gitlab_arm64:
allow_failure: true
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e linter.go --cpus 12 --debug --timeout 60
stage: lint
tags:
- macos:monterey-arm64
- specific:true
- variables:
- PYTHON_RUNTIMES: '3'
lint_windows-x64
lint_windows-x64:
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e AWS_NETWORKING=true -e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME}
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
? ----------------------------------
+ -e GOMODCACHE="c:\modcache" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:$Env:DATADOG_AGENT_WINBUILDIMAGES
c:\mnt\tasks\winbuildscripts\lint.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: lint
tags:
- runner:windows-docker
- windowsversion:1809
timeout: 1h30m
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
tests_deb-arm64-py3
tests_deb-arm64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:arm64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_deb-x64-py3
tests_deb-x64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_dogstatsd_deb-x64
tests_flavor_dogstatsd_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor dogstatsd
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_heroku_deb-x64
tests_flavor_heroku_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor heroku
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_flavor_iot_deb-x64
tests_flavor_iot_deb-x64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor iot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_macos
tests_macos:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh "junit-*-repacked.tgz"
artifacts:
expire_in: 2 weeks
paths:
- test_output.json
- junit-*-repacked.tgz
reports:
junit:
- '**/junit-out-*.xml'
when: always
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- setup_agent_version
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
- if: $CI_COMMIT_BRANCH == "main" || $DEPLOY_AGENT == "true" || $RUN_ALL_BUILDS
== "true" || $DDR_WORKFLOW_ID != null
- if: $RUN_UNIT_TESTS == "on"
- changes:
compare_to: main
paths:
- pkg/fleet/**/*
variables:
FAST_TESTS: 'true'
- allow_failure: true
when: manual
script:
- "if [[ \"$(( RANDOM % 2 ))\" == \"1\" ]]; then\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_1 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 1\"\nelse\n GITHUB_KEY_B64=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 key_b64) || exit $?; export GITHUB_KEY_B64\n GITHUB_APP_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 app_id) || exit $?; export GITHUB_APP_ID\n GITHUB_INSTALLATION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh\
\ $MACOS_GITHUB_APP_2 installation_id) || exit $?; export GITHUB_INSTALLATION_ID\n\
\ echo \"Using GitHub App instance 2\"\nfi\n"
- $S3_CP_CMD $S3_ARTIFACTS_URI/agent-version.cache .
- export VERSION_CACHE_CONTENT=$(cat agent-version.cache | base64 -)
- python3 -m pip install -r tasks/libs/requirements-github.txt
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" = "true" ]]; then FAST_TESTS_FLAG="--fast-tests true"; fi
- inv -e github.trigger-macos --workflow-type "test" --datadog-agent-ref "$CI_COMMIT_SHA"
--version-cache "$VERSION_CACHE_CONTENT" $FAST_TESTS_FLAG --test-washer $COVERAGE_CACHE_FLAG
stage: source_test
tags:
- arch:amd64
timeout: 6h
- variables:
- PYTHON_RUNTIMES: '3'
tests_macos_gitlab_amd64
tests_macos_gitlab_amd64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
tags:
- macos:monterey-amd64
- specific:true
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_macos_gitlab_arm64
tests_macos_gitlab_arm64:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
allow_failure: true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- 'eval $(gimme $(cat .go-version))
export GOPATH=$GOROOT
echo "Don''t forget to regularly delete unused versions. Here are the installed
versions and their memory usage on the runner:"
du -sh $HOME/.gimme/versions/*
'
- "PYTHON_REPO_VERSION=$(cat .python-version)\nPYTHON_VERSION=$(python3 --version\
\ | awk '{print $2}' | sed 's/\\.[0-9]*$//')\nif [ \"$PYTHON_REPO_VERSION\" !=\
\ \"$PYTHON_VERSION\" ]; then\n echo \"Python version $PYTHON_VERSION is different\
\ from $PYTHON_REPO_VERSION in .python-version. Cleaning the environment.\"\n\
\ pyenv uninstall -f datadog-agent\n echo \"Installing Python $PYTHON_REPO_VERSION...\"\
\n pyenv virtualenv 3.11.8 datadog-agent\n pyenv activate datadog-agent\nelse\n\
\ echo \"Python current version $PYTHON_VERSION is the same as .python-version.\
\ Keeping the existing environment.\"\nfi\n"
- python3 -m pip install -r requirements.txt -r tasks/libs/requirements-github.txt
- pyenv rehash
- inv -e rtloader.make
- inv -e rtloader.install
- inv -e install-tools
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e test --rerun-fails=2 --race --profile --cpus 12 --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" $FAST_TESTS_FLAG --test-washer
- inv -e invoke-unit-tests
stage: source_test
tags:
- macos:monterey-arm64
- specific:true
variables:
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_rpm-arm64-py3
tests_rpm-arm64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:arm64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_rpm-x64-py3
tests_rpm-x64-py3:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
- CODECOV_TOKEN=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $CODECOV_TOKEN) || exit
$?; export CODECOV_TOKEN
- inv -e coverage.upload-to-codecov $COVERAGE_CACHE_FLAG || true
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
junit:
- '**/junit-out-*.xml'
when: always
before_script:
- source /root/.bashrc && conda activate $CONDA_ENV
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- inv -e rtloader.make --install-prefix=$CI_PROJECT_DIR/dev
- inv -e rtloader.install
- inv -e rtloader.format --raise-if-changed
- inv -e rtloader.test
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
- inv -e install-tools
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
- FAST_TESTS_FLAG=""
- if [[ "$FAST_TESTS" == "true" ]]; then FAST_TESTS_FLAG="--only-impacted-packages";
fi
- inv -e sds.build-library
- inv -e agent.build
- inv -e test $FLAVORS --include-sds --race --profile --rerun-fails=2 --coverage
--cpus $KUBERNETES_CPU_REQUEST $EXTRA_OPTS --save-result-json $TEST_OUTPUT_FILE
--junit-tar "junit-${CI_JOB_NAME}.tgz" --build-stdlib $FAST_TESTS_FLAG --test-washer
stage: source_test
tags:
- arch:amd64
variables:
CONDA_ENV: ddpy3
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
EXTRA_OPTS: --build-exclude=systemd
FLAVORS: --flavor base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
- PYTHON_RUNTIMES: '3'
TEST_OUTPUT_FILE: test_output.json
tests_windows-x64
tests_windows-x64:
artifacts:
expire_in: 2 weeks
paths:
- $TEST_OUTPUT_FILE
- junit-*.tgz
reports:
junit:
- '**/junit-out-*.xml'
when: always
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_UNIT_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH == "main"
variables:
COVERAGE_CACHE_FLAG: --push-coverage-cache
FAST_TESTS: 'false'
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_COMMIT_TAG != null
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $CI_PIPELINE_SOURCE == "trigger" || $CI_PIPELINE_SOURCE == "pipeline"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- if: $RUN_UNIT_TESTS == "on"
variables:
COVERAGE_CACHE_FLAG: ''
FAST_TESTS: 'false'
- variables:
COVERAGE_CACHE_FLAG: --pull-coverage-cache
FAST_TESTS: 'true'
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- $FAST_TESTS_FLAG=""
- If ($FAST_TESTS -eq "true") { $FAST_TESTS_FLAG="--only-impacted-packages" }
- 'docker run --rm -m 16384M --storage-opt "size=50GB" -v "$(Get-Location):c:\mnt"
-e DD_ENV=prod -e GITLAB_CI=true -e CI_JOB_URL="${CI_JOB_URL}" -e CI_JOB_NAME="${CI_JOB_NAME}"
-e CI_JOB_ID="${CI_JOB_ID}" -e CI_PIPELINE_ID="${CI_PIPELINE_ID}" -e CI_PIPELINE_SOURCE="${CI_PIPELINE_SOURCE}"
-e CI_REPOSITORY_URL="${CI_REPOSITORY_URL}" -e CI_COMMIT_SHA="${CI_COMMIT_SHA}"
-e CI_PROJECT_NAME="${CI_PROJECT_NAME}" -e CI_COMMIT_REF_NAME="${CI_COMMIT_REF_NAME}"
- -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e PY_RUNTIMES="${PYTHON_RUNTIMES}"
- -e GOMODCACHE="c:\modcache" -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e
- PIP_INDEX_URL="${PIP_INDEX_URL}" -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e
- EXTRA_OPTS="${FAST_TESTS_FLAG}" -e TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}"
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e GOMODCACHE="c:\modcache"
+ -e JUNIT_TAR="c:\mnt\junit-${CI_JOB_NAME}.tgz" -e PIP_INDEX_URL="${PIP_INDEX_URL}"
+ -e TEST_OUTPUT_FILE="${TEST_OUTPUT_FILE}" -e EXTRA_OPTS="${FAST_TESTS_FLAG}" -e
+ TEST_WASHER=true -e GO_TEST_SKIP_FLAKE="${GO_TEST_SKIP_FLAKE}" -e API_KEY_ORG2="${API_KEY_ORG2}"
- -e API_KEY_ORG2="${API_KEY_ORG2}" -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
? ----------------------------------
+ -e CODECOV_TOKEN="${CODECOV_TOKEN}" -e S3_PERMANENT_ARTIFACTS_URI="${S3_PERMANENT_ARTIFACTS_URI}"
-e COVERAGE_CACHE_FLAG="${COVERAGE_CACHE_FLAG}" 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\unittests.bat
'
- If ($lastExitCode -ne "0") { exit "$lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
TEST_OUTPUT_FILE: test_output.json
tests_windows_secagent_x64
tests_windows_secagent_x64:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\secagent.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
tests_windows_sysprobe_x64
tests_windows_sysprobe_x64:
artifacts:
paths:
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-system-probe-check/files
when: always
needs:
- go_deps
- go_tools_deps
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- 'docker run --rm -m 16384M -v "$(Get-Location):c:\mnt" -e AWS_NETWORKING=true
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e SIGN_WINDOWS_DD_WCS=true
- -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL}
- 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e GOMODCACHE="c:\modcache" -e PIP_INDEX_URL=${PIP_INDEX_URL} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\sysprobe.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: source_test
tags:
- runner:windows-docker
- windowsversion:1809
variables:
ARCH: x64
- PYTHON_RUNTIMES: 3
windows_msi_and_bosh_zip_x64-a7
windows_msi_and_bosh_zip_x64-a7:
after_script:
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
before_script:
- set RELEASE_VERSION $RELEASE_VERSION_7
needs:
- go_mod_tidy_check
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = 'Stop'
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_JOB_ID=${CI_JOB_ID}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG} -e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME}
-e OMNIBUS_TARGET=${OMNIBUS_TARGET} -e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION"
- -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
? ----------------------------------
+ -e MAJOR_VERSION="$AGENT_MAJOR_VERSION" -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION"
-e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
-e TARGET_ARCH="$ARCH" -e DEBUG_CUSTOMACTION="$DEBUG_CUSTOMACTION" -e BUCKET_BRANCH="$BUCKET_BRANCH"
-e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
-e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e GO_VERSION_CHECK="true"
-e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
-e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
timeout: 2h
variables:
AGENT_MAJOR_VERSION: 7
ARCH: x64
OMNIBUS_TARGET: main
- PYTHON_RUNTIMES: '3'
windows_zip_agent_binaries_x64-a7
windows_zip_agent_binaries_x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- omnibus/pkg
before_script:
- set RELEASE_VERSION $RELEASE_VERSION_7
needs:
- go_mod_tidy_check
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- $ErrorActionPreference = "Stop"
- $_instance_id = (iwr -UseBasicParsing http://169.254.169.254/latest/meta-data/instance-id).content
; Write-Host "Running on instance $($_instance_id)"
- if (Test-Path omnibus\pkg) { remove-item -recurse -force omnibus\pkg }
- mkdir omnibus\pkg
- 'docker run --rm -m 8192M -v "$(Get-Location):c:\mnt" -e CI_COMMIT_BRANCH=${CI_COMMIT_BRANCH}
-e CI_PIPELINE_ID=${CI_PIPELINE_ID} -e CI_PROJECT_NAME=${CI_PROJECT_NAME} -e CI_JOB_NAME_SLUG=${CI_JOB_NAME_SLUG}
-e CI_COMMIT_REF_NAME=${CI_COMMIT_REF_NAME} -e OMNIBUS_TARGET=${OMNIBUS_TARGET}
-e WINDOWS_BUILDER=true -e RELEASE_VERSION="$RELEASE_VERSION" -e MAJOR_VERSION="$AGENT_MAJOR_VERSION"
- -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e PY_RUNTIMES="$PYTHON_RUNTIMES"
? ^^^^^^^^ - ^^^^^^^^^^^^^^^^
+ -e INTEGRATIONS_CORE_VERSION="$INTEGRATIONS_CORE_VERSION" -e GOMODCACHE="c:\modcache"
? ^^ ++++++ ^^^^^^^^^^^
+ -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true -e BUCKET_BRANCH="$BUCKET_BRANCH"
+ -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET" -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET"
+ -e USE_S3_CACHING="$USE_S3_CACHING" -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG}
- -e GOMODCACHE="c:\modcache" -e AWS_NETWORKING=true -e SIGN_WINDOWS_DD_WCS=true
- -e BUCKET_BRANCH="$BUCKET_BRANCH" -e INTEGRATION_WHEELS_CACHE_BUCKET="$INTEGRATION_WHEELS_CACHE_BUCKET"
- -e S3_OMNIBUS_CACHE_BUCKET="$S3_OMNIBUS_CACHE_BUCKET" -e USE_S3_CACHING="$USE_S3_CACHING"
- -e BUNDLE_MIRROR__RUBYGEMS__ORG=${BUNDLE_MIRROR__RUBYGEMS__ORG} -e PIP_INDEX_URL=${PIP_INDEX_URL}
- -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
+ -e PIP_INDEX_URL=${PIP_INDEX_URL} -e API_KEY_ORG2=${API_KEY_ORG2} 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/windows_1809_${ARCH}${Env:DATADOG_AGENT_WINBUILDIMAGES_SUFFIX}:${Env:DATADOG_AGENT_WINBUILDIMAGES}
? ++++++++++++++++++++++++++++++++++
c:\mnt\tasks\winbuildscripts\buildwin.bat
'
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
- get-childitem omnibus\pkg
- Invoke-Expression "$S3_CP_CMD omnibus\pkg\version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json"
- If ($lastExitCode -ne "0") { throw "Previous command returned $lastExitCode" }
stage: package_build
tags:
- runner:windows-docker
- windowsversion:1809
variables:
AGENT_MAJOR_VERSION: 7
ARCH: x64
OMNIBUS_TARGET: agent_binaries
Changes Summary
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 40 | 0 | 0 |
:information_source: Diff available in the job log.
![agent-platform-auto-pr[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}
Test changes on VM
Use this command from test-infra-definitions to manually test this PR changes on a VM:
inv create-vm --pipeline-id=46586140 --os-family=ubuntu
Note: This applies to commit e77ed0ac
Regression Detector
Regression Detector Results
Run ID: afa508f2-b1ff-4fc4-a0ed-d687929b9dbf Metrics dashboard Target profiles
Baseline: 5bb432c5c9e121bdf9b2e22a335b9112614406e5 Comparison: 0b65893809b808102a51387279685004d2a0e00f
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
No significant changes in experiment optimization goals
Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | +1.01 | [+0.29, +1.72] | 1 | Logs |
| ➖ | pycheck_lots_of_tags | % cpu utilization | +0.31 | [-2.17, +2.80] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | +0.31 | [+0.25, +0.37] | 1 | Logs |
| ➖ | quality_gate_idle | memory utilization | +0.29 | [+0.25, +0.34] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_300ms_latency | egress throughput | +0.12 | [-0.06, +0.29] | 1 | Logs |
| ➖ | file_to_blackhole_500ms_latency | egress throughput | +0.11 | [-0.14, +0.35] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | +0.01 | [-0.09, +0.11] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.01, +0.01] | 1 | Logs |
| ➖ | basic_py_check | % cpu utilization | -0.00 | [-2.71, +2.70] | 1 | Logs |
| ➖ | file_tree | memory utilization | -0.00 | [-0.13, +0.13] | 1 | Logs |
| ➖ | file_to_blackhole_100ms_latency | egress throughput | -0.00 | [-0.23, +0.22] | 1 | Logs |
| ➖ | file_to_blackhole_0ms_latency | egress throughput | -0.01 | [-0.34, +0.33] | 1 | Logs |
| ➖ | idle | memory utilization | -0.03 | [-0.08, +0.01] | 1 | Logs bounds checks dashboard |
| ➖ | quality_gate_idle_all_features | memory utilization | -0.12 | [-0.22, -0.03] | 1 | Logs bounds checks dashboard |
| ➖ | idle_all_features | memory utilization | -0.36 | [-0.46, -0.25] | 1 | Logs bounds checks dashboard |
| ➖ | file_to_blackhole_1000ms_latency | egress throughput | -0.41 | [-0.89, +0.07] | 1 | Logs |
| ➖ | otel_to_otel_logs | ingress throughput | -0.79 | [-1.60, +0.01] | 1 | Logs |
Bounds Checks
| perf | experiment | bounds_check_name | replicates_passed |
|---|---|---|---|
| ✅ | file_to_blackhole_0ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_1000ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_100ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_300ms_latency | memory_usage | 10/10 |
| ✅ | file_to_blackhole_500ms_latency | memory_usage | 10/10 |
| ✅ | idle | memory_usage | 10/10 |
| ✅ | idle_all_features | memory_usage | 10/10 |
| ✅ | quality_gate_idle | memory_usage | 10/10 |
| ✅ | quality_gate_idle_all_features | memory_usage | 10/10 |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
![cit-pr-commenter[bot] avatar](https://avatars.githubusercontent.com/in/330445?v=4 "Published by a pub.dev verified publisher"){kind=small}
/trigger-ci --variable RUN_ALL_BUILDS=true --variable RUN_KITCHEN_TESTS=true --variable RUN_E2E_TESTS=on --variable RUN_UNIT_TESTS=on --variable RUN_KMT_TESTS=on
:steam_locomotive: MergeQueue: pull request added to the queue
The median merge time in main is 22m.
Use /merge -c to cancel this operation!