datadog-agent
datadog-agent copied to clipboard
Published
20 hours ago •
DataDog
DataDog
Add fips mode to the Agent
WIP, still a rough draft because I have some questions
Questions
- Do we want to run a nightly build + tests with upstream GO. If yes, how can we do that
- Do we need some naming convention to differentiate package (ms go vs upstream go) ?
- How can we cache openssl omnibus software ?
- Should we mention the toolchain somewhere in the package or in the
agent versionsubcommand ? - Should we test most of the code with FIPS enabled or only a small subset and consider FIPS disabled as the main mode
- Who should own what ?
- Do we need to add anything on the delivery side before merging this PR or is the current pipeline enough ?
TODOs
- Windows
- Option to crash if FIPS is not enabled
- Add e2e tests at the end
Regression Detector
Regression Detector Results
Run ID: 8f8f2793-eb24-48cb-b96a-4e5386b84d52 Metrics dashboard Target profiles
Baseline: 22686e5ddf11ef9364a7c9b46a0855967cc9e106 Comparison: 98faafe8103e5395197f5ca4ccb5ac01fb5960d3
Performance changes are noted in the perf column of each table:
- ✅ = significantly better comparison variant performance
- ❌ = significantly worse comparison variant performance
- ➖ = no significant change in performance
No significant changes in experiment optimization goals
Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%
There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.
Fine details of change detection per experiment
| perf | experiment | goal | Δ mean % | Δ mean % CI | trials | links |
|---|---|---|---|---|---|---|
| ➖ | file_tree | memory utilization | +4.08 | [+4.00, +4.17] | 1 | Logs |
| ➖ | idle | memory utilization | +2.96 | [+2.91, +3.01] | 1 | Logs |
| ➖ | otel_to_otel_logs | ingress throughput | +1.10 | [+0.29, +1.91] | 1 | Logs |
| ➖ | pycheck_lots_of_tags | % cpu utilization | +0.45 | [-2.03, +2.93] | 1 | Logs |
| ➖ | tcp_dd_logs_filter_exclude | ingress throughput | +0.00 | [-0.01, +0.01] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api | ingress throughput | -0.00 | [-0.09, +0.09] | 1 | Logs |
| ➖ | uds_dogstatsd_to_api_cpu | % cpu utilization | -0.03 | [-0.77, +0.72] | 1 | Logs |
| ➖ | tcp_syslog_to_blackhole | ingress throughput | -0.38 | [-0.44, -0.33] | 1 | Logs |
| ➖ | basic_py_check | % cpu utilization | -1.22 | [-3.99, +1.55] | 1 | Logs |
Bounds Checks
| perf | experiment | bounds_check_name | replicates_passed |
|---|---|---|---|
| ✅ | idle | memory_usage | 10/10 |
Explanation
A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".
For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:
-
Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
-
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
-
Its configuration does not mark it "erratic".
![pr-commenter[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}
Gitlab CI Configuration Changes
Modified Jobs
stages (configuration)
stages:
- .pre
- setup
- maintenance_jobs
- deps_build
- deps_fetch
- source_test
- source_test_stats
- software_composition_analysis
- binary_build
- package_deps_build
- kernel_matrix_testing_prepare
- kernel_matrix_testing_system_probe
- kernel_matrix_testing_security_agent
- kernel_matrix_testing_cleanup
- integration_test
- benchmarks
- package_build
- packaging
- pkg_metrics
- kitchen_deploy
- kitchen_testing
- container_build
- container_scan
- check_deploy
- dev_container_deploy
- deploy_containers
- deploy_packages
- deploy_cws_instrumentation
- deploy_dca
+ - check_fips_compliance
- trigger_release
- choco_build
- choco_deploy
- internal_image_deploy
- install_script_testing
- e2e_pre_test
- e2e
- e2e_k8s
- e2e_install_packages
- kitchen_cleanup
- functional_test
- functional_test_cleanup
- junit_upload
- internal_kubernetes_deploy
- post_rc_build
- check_merge
- notify
- .post
variables (configuration)
variables:
AGENT_BINARIES_DIR: bin/agent
AGENT_GITHUB_APP_ID: ci.datadog-agent.platform-github-app-id
AGENT_GITHUB_INSTALLATION_ID: ci.datadog-agent.platform-github-app-installation-id
AGENT_GITHUB_KEY: ci.datadog-agent.platform-github-app-key
AGENT_QA_PROFILE: ci.datadog-agent.agent-qa-profile
API_KEY_DDDEV: ci.datadog-agent.datadog_api_key
API_KEY_ORG2: ci.datadog-agent.datadog_api_key_org2
APP_KEY_ORG2: ci.datadog-agent.datadog_app_key_org2
ARTIFACT_DOWNLOAD_ATTEMPTS: 2
BTFHUB_ARCHIVE_BRANCH: main
BUCKET_BRANCH: dev
CHANGELOG_COMMIT_SHA: ci.datadog-agent.gitlab_changelog_commit_sha
CHOCOLATEY_API_KEY: ci.datadog-agent.chocolatey_api_key
CLANG_LLVM_VER: 12.0.1
CLUSTER_AGENT_BINARIES_DIR: bin/datadog-cluster-agent
CLUSTER_AGENT_CLOUDFOUNDRY_BINARIES_DIR: bin/datadog-cluster-agent-cloudfoundry
CODECOV_TOKEN: ci.datadog-agent.codecov_token
CWS_INSTRUMENTATION_BINARIES_DIR: bin/cws-instrumentation
- DATADOG_AGENT_ARMBUILDIMAGES: v44534774-f5cc3e24
+ DATADOG_AGENT_ARMBUILDIMAGES: v45066542-a714c2f1
- DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX: ''
? ^^
+ DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX: _test_only
? ^^^^^^^^^^
- DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v44534774-f5cc3e24
? ^^ ^^^^ ^^^^^^^
+ DATADOG_AGENT_BTF_GEN_BUILDIMAGES: v45066542-a714c2f1
? +++++ ^^^^^ ^^ ^
- DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX: ''
? ^^
+ DATADOG_AGENT_BTF_GEN_BUILDIMAGES_SUFFIX: _test_only
? ^^^^^^^^^^
- DATADOG_AGENT_BUILDIMAGES: v44534774-f5cc3e24
+ DATADOG_AGENT_BUILDIMAGES: v45066542-a714c2f1
- DATADOG_AGENT_BUILDIMAGES_SUFFIX: ''
? ^^
+ DATADOG_AGENT_BUILDIMAGES_SUFFIX: _test_only
? ^^^^^^^^^^
DATADOG_AGENT_EMBEDDED_PATH: /opt/datadog-agent/embedded
- DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v44534774-f5cc3e24
? ^^ ^^^^ ^^^^^^^
+ DATADOG_AGENT_SYSPROBE_BUILDIMAGES: v45066542-a714c2f1
? +++++ ^^^^^ ^^ ^
- DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX: ''
? ^^
+ DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX: _test_only
? ^^^^^^^^^^
- DATADOG_AGENT_WINBUILDIMAGES: v44534774-f5cc3e24
+ DATADOG_AGENT_WINBUILDIMAGES: v45066542-a714c2f1
- DATADOG_AGENT_WINBUILDIMAGES_SUFFIX: ''
? ^^
+ DATADOG_AGENT_WINBUILDIMAGES_SUFFIX: _test_only
? ^^^^^^^^^^
DD_AGENT_TESTING_DIR: $CI_PROJECT_DIR/test/kitchen
DD_PKG_VERSION: latest
DEB_GPG_KEY: ci.datadog-agent.deb_signing_private_key_${DEB_GPG_KEY_ID}
DEB_GPG_KEY_ID: c0962c7d
DEB_GPG_KEY_NAME: Datadog, Inc. APT key
DEB_RPM_TESTING_BUCKET_BRANCH: testing
DEB_S3_BUCKET: apt.datad0g.com
DEB_SIGNING_PASSPHRASE: ci.datadog-agent.deb_signing_key_passphrase_${DEB_GPG_KEY_ID}
DEB_TESTING_S3_BUCKET: apttesting.datad0g.com
DOCKER_REGISTRY_LOGIN: ci.datadog-agent.docker_hub_login
DOCKER_REGISTRY_PWD: ci.datadog-agent.docker_hub_pwd
DOCKER_REGISTRY_URL: docker.io
DOGSTATSD_BINARIES_DIR: bin/dogstatsd
E2E_TESTS_API_KEY: ci.datadog-agent.e2e_tests_api_key
E2E_TESTS_APP_KEY: ci.datadog-agent.e2e_tests_app_key
E2E_TESTS_AZURE_CLIENT_ID: ci.datadog-agent.e2e_tests_azure_client_id
E2E_TESTS_AZURE_CLIENT_SECRET: ci.datadog-agent.e2e_tests_azure_client_secret
E2E_TESTS_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.e2e_tests_azure_subscription_id
E2E_TESTS_AZURE_TENANT_ID: ci.datadog-agent.e2e_tests_azure_tenant_id
E2E_TESTS_GCP_CREDENTIALS: ci.datadog-agent.e2e_tests_gcp_credentials
E2E_TESTS_RC_KEY: ci.datadog-agent.e2e_tests_rc_key
EXECUTOR_JOB_SECTION_ATTEMPTS: 2
FF_KUBERNETES_HONOR_ENTRYPOINT: true
FF_SCRIPT_SECTIONS: 1
GENERAL_ARTIFACTS_CACHE_BUCKET_URL: https://dd-agent-omnibus.s3.amazonaws.com
GET_SOURCES_ATTEMPTS: 2
GITHUB_PR_COMMENTER_APP_KEY: pr-commenter.github_app_key
GITHUB_PR_COMMENTER_INSTALLATION_ID: pr-commenter.github_installation_id
GITHUB_PR_COMMENTER_INTEGRATION_ID: pr-commenter.github_integration_id
GITLAB_FULL_API_TOKEN: ci.datadog-agent.gitlab_full_api_token
GITLAB_READ_API_TOKEN: ci.datadog-agent.gitlab_read_api_token
GITLAB_SCHEDULER_TOKEN: ci.datadog-agent.gitlab_pipelines_scheduler_token
GO_TEST_SKIP_FLAKE: 'true'
INSTALL_SCRIPT_API_KEY: ci.agent-linux-install-script.datadog_api_key_2
INTEGRATION_WHEELS_CACHE_BUCKET: dd-agent-omnibus
JIRA_READ_API_TOKEN: ci.datadog-agent.jira_read_api_token
KERNEL_MATRIX_TESTING_ARM_AMI_ID: ami-021f04c00ecfa8590
KERNEL_MATRIX_TESTING_X86_AMI_ID: ami-0c54d42f8f4180b0c
KITCHEN_AZURE_CLIENT_ID: ci.datadog-agent.azure_kitchen_client_id
KITCHEN_AZURE_CLIENT_SECRET: ci.datadog-agent.azure_kitchen_client_secret
KITCHEN_AZURE_SUBSCRIPTION_ID: ci.datadog-agent.azure_kitchen_subscription_id
KITCHEN_AZURE_TENANT_ID: ci.datadog-agent.azure_kitchen_tenant_id
KITCHEN_EC2_SSH_KEY: ci.datadog-agent.aws_ec2_kitchen_ssh_key
KITCHEN_INFRASTRUCTURE_FLAKES_RETRY: 2
MACOS_GITHUB_APP_ID: ci.datadog-agent.macos_github_app_id
MACOS_GITHUB_APP_ID_2: ci.datadog-agent.macos_github_app_id_2
MACOS_GITHUB_INSTALLATION_ID: ci.datadog-agent.macos_github_installation_id
MACOS_GITHUB_INSTALLATION_ID_2: ci.datadog-agent.macos_github_installation_id_2
MACOS_GITHUB_KEY: ci.datadog-agent.macos_github_key_b64
MACOS_GITHUB_KEY_2: ci.datadog-agent.macos_github_key_b64_2
MACOS_S3_BUCKET: dd-agent-macostesting
OMNIBUS_BASE_DIR: /omnibus
OMNIBUS_GIT_CACHE_DIR: /tmp/omnibus-git-cache
OMNIBUS_PACKAGE_DIR: $CI_PROJECT_DIR/omnibus/pkg/
OMNIBUS_PACKAGE_DIR_SUSE: $CI_PROJECT_DIR/omnibus/suse/pkg
PROCESS_S3_BUCKET: datad0g-process-agent
RELEASE_VERSION_6: nightly
RELEASE_VERSION_7: nightly-a7
RESTORE_CACHE_ATTEMPTS: 2
RPM_GPG_KEY: ci.datadog-agent.rpm_signing_private_key_${RPM_GPG_KEY_ID}
RPM_GPG_KEY_ID: b01082d3
RPM_GPG_KEY_NAME: Datadog, Inc. RPM key
RPM_S3_BUCKET: yum.datad0g.com
RPM_SIGNING_PASSPHRASE: ci.datadog-agent.rpm_signing_key_passphrase_${RPM_GPG_KEY_ID}
RPM_TESTING_S3_BUCKET: yumtesting.datad0g.com
RUN_E2E_TESTS: auto
RUN_KMT_TESTS: auto
RUN_UNIT_TESTS: auto
S3_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME/$CI_PIPELINE_ID
S3_CP_CMD: aws s3 cp $S3_CP_OPTIONS
S3_CP_OPTIONS: --no-progress --region us-east-1 --sse AES256
S3_DD_AGENT_OMNIBUS_BTFS_URI: s3://dd-agent-omnibus/btfs
S3_DD_AGENT_OMNIBUS_LLVM_URI: s3://dd-agent-omnibus/llvm
S3_DSD6_URI: s3://dsd6-staging
S3_OMNIBUS_CACHE_BUCKET: dd-ci-datadog-agent-omnibus-cache-build-stable
S3_PERMANENT_ARTIFACTS_URI: s3://dd-ci-persistent-artefacts-build-stable/$CI_PROJECT_NAME
S3_PROJECT_ARTIFACTS_URI: s3://dd-ci-artefacts-build-stable/$CI_PROJECT_NAME
S3_RELEASE_ARTIFACTS_URI: s3://dd-release-artifacts/$CI_PROJECT_NAME/$CI_PIPELINE_ID
S3_RELEASE_INSTALLER_ARTIFACTS_URI: s3://dd-release-artifacts/datadog-installer/$CI_PIPELINE_ID
S3_SBOM_STORAGE_URI: s3://sbom-root-us1-ddbuild-io/$CI_PROJECT_NAME/$CI_PIPELINE_ID
SLACK_AGENT_CI_TOKEN: ci.datadog-agent.slack_agent_ci_token
SMP_ACCOUNT_ID: ci.datadog-agent.single-machine-performance-account-id
SMP_AGENT_TEAM_ID: ci.datadog-agent.single-machine-performance-agent-team-id
SMP_API: ci.datadog-agent.single-machine-performance-api
SMP_BOT_ACCESS_KEY: ci.datadog-agent.single-machine-performance-bot-access-key
SMP_BOT_ACCESS_KEY_ID: ci.datadog-agent.single-machine-performance-bot-access-key-id
SSH_KEY: ci.datadog-agent.ssh_key
SSH_KEY_RSA: ci.datadog-agent.ssh_key_rsa
SSH_PUBLIC_KEY_RSA: ci.datadog-agent.ssh_public_key_rsa
STATIC_BINARIES_DIR: bin/static
SYSTEM_PROBE_BINARIES_DIR: bin/system-probe
USE_S3_CACHING: --omnibus-s3-cache
VCPKG_BLOB_SAS_URL: ci.datadog-agent-buildimages.vcpkg_blob_sas_url
WINDOWS_BUILDS_S3_BUCKET: $WIN_S3_BUCKET/builds
WINDOWS_TESTING_S3_BUCKET_A6: pipelines/A6/$CI_PIPELINE_ID
WINDOWS_TESTING_S3_BUCKET_A7: pipelines/A7/$CI_PIPELINE_ID
WINGET_PAT: ci.datadog-agent.winget_pat
WIN_S3_BUCKET: dd-agent-mstesting
.agent_build_common
.agent_build_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
.kmt_setup_env
.kmt_setup_env:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- export AWS_PROFILE=agent-qa-ci
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
$CI_PROJECT_DIR/libvirt/dnsmasq
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
'{}' > /tmp/ddvm-xml-'{}'.txt\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
/tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
chown 1000:1000 /tmp/dnsmasq/*"
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
$CI_PROJECT_DIR/libvirt/log
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
$CI_PROJECT_DIR/libvirt/xml
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
$CI_PROJECT_DIR/libvirt/qemu
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
$CI_PROJECT_DIR/libvirt/dnsmasq
- "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n"
- cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
"ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
--tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
--daemon -log-file /home/ubuntu/daemon.log"
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/stack.output
- $CI_PROJECT_DIR/libvirt
- $VMCONFIG_FILE
when: always
before_script:
+ - source /root/.bashrc
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
$STACK_DIR
- pulumi login $(cat $STACK_DIR | tr -d '\n')
- inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
--vmconfig-template=$TEST_COMPONENT --memory=12288
- inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
$INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
--infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
--run-agent
- jq "." $CI_PROJECT_DIR/stack.output
- pulumi logout
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
AWS_REGION: us-east-1
INFRA_ENV: aws/agent-qa
KITCHEN_EC2_REGION: us-east-1
KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
PIPELINE_ID: $CI_PIPELINE_ID
RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
STACK_DIR: $CI_PROJECT_DIR/stack.dir
TEAM: ebpf-platform
VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json
.package_deb_common
.package_deb_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
.package_rpm_common
.package_rpm_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
.package_suse_rpm_common
.package_suse_rpm_common:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
variables:
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
.prepare_secagent_ebpf_functional_tests
.prepare_secagent_ebpf_functional_tests:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="security-agent"
- mkdir -p /opt/datadog-agent/embedded/bin
- cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
- cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
- invoke -e security-agent.kitchen-prepare --skip-linters
stage: source_test
.prepare_sysprobe_ebpf_functional_tests
.prepare_sysprobe_ebpf_functional_tests:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="system-probe"
stage: source_test
variables:
KUBERNETES_CPU_REQUEST: 4
.system-probe_build_common
.system-probe_build_common:
artifacts:
expire_in: 2 weeks
paths:
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+ - source /root/.bashrc
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv check-go-version
- inv -e system-probe.build --strip-object-files --no-bundle
- objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
&& exit 1
- inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
variables:
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 12Gi
KUBERNETES_MEMORY_REQUEST: 6Gi
.tests_linux_ebpf
.tests_linux_ebpf:
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
needs:
- go_deps
- go_tools_deps
script:
- inv -e install-tools
- inv -e system-probe.object-files
- invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
- invoke -e security-agent.run-ebpf-unit-tests --verbose
- invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
stresstests trivy containerd linux_bpf ebpf_bindata"
stage: source_test
variables:
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
.upload_secagent_tests
.upload_secagent_tests:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
allow_failure: true
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
rules:
- allow_failure: true
if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/ebpf/**/*
- pkg/security/**/*
- pkg/eventmonitor/**/*
- test/kitchen/site-cookbooks/dd-security-agent-check/**/*
- test/kitchen/test/integration/security-agent-test/**/*
- test/kitchen/test/integration/security-agent-stress/**/*
- .gitlab/functional_test/security_agent.yml
- .gitlab/kernel_matrix_testing/security_agent.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/security_agent.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
- allow_failure: true
when: manual
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
variables:
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: security-agent
.upload_sysprobe_tests
.upload_sysprobe_tests:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/collector/corechecks/ebpf/**/*
- pkg/collector/corechecks/servicediscovery/module/*
- pkg/ebpf/**/*
- pkg/network/**/*
- pkg/process/monitor/*
- pkg/util/kernel/**/*
- .gitlab/kernel_matrix_testing/system_probe.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/system_probe.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
variables:
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: system-probe
agent_deb-arm64-a6
agent_deb-arm64-a6:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- datadog-agent-6-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 6
DD_PKG_ARCH: arm64
DD_PROJECT: agent
DESTINATION_DEB: datadog-agent_6_arm64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_6
agent_deb-arm64-a7
agent_deb-arm64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- datadog-agent-7-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: agent
DESTINATION_DEB: datadog-agent_7_arm64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_7
agent_deb-x64-a6
agent_deb-x64-a6:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-6-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 6
DD_PKG_ARCH: x86_64
DD_PROJECT: agent
DESTINATION_DEB: datadog-agent_6_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_6
agent_deb-x64-a7
agent_deb-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-7-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: agent
DESTINATION_DEB: datadog-agent_7_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_7
agent_rpm-arm64-a6
agent_rpm-arm64-a6:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- datadog-agent-6-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 6
DD_PKG_ARCH: arm64
DD_PROJECT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_6
agent_rpm-arm64-a7
agent_rpm-arm64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- datadog-agent-7-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
agent_rpm-x64-a6
agent_rpm-x64-a6:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-6-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 6
DD_PKG_ARCH: x86_64
DD_PROJECT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_6
agent_rpm-x64-a7
agent_rpm-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-7-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
agent_suse-arm64-a7
agent_suse-arm64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- datadog-agent-7-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PRODUCT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
agent_suse-x64-a6
agent_suse-x64-a6:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-6-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 6
DD_PKG_ARCH: x86_64
DD_PRODUCT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_6
agent_suse-x64-a7
agent_suse-x64-a7:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- datadog-agent-7-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PRODUCT: agent
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/agent-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
build_system-probe-arm64
build_system-probe-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+ - source /root/.bashrc
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv check-go-version
- inv -e system-probe.build --strip-object-files --no-bundle
- objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
&& exit 1
- inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
stage: binary_build
tags:
- arch:arm64
variables:
ARCH: arm64
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 12Gi
KUBERNETES_MEMORY_REQUEST: 6Gi
build_system-probe-x64
build_system-probe-x64:
artifacts:
expire_in: 2 weeks
paths:
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz.sum
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -depth
- find "$CI_BUILDS_DIR" ! -path '*DataDog/datadog-agent*' -delete || true
+ - source /root/.bashrc
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv check-go-version
- inv -e system-probe.build --strip-object-files --no-bundle
- objdump -p $CI_PROJECT_DIR/$SYSTEM_PROBE_BINARIES_DIR/system-probe | egrep 'GLIBC_2\.(1[8-9]|[2-9][0-9])'
&& exit 1
- inv -e system-probe.save-build-outputs $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
stage: binary_build
tags:
- arch:amd64
variables:
ARCH: amd64
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 12Gi
KUBERNETES_MEMORY_REQUEST: 6Gi
datadog-agent-6-arm64
datadog-agent-6-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_6
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 6
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
PYTHON_RUNTIMES: '2,3'
datadog-agent-6-x64
datadog-agent-6-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_6
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 6
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
PYTHON_RUNTIMES: '2,3'
datadog-agent-7-arm64
datadog-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
PYTHON_RUNTIMES: '3'
datadog-agent-7-x64
datadog-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: base
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-arm64
datadog-ot-agent-7-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-arm64
- go_deps
- generate_minimized_btfs_arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: arm64
PYTHON_RUNTIMES: '3'
datadog-ot-agent-7-x64
datadog-ot-agent-7-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script:
- export RELEASE_VERSION=$RELEASE_VERSION_7
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- go_mod_tidy_check
- build_system-probe-x64
- go_deps
- generate_minimized_btfs_x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- rm -rf $OMNIBUS_PACKAGE_DIR/*
- tar -xf $CI_PROJECT_DIR/sysprobe-build-outputs.tar.xz
- mkdir -p /tmp/system-probe
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.${PACKAGE_ARCH} /tmp/system-probe/llc-bpf
- cp $CI_PROJECT_DIR/minimized-btfs.tar.xz /tmp/system-probe/minimized-btfs.tar.xz
- chmod 0744 /tmp/system-probe/clang-bpf /tmp/system-probe/llc-bpf
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --fips-mode --release-version "$RELEASE_VERSION" --major-version
? ++++++++++++
- --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
+ "$AGENT_MAJOR_VERSION" --python-runtimes "$PYTHON_RUNTIMES" --base-dir $OMNIBUS_BASE_DIR ${USE_S3_CACHING}
? +++++++++++++++++++++++
--skip-deps --go-mod-cache="$GOPATH/pkg/mod" --system-probe-bin=/tmp/system-probe
--flavor "$FLAVOR"
- ls -la $OMNIBUS_PACKAGE_DIR
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/version-manifest.json $S3_SBOM_STORAGE_URI/$CI_JOB_NAME/version-manifest.json
stage: package_build
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
FLAVOR: ot
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
PACKAGE_ARCH: amd64
PYTHON_RUNTIMES: '3'
dogstatsd_deb-arm64
dogstatsd_deb-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- dogstatsd-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: dogstatsd
DESTINATION_DEB: datadog-dogstatsd_arm64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_7
dogstatsd_deb-x64
dogstatsd_deb-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- dogstatsd-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: dogstatsd
DESTINATION_DEB: datadog-dogstatsd_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-deb.txt
RELEASE_VERSION: $RELEASE_VERSION_7
dogstatsd_rpm-x64
dogstatsd_rpm-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- dogstatsd-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: dogstatsd
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
dogstatsd_suse-x64
dogstatsd_suse-x64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- dogstatsd-x64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: dogstatsd
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: test/required_files/dogstatsd-rpm.txt
RELEASE_VERSION: $RELEASE_VERSION_7
installer_deb-amd64
installer_deb-amd64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- installer-amd64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: installer
DESTINATION_DEB: datadog-installer_7_amd64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
installer_deb-arm64
installer_deb-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/deb_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- installer-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- echo "About to package for $RELEASE_VERSION"
- set +x
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- DEB_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $DEB_SIGNING_PASSPHRASE)
|| exit $?; export DEB_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project
+ ${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- $S3_CP_CMD $OMNIBUS_PACKAGE_DIR/datadog-${DD_PROJECT}_*_${PACKAGE_ARCH}.deb $S3_ARTIFACTS_URI/$DESTINATION_DEB
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: installer
DESTINATION_DEB: datadog-installer_7_arm64.deb
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
installer_rpm-amd64
installer_rpm-amd64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- installer-amd64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: installer
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
installer_rpm-arm64
installer_rpm-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- installer-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: installer
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
installer_suse_rpm-amd64
installer_suse_rpm-amd64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_x64$DATADOG_AGENT_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_BUILDIMAGES
needs:
- installer-amd64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:amd64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: x86_64
DD_PROJECT: installer
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: amd64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
installer_suse_rpm-arm64
installer_suse_rpm-arm64:
artifacts:
expire_in: 2 weeks
paths:
- $OMNIBUS_PACKAGE_DIR_SUSE
before_script: null
cache:
- key:
files:
- omnibus/Gemfile
- release.json
prefix: omnibus-deps-$CI_JOB_NAME-$OMNIBUS_RUBY_VERSION-$OMNIBUS_SOFTWARE
paths:
- omnibus/vendor/bundle
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/rpm_arm64$DATADOG_AGENT_ARMBUILDIMAGES_SUFFIX:$DATADOG_AGENT_ARMBUILDIMAGES
needs:
- installer-arm64
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- echo "About to build for $RELEASE_VERSION"
- pushd omnibus && bundle config set --local path 'vendor/bundle' && popd
- printf -- "$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_GPG_KEY)" | gpg --import
--batch
- EXIT="${PIPESTATUS[0]}"; if [ $EXIT -ne 0 ]; then echo "Unable to locate credentials
needs gitlab runner restart"; exit $EXIT; fi
- RPM_SIGNING_PASSPHRASE=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $RPM_SIGNING_PASSPHRASE)
|| exit $?; export RPM_SIGNING_PASSPHRASE
- - inv -e omnibus.build --release-version "$RELEASE_VERSION" --major-version "$AGENT_MAJOR_VERSION"
? -----------------------
+ - inv -e omnibus.build --release-version "$RELEASE_VERSION" --fips-mode --major-version
? ++++++++++++
- --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT} ${OMNIBUS_EXTRA_ARGS}
? ----------------------
+ "$AGENT_MAJOR_VERSION" --base-dir $OMNIBUS_BASE_DIR --skip-deps --target-project=${DD_PROJECT}
? +++++++++++++++++++++++
+ ${OMNIBUS_EXTRA_ARGS}
- ls -la $OMNIBUS_PACKAGE_DIR/
- curl -sSL "https://dd-package-tools.s3.amazonaws.com/dd-pkg/${DD_PKG_VERSION}/dd-pkg_Linux_${DD_PKG_ARCH}.tar.gz"
| tar -xz -C /usr/local/bin dd-pkg
- find $OMNIBUS_PACKAGE_DIR -iregex '.*\.\(deb\|rpm\)' | xargs dd-pkg lint
- "if [ -n \"$PACKAGE_REQUIRED_FILES_LIST\" ]; then\n find $OMNIBUS_PACKAGE_DIR\
\ \\( -name '*.deb' -or -name '*.rpm' \\) -a -not -name '*-dbg[_-]*' | xargs dd-pkg\
\ check-files --required-files ${PACKAGE_REQUIRED_FILES_LIST}\nfi\n"
- mkdir -p $OMNIBUS_PACKAGE_DIR_SUSE && mv $OMNIBUS_PACKAGE_DIR/*.rpm $OMNIBUS_PACKAGE_DIR_SUSE/
stage: packaging
tags:
- arch:arm64
variables:
AGENT_MAJOR_VERSION: 7
DD_PKG_ARCH: arm64
DD_PROJECT: installer
KUBERNETES_CPU_REQUEST: 16
KUBERNETES_MEMORY_LIMIT: 32Gi
KUBERNETES_MEMORY_REQUEST: 32Gi
OMNIBUS_EXTRA_ARGS: --host-distribution=suse
OMNIBUS_PACKAGE_ARTIFACT_DIR: $OMNIBUS_PACKAGE_DIR
PACKAGE_ARCH: arm64
PACKAGE_REQUIRED_FILES_LIST: ''
RELEASE_VERSION: $RELEASE_VERSION_7
kmt_setup_env_secagent_arm64
kmt_setup_env_secagent_arm64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- export AWS_PROFILE=agent-qa-ci
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
$CI_PROJECT_DIR/libvirt/dnsmasq
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
'{}' > /tmp/ddvm-xml-'{}'.txt\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
/tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
chown 1000:1000 /tmp/dnsmasq/*"
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
$CI_PROJECT_DIR/libvirt/log
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
$CI_PROJECT_DIR/libvirt/xml
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
$CI_PROJECT_DIR/libvirt/qemu
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
$CI_PROJECT_DIR/libvirt/dnsmasq
- "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n"
- cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
"ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
--tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
--daemon -log-file /home/ubuntu/daemon.log"
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/stack.output
- $CI_PROJECT_DIR/libvirt
- $VMCONFIG_FILE
when: always
before_script:
+ - source /root/.bashrc
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- allow_failure: true
if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/ebpf/**/*
- pkg/security/**/*
- pkg/eventmonitor/**/*
- test/kitchen/site-cookbooks/dd-security-agent-check/**/*
- test/kitchen/test/integration/security-agent-test/**/*
- test/kitchen/test/integration/security-agent-stress/**/*
- .gitlab/functional_test/security_agent.yml
- .gitlab/kernel_matrix_testing/security_agent.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/security_agent.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
- allow_failure: true
when: manual
script:
- echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
$STACK_DIR
- pulumi login $(cat $STACK_DIR | tr -d '\n')
- inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
--vmconfig-template=$TEST_COMPONENT --memory=12288
- inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
$INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
--infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
--run-agent
- jq "." $CI_PROJECT_DIR/stack.output
- pulumi logout
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
AMI_ID_ARG: --arm-ami-id=$KERNEL_MATRIX_TESTING_ARM_AMI_ID
ARCH: arm64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
AWS_REGION: us-east-1
INFRA_ENV: aws/agent-qa
INSTANCE_TYPE: m6gd.metal
INSTANCE_TYPE_ARG: --instance-type-arm=$INSTANCE_TYPE
KITCHEN_EC2_REGION: us-east-1
KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-arm
PIPELINE_ID: $CI_PIPELINE_ID
RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
STACK_DIR: $CI_PROJECT_DIR/stack.dir
TEAM: ebpf-platform
TEST_COMPONENT: security-agent
TEST_SETS: all_tests
VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json
kmt_setup_env_secagent_x64
kmt_setup_env_secagent_x64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- export AWS_PROFILE=agent-qa-ci
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
$CI_PROJECT_DIR/libvirt/dnsmasq
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
'{}' > /tmp/ddvm-xml-'{}'.txt\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
/tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
chown 1000:1000 /tmp/dnsmasq/*"
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
$CI_PROJECT_DIR/libvirt/log
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
$CI_PROJECT_DIR/libvirt/xml
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
$CI_PROJECT_DIR/libvirt/qemu
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
$CI_PROJECT_DIR/libvirt/dnsmasq
- "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n"
- cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
"ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
--tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
--daemon -log-file /home/ubuntu/daemon.log"
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/stack.output
- $CI_PROJECT_DIR/libvirt
- $VMCONFIG_FILE
when: always
before_script:
+ - source /root/.bashrc
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- allow_failure: true
if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/ebpf/**/*
- pkg/security/**/*
- pkg/eventmonitor/**/*
- test/kitchen/site-cookbooks/dd-security-agent-check/**/*
- test/kitchen/test/integration/security-agent-test/**/*
- test/kitchen/test/integration/security-agent-stress/**/*
- .gitlab/functional_test/security_agent.yml
- .gitlab/kernel_matrix_testing/security_agent.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/security_agent.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
- allow_failure: true
when: manual
script:
- echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
$STACK_DIR
- pulumi login $(cat $STACK_DIR | tr -d '\n')
- inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
--vmconfig-template=$TEST_COMPONENT --memory=12288
- inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
$INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
--infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
--run-agent
- jq "." $CI_PROJECT_DIR/stack.output
- pulumi logout
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
AMI_ID_ARG: --x86-ami-id=$KERNEL_MATRIX_TESTING_X86_AMI_ID
ARCH: x86_64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
AWS_REGION: us-east-1
INFRA_ENV: aws/agent-qa
INSTANCE_TYPE: m5d.metal
INSTANCE_TYPE_ARG: --instance-type-x86=$INSTANCE_TYPE
KITCHEN_EC2_REGION: us-east-1
KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-x86
PIPELINE_ID: $CI_PIPELINE_ID
RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
STACK_DIR: $CI_PROJECT_DIR/stack.dir
TEAM: ebpf-platform
TEST_COMPONENT: security-agent
TEST_SETS: all_tests
VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json
kmt_setup_env_sysprobe_arm64
kmt_setup_env_sysprobe_arm64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- export AWS_PROFILE=agent-qa-ci
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
$CI_PROJECT_DIR/libvirt/dnsmasq
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
'{}' > /tmp/ddvm-xml-'{}'.txt\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
/tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
chown 1000:1000 /tmp/dnsmasq/*"
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
$CI_PROJECT_DIR/libvirt/log
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
$CI_PROJECT_DIR/libvirt/xml
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
$CI_PROJECT_DIR/libvirt/qemu
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
$CI_PROJECT_DIR/libvirt/dnsmasq
- "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n"
- cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
"ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
--tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
--daemon -log-file /home/ubuntu/daemon.log"
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/stack.output
- $CI_PROJECT_DIR/libvirt
- $VMCONFIG_FILE
when: always
before_script:
+ - source /root/.bashrc
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/collector/corechecks/ebpf/**/*
- pkg/collector/corechecks/servicediscovery/module/*
- pkg/ebpf/**/*
- pkg/network/**/*
- pkg/process/monitor/*
- pkg/util/kernel/**/*
- .gitlab/kernel_matrix_testing/system_probe.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/system_probe.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
script:
- echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
$STACK_DIR
- pulumi login $(cat $STACK_DIR | tr -d '\n')
- inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
--vmconfig-template=$TEST_COMPONENT --memory=12288
- inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
$INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
--infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
--run-agent
- jq "." $CI_PROJECT_DIR/stack.output
- pulumi logout
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
AMI_ID_ARG: --arm-ami-id=$KERNEL_MATRIX_TESTING_ARM_AMI_ID
ARCH: arm64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
AWS_REGION: us-east-1
INFRA_ENV: aws/agent-qa
INSTANCE_TYPE: m6gd.metal
INSTANCE_TYPE_ARG: --instance-type-arm=$INSTANCE_TYPE
KITCHEN_EC2_REGION: us-east-1
KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-arm
PIPELINE_ID: $CI_PIPELINE_ID
RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
STACK_DIR: $CI_PROJECT_DIR/stack.dir
TEAM: ebpf-platform
TEST_COMPONENT: system-probe
TEST_SETS: only_usm,no_usm
VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json
kmt_setup_env_sysprobe_x64
kmt_setup_env_sysprobe_x64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- export AWS_PROFILE=agent-qa-ci
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- mkdir -p $CI_PROJECT_DIR/libvirt/log/$ARCH $CI_PROJECT_DIR/libvirt/xml $CI_PROJECT_DIR/libvirt/qemu
$CI_PROJECT_DIR/libvirt/dnsmasq
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | grep -v -E '^$' | xargs -I '{}' sh -c \"sudo virsh dumpxml
'{}' > /tmp/ddvm-xml-'{}'.txt\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"sudo virsh list --name | xargs -I '{}' sh -c \"sudo cp /var/log/libvirt/qemu/'{}'.log
/tmp/qemu-ddvm-'{}'.log && sudo chown 1000:1000 /tmp/qemu-ddvm*\""
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"mkdir /tmp/dnsmasq && sudo cp /var/lib/libvirt/dnsmasq/* /tmp/dnsmasq/ && sudo
chown 1000:1000 /tmp/dnsmasq/*"
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-*.log"
$CI_PROJECT_DIR/libvirt/log
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/ddvm-xml-*"
$CI_PROJECT_DIR/libvirt/xml
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/qemu-ddvm-*.log"
$CI_PROJECT_DIR/libvirt/qemu
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP:/tmp/dnsmasq/*"
$CI_PROJECT_DIR/libvirt/dnsmasq
- "GO_ARCH=$ARCH\nif [ \"${ARCH}\" == \"x86_64\" ]; then\n GO_ARCH=amd64\nfi\n"
- cd test/new-e2e && GOOS=linux GOARCH="${GO_ARCH}" go build system-probe/vm-metrics/vm-metrics.go
- scp -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE $CI_PROJECT_DIR/test/new-e2e/vm-metrics
"ubuntu@$INSTANCE_IP:/home/ubuntu/vm-metrics"
- ssh -o StrictHostKeyChecking=no -i $AWS_EC2_SSH_KEY_FILE "ubuntu@$INSTANCE_IP"
"/home/ubuntu/vm-metrics -statsd-host=127.0.0.1 -statsd-port=8125 -libvirt-uri=/var/run/libvirt/libvirt-sock-ro
--tag \"arch:${ARCH}\" --tag \"test-component:${TEST_COMPONENT}\" --tag \"ci-pipeline-id:${CI_PIPELINE_ID}\"
--daemon -log-file /home/ubuntu/daemon.log"
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/stack.output
- $CI_PROJECT_DIR/libvirt
- $VMCONFIG_FILE
when: always
before_script:
+ - source /root/.bashrc
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/collector/corechecks/ebpf/**/*
- pkg/collector/corechecks/servicediscovery/module/*
- pkg/ebpf/**/*
- pkg/network/**/*
- pkg/process/monitor/*
- pkg/util/kernel/**/*
- .gitlab/kernel_matrix_testing/system_probe.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/system_probe.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
script:
- echo "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE" >
$STACK_DIR
- pulumi login $(cat $STACK_DIR | tr -d '\n')
- inv -e kmt.gen-config --ci --arch=$ARCH --output-file=$VMCONFIG_FILE --sets=$TEST_SETS
--vmconfig-template=$TEST_COMPONENT --memory=12288
- inv -e system-probe.start-microvms --provision-instance --provision-microvms --vmconfig=$VMCONFIG_FILE
$INSTANCE_TYPE_ARG $AMI_ID_ARG --ssh-key-name=$AWS_EC2_SSH_KEY_NAME --ssh-key-path=$AWS_EC2_SSH_KEY_FILE
--infra-env=$INFRA_ENV --stack-name=kernel-matrix-testing-${TEST_COMPONENT}-${ARCH}-${CI_PIPELINE_ID}
--run-agent
- jq "." $CI_PROJECT_DIR/stack.output
- pulumi logout
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
AMI_ID_ARG: --x86-ami-id=$KERNEL_MATRIX_TESTING_X86_AMI_ID
ARCH: x86_64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
AWS_EC2_SSH_KEY_NAME: datadog-agent-ci
AWS_REGION: us-east-1
INFRA_ENV: aws/agent-qa
INSTANCE_TYPE: m5d.metal
INSTANCE_TYPE_ARG: --instance-type-x86=$INSTANCE_TYPE
KITCHEN_EC2_REGION: us-east-1
KITCHEN_EC2_SG_IDS: sg-019917348cb0eb7e7
KITCHEN_EC2_SUBNET: subnet-05d7c6b1b5cfea811
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
LibvirtSSHKey: $CI_PROJECT_DIR/libvirt_rsa-x86
PIPELINE_ID: $CI_PIPELINE_ID
RESOURCE_TAGS: instance-type:${INSTANCE_TYPE},arch:${ARCH},test-component:${TEST_COMPONENT},git-branch:${CI_COMMIT_REF_NAME}
STACK_DIR: $CI_PROJECT_DIR/stack.dir
TEAM: ebpf-platform
TEST_COMPONENT: system-probe
TEST_SETS: only_usm,no_usm
VMCONFIG_FILE: ${CI_PROJECT_DIR}/vmconfig-${CI_PIPELINE_ID}-${ARCH}.json
prepare_secagent_ebpf_functional_tests_arm64
prepare_secagent_ebpf_functional_tests_arm64:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="security-agent"
- mkdir -p /opt/datadog-agent/embedded/bin
- cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
- cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
- invoke -e security-agent.kitchen-prepare --skip-linters
stage: source_test
tags:
- arch:arm64
variables:
ARCH: arm64
prepare_secagent_ebpf_functional_tests_x64
prepare_secagent_ebpf_functional_tests_x64:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
- $DD_AGENT_TESTING_DIR/site-cookbooks/dd-security-agent-check/files
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="security-agent"
- mkdir -p /opt/datadog-agent/embedded/bin
- cp /tmp/clang-bpf /opt/datadog-agent/embedded/bin/clang-bpf
- cp /tmp/llc-bpf /opt/datadog-agent/embedded/bin/llc-bpf
- invoke -e security-agent.kitchen-prepare --skip-linters
stage: source_test
tags:
- arch:amd64
variables:
ARCH: amd64
prepare_sysprobe_ebpf_functional_tests_arm64
prepare_sysprobe_ebpf_functional_tests_arm64:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="system-probe"
stage: source_test
tags:
- arch:arm64
variables:
ARCH: arm64
KUBERNETES_CPU_REQUEST: 4
prepare_sysprobe_ebpf_functional_tests_x64
prepare_sysprobe_ebpf_functional_tests_x64:
artifacts:
paths:
- $CI_PROJECT_DIR/kmt-deps
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- inv -e install-tools
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/bin
- mkdir -p $DATADOG_AGENT_EMBEDDED_PATH/include
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/clang-$CLANG_LLVM_VER.$ARCH /tmp/clang-bpf
- $S3_CP_CMD $S3_PERMANENT_ARTIFACTS_URI/llc-$CLANG_LLVM_VER.$ARCH /tmp/llc-bpf
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
rules:
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- when: on_success
script:
- inv -e kmt.prepare --ci --component="system-probe"
stage: source_test
tags:
- arch:amd64
variables:
ARCH: amd64
KUBERNETES_CPU_REQUEST: 4
security_go_generate_check
security_go_generate_check:
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
- pip3 install wheel
- pip3 install -r docs/cloud-workload-security/scripts/requirements-docs.txt
- inv -e install-tools
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- inv -e security-agent.go-generate-check
stage: source_test
tags:
- arch:amd64
variables:
KUBERNETES_CPU_REQUEST: 4
tests_ebpf_arm64
tests_ebpf_arm64:
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- inv -e install-tools
- inv -e system-probe.object-files
- invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
- invoke -e security-agent.run-ebpf-unit-tests --verbose
- invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
stresstests trivy containerd linux_bpf ebpf_bindata"
stage: source_test
tags:
- arch:arm64
variables:
ARCH: arm64
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
TASK_ARCH: arm64
tests_ebpf_x64
tests_ebpf_x64:
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_tools.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_tools.tar.xz
+ - source /root/.bashrc
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- go_tools_deps
script:
- inv -e install-tools
- inv -e system-probe.object-files
- invoke -e linter.go --build system-probe-unit-tests --cpus 4 --targets ./pkg
- invoke -e security-agent.run-ebpf-unit-tests --verbose
- invoke -e linter.go --targets=./pkg/security/tests --cpus 4 --build-tags="functionaltests
stresstests trivy containerd linux_bpf ebpf_bindata"
stage: source_test
tags:
- arch:amd64
variables:
ARCH: amd64
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 16Gi
TASK_ARCH: x64
upload_secagent_tests_arm64
upload_secagent_tests_arm64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
allow_failure: true
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- prepare_secagent_ebpf_functional_tests_arm64
rules:
- allow_failure: true
if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/ebpf/**/*
- pkg/security/**/*
- pkg/eventmonitor/**/*
- test/kitchen/site-cookbooks/dd-security-agent-check/**/*
- test/kitchen/test/integration/security-agent-test/**/*
- test/kitchen/test/integration/security-agent-stress/**/*
- .gitlab/functional_test/security_agent.yml
- .gitlab/kernel_matrix_testing/security_agent.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/security_agent.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
- allow_failure: true
when: manual
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
tags:
- arch:arm64
variables:
ARCH: arm64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
INSTANCE_TYPE: m6gd.metal
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: security-agent
upload_secagent_tests_x64
upload_secagent_tests_x64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
allow_failure: true
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- prepare_secagent_ebpf_functional_tests_x64
rules:
- allow_failure: true
if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/ebpf/**/*
- pkg/security/**/*
- pkg/eventmonitor/**/*
- test/kitchen/site-cookbooks/dd-security-agent-check/**/*
- test/kitchen/test/integration/security-agent-test/**/*
- test/kitchen/test/integration/security-agent-stress/**/*
- .gitlab/functional_test/security_agent.yml
- .gitlab/kernel_matrix_testing/security_agent.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/security_agent.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
- allow_failure: true
when: manual
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
ARCH: x86_64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
INSTANCE_TYPE: m5d.metal
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: security-agent
upload_sysprobe_tests_arm64
upload_sysprobe_tests_arm64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_arm64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- prepare_sysprobe_ebpf_functional_tests_arm64
- tests_ebpf_arm64
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/collector/corechecks/ebpf/**/*
- pkg/collector/corechecks/servicediscovery/module/*
- pkg/ebpf/**/*
- pkg/network/**/*
- pkg/process/monitor/*
- pkg/util/kernel/**/*
- .gitlab/kernel_matrix_testing/system_probe.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/system_probe.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
tags:
- arch:arm64
variables:
ARCH: arm64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
INSTANCE_TYPE: m6gd.metal
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: system-probe
upload_sysprobe_tests_x64
upload_sysprobe_tests_x64:
after_script:
- DD_API_KEY=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $API_KEY_ORG2) || exit $?;
export DD_API_KEY
- inv -e kmt.tag-ci-job
artifacts:
paths:
- $CI_PROJECT_DIR/connector-${ARCH}
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- touch $AWS_EC2_SSH_KEY_FILE && chmod 600 $AWS_EC2_SSH_KEY_FILE
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_KEY > $AWS_EC2_SSH_KEY_FILE || exit
$?
- echo "" >> $AWS_EC2_SSH_KEY_FILE
- chmod 600 $AWS_EC2_SSH_KEY_FILE
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/datadog-agent-buildimages/system-probe_x64$DATADOG_AGENT_SYSPROBE_BUILDIMAGES_SUFFIX:$DATADOG_AGENT_SYSPROBE_BUILDIMAGES
needs:
- go_deps
- prepare_sysprobe_ebpf_functional_tests_x64
- tests_ebpf_x64
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_KMT_TESTS == 'on'
- changes:
compare_to: main
paths:
- pkg/collector/corechecks/ebpf/**/*
- pkg/collector/corechecks/servicediscovery/module/*
- pkg/ebpf/**/*
- pkg/network/**/*
- pkg/process/monitor/*
- pkg/util/kernel/**/*
- .gitlab/kernel_matrix_testing/system_probe.yml
- .gitlab/kernel_matrix_testing/common.yml
- .gitlab/source_test/ebpf.yml
- test/new-e2e/system-probe/**/*
- test/new-e2e/scenarios/system-probe/**/*
- test/new-e2e/pkg/runner/**/*
- test/new-e2e/pkg/utils/**/*
- test/new-e2e/go.mod
- tasks/system_probe.py
- tasks/kmt.py
- tasks/kernel_matrix_testing/*
script:
+ - source /root/.bashrc
- pushd $CI_PROJECT_DIR/kmt-deps/ci/$ARCH
- tar czvf $TEST_ARCHIVE_NAME opt
- popd
- FILTER_TEAM="Name=tag:team,Values=ebpf-platform"
- FILTER_MANAGED="Name=tag:managed-by,Values=pulumi"
- FILTER_STATE="Name=instance-state-name,Values=running"
- FILTER_PIPELINE="Name=tag:pipeline-id,Values=${CI_PIPELINE_ID}"
- FILTER_ARCH="Name=tag:arch,Values=${ARCH}"
- FILTER_INSTANCE_TYPE="Name=tag:instance-type,Values=${INSTANCE_TYPE}"
- FILTER_TEST_COMPONENT="Name=tag:test-component,Values=${TEST_COMPONENT}"
- QUERY_INSTANCE_IDS='Reservations[*].Instances[*].InstanceId'
- QUERY_PRIVATE_IPS='Reservations[*].Instances[*].PrivateIpAddress'
- "COUNTER=0\nwhile [[ $(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED\
\ $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE\
\ --output text --query $QUERY_INSTANCE_IDS | wc -l ) != \"1\" && $COUNTER -le\
\ 80 ]]; do COUNTER=$[$COUNTER +1]; echo \"[${COUNTER}] Waiting for instance\"\
; sleep 30; done\n# check that instance is ready, or fail\nif [ $(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS | wc -l) -ne\
\ \"1\" ]; then\n echo \"Instance NOT found\"\n touch ${CI_PROJECT_DIR}/instance_not_found\n\
\ \"false\"\nfi\necho \"Instance found\"\nINSTANCE_ID=$(aws ec2 describe-instances\
\ --filters $FILTER_TEAM $FILTER_MANAGED $FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT\
\ $FILTER_INSTANCE_TYPE --output text --query $QUERY_INSTANCE_IDS)\naws ec2 wait\
\ instance-status-ok --instance-ids $INSTANCE_ID\nsleep 10\n"
- INSTANCE_IP=$(aws ec2 describe-instances --filters $FILTER_TEAM $FILTER_MANAGED
$FILTER_STATE $FILTER_PIPELINE $FILTER_TEST_COMPONENT $FILTER_INSTANCE_TYPE --output
text --query $QUERY_PRIVATE_IPS)
- echo "$ARCH-instance-ip" $INSTANCE_IP
- mkdir -p ~/.ssh && chmod 700 ~/.ssh
- echo -e "Host metal_instance\nHostname $INSTANCE_IP\nUser ubuntu\nStrictHostKeyChecking
no\nIdentityFile $AWS_EC2_SSH_KEY_FILE\n" | tee -a ~/.ssh/config
- chmod 600 ~/.ssh/config
- scp $CI_PROJECT_DIR/kmt-deps/ci/$ARCH/$TEST_ARCHIVE_NAME metal_instance:/opt/kernel-version-testing/
- pushd $CI_PROJECT_DIR/test/new-e2e
- go build -o $CI_PROJECT_DIR/connector-${ARCH} $CI_PROJECT_DIR/test/new-e2e/system-probe/connector/main.go
- popd
- scp $CI_PROJECT_DIR/connector-${ARCH} metal_instance:/home/ubuntu/connector
stage: kernel_matrix_testing_prepare
tags:
- arch:amd64
variables:
ARCH: x86_64
AWS_EC2_SSH_KEY_FILE: $CI_PROJECT_DIR/ssh_key
INSTANCE_TYPE: m5d.metal
TEST_ARCHIVE_NAME: tests-$ARCH.tar.gz
TEST_COMPONENT: system-probe
Added Jobs
.fips-compliance-e2e
.fips-compliance-e2e:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $E2E_OUTPUT_DIR
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_e2e.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
|| exit $?
- touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
$SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
- pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
- ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
|| exit $?; export ARM_CLIENT_ID
- ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
|| exit $?; export ARM_CLIENT_SECRET
- ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
|| exit $?; export ARM_TENANT_ID
- ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
|| exit $?; export ARM_SUBSCRIPTION_ID
- GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
|| exit $?; export GOOGLE_CREDENTIALS
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_e2e_deps
rules:
- if: $RUN_E2E_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_E2E_TESTS == "on"
when: on_success
- if: $CI_COMMIT_BRANCH == "main"
when: on_success
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
when: on_success
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: on_success
- changes:
compare_to: main
paths:
- test/new-e2e/pkg/**/*
- test/new-e2e/go.mod
- changes:
compare_to: main
paths:
- cmd/**/*
- pkg/**/*
- comp/**/*
- test/new-e2e/tests/agent-subcommands/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
-c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
--junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
stage: check_fips_compliance
tags:
- arch:amd64
variables:
E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
E2E_PIPELINE_ID: $CI_PIPELINE_ID
E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
SHOULD_RUN_IN_FLAKES_FINDER: 'true'
TARGETS: ./tests/fips-compliance
TEAM: agent-shared-components
fips-compliance-e2e-linux
fips-compliance-e2e-linux:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $E2E_OUTPUT_DIR
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_e2e.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
|| exit $?
- touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
$SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
- pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
- ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
|| exit $?; export ARM_CLIENT_ID
- ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
|| exit $?; export ARM_CLIENT_SECRET
- ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
|| exit $?; export ARM_TENANT_ID
- ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
|| exit $?; export ARM_SUBSCRIPTION_ID
- GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
|| exit $?; export GOOGLE_CREDENTIALS
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_e2e_deps
- qa_agent
parallel:
matrix:
- EXTRA_PARAMS: --run "TestLinuxFIPSComplianceSuite"
- EXTRA_PARAMS: --run "TestFIPSCiphersSuite"
rules:
- if: $RUN_E2E_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_E2E_TESTS == "on"
when: on_success
- if: $CI_COMMIT_BRANCH == "main"
when: on_success
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
when: on_success
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: on_success
- changes:
compare_to: main
paths:
- test/new-e2e/pkg/**/*
- test/new-e2e/go.mod
- changes:
compare_to: main
paths:
- cmd/**/*
- pkg/**/*
- comp/**/*
- test/new-e2e/tests/agent-subcommands/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
-c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
--junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
stage: check_fips_compliance
tags:
- arch:amd64
variables:
E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
E2E_PIPELINE_ID: $CI_PIPELINE_ID
E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
SHOULD_RUN_IN_FLAKES_FINDER: 'true'
TARGETS: ./tests/fips-compliance
TEAM: agent-shared-components
fips-compliance-e2e-windows
fips-compliance-e2e-windows:
after_script:
- $CI_PROJECT_DIR/tools/ci/junit_upload.sh
artifacts:
expire_in: 2 weeks
paths:
- $E2E_OUTPUT_DIR
- junit-*.tgz
reports:
annotations:
- $EXTERNAL_LINKS_PATH
when: always
before_script:
- mkdir -p $GOPATH/pkg/mod/cache && tar xJf modcache_e2e.tar.xz -C $GOPATH/pkg/mod/cache
- rm -f modcache_e2e.tar.xz
- mkdir -p ~/.aws
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $AGENT_QA_PROFILE >> ~/.aws/config ||
exit $?
- export AWS_PROFILE=agent-qa-ci
- $CI_PROJECT_DIR/tools/ci/fetch_secret.sh $SSH_PUBLIC_KEY_RSA > $E2E_PUBLIC_KEY_PATH
|| exit $?
- touch $E2E_PRIVATE_KEY_PATH && chmod 600 $E2E_PRIVATE_KEY_PATH && $CI_PROJECT_DIR/tools/ci/fetch_secret.sh
$SSH_KEY_RSA > $E2E_PRIVATE_KEY_PATH || exit $?
- pulumi login "s3://dd-pulumi-state?region=us-east-1&awssdk=v2&profile=$AWS_PROFILE"
- ARM_CLIENT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_ID)
|| exit $?; export ARM_CLIENT_ID
- ARM_CLIENT_SECRET=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_CLIENT_SECRET)
|| exit $?; export ARM_CLIENT_SECRET
- ARM_TENANT_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_TENANT_ID)
|| exit $?; export ARM_TENANT_ID
- ARM_SUBSCRIPTION_ID=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_AZURE_SUBSCRIPTION_ID)
|| exit $?; export ARM_SUBSCRIPTION_ID
- GOOGLE_CREDENTIALS=$($CI_PROJECT_DIR/tools/ci/fetch_secret.sh $E2E_TESTS_GCP_CREDENTIALS)
|| exit $?; export GOOGLE_CREDENTIALS
- inv -e gitlab.generate-ci-visibility-links --output=$EXTERNAL_LINKS_PATH
image: 486234852809.dkr.ecr.us-east-1.amazonaws.com/ci/test-infra-definitions/runner$TEST_INFRA_DEFINITIONS_BUILDIMAGES_SUFFIX:$TEST_INFRA_DEFINITIONS_BUILDIMAGES
needs:
- go_e2e_deps
- deploy_windows_testing-a7
parallel:
matrix:
- EXTRA_PARAMS: --run "TestWindowsFIPSComplianceSuite"
rules:
- if: $RUN_E2E_TESTS == "off"
when: never
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- if: $RUN_E2E_TESTS == "on"
when: on_success
- if: $CI_COMMIT_BRANCH == "main"
when: on_success
- if: $CI_COMMIT_BRANCH =~ /^[0-9]+\.[0-9]+\.x$/
when: on_success
- if: $CI_COMMIT_TAG =~ /^[0-9]+\.[0-9]+\.[0-9]+-rc\.[0-9]+$/
when: on_success
- changes:
compare_to: main
paths:
- test/new-e2e/pkg/**/*
- test/new-e2e/go.mod
- changes:
compare_to: main
paths:
- cmd/**/*
- pkg/**/*
- comp/**/*
- test/new-e2e/tests/agent-subcommands/**/*
- if: $CI_COMMIT_BRANCH =~ /^mq-working-branch-/
when: never
- allow_failure: true
when: manual
script:
- inv -e new-e2e-tests.run --targets $TARGETS -c ddagent:imagePullRegistry=669783387624.dkr.ecr.us-east-1.amazonaws.com
-c ddagent:imagePullUsername=AWS -c ddagent:imagePullPassword=$(aws ecr get-login-password)
--junit-tar junit-${CI_JOB_ID}.tgz ${EXTRA_PARAMS} --test-washer
stage: check_fips_compliance
tags:
- arch:amd64
variables:
E2E_COMMIT_SHA: $CI_COMMIT_SHORT_SHA
E2E_KEY_PAIR_NAME: datadog-agent-ci-rsa
E2E_OUTPUT_DIR: $CI_PROJECT_DIR/e2e-output
E2E_PIPELINE_ID: $CI_PIPELINE_ID
E2E_PRIVATE_KEY_PATH: /tmp/agent-qa-ssh-key
E2E_PUBLIC_KEY_PATH: /tmp/agent-qa-ssh-key.pub
EXTERNAL_LINKS_PATH: external_links_$CI_JOB_ID.json
KUBERNETES_CPU_REQUEST: 6
KUBERNETES_MEMORY_LIMIT: 16Gi
KUBERNETES_MEMORY_REQUEST: 12Gi
SHOULD_RUN_IN_FLAKES_FINDER: 'true'
TARGETS: ./tests/fips-compliance
TEAM: agent-shared-components
Changes Summary
| Removed | Modified | Added | Renamed |
|---|---|---|---|
| 0 | 57 | 3 | 0 |
Test changes on VM
Use this command from test-infra-definitions to manually test this PR changes on a VM:
inv create-vm --pipeline-id=45072228 --os-family=ubuntu
Note: This applies to commit 98faafe8
Serverless Benchmark Results
BenchmarkStartEndInvocation comparison between 3ca96b63fbc790ec56c2a69223faf24d5aeb3fcb and 98c1d481ee74200c0881b41385c94edd6e4ad5a1.
tl;dr
Use these benchmarks as an insight tool during development.
-
Skim down the
vs basecolumn in each chart. If there is a~, then there was no statistically significant change to the benchmark. Otherwise, ensure the estimated percent change is either negative or very small. -
The last row of each chart is the
geomean. Ensure this percentage is either negative or very small.
What is this benchmarking?
The BenchmarkStartEndInvocation compares the amount of time it takes to call the start-invocation and end-invocation endpoints. For universal instrumentation languages (Dotnet, Golang, Java, Ruby), this represents the majority of the duration overhead added by our tracing layer.
The benchmark is run using a large variety of lambda request payloads. In the charts below, there is one row for each event payload type.
How do I interpret these charts?
The charts below comes from benchstat. They represent the statistical change in duration (sec/op), memory overhead (B/op), and allocations (allocs/op).
The benchstat docs explain how to interpret these charts.
Before the comparison table, we see common file-level configuration. If there are benchmarks with different configuration (for example, from different packages), benchstat will print separate tables for each configuration.
The table then compares the two input files for each benchmark. It shows the median and 95% confidence interval summaries for each benchmark before and after the change, and an A/B comparison under "vs base". ... The p-value measures how likely it is that any differences were due to random chance (i.e., noise). The "~" means benchstat did not detect a statistically significant difference between the two inputs. ...
Note that "statistically significant" is not the same as "large": with enough low-noise data, even very small changes can be distinguished from noise and considered statistically significant. It is, of course, generally easier to distinguish large changes from noise.
Finally, the last row of the table shows the geometric mean of each column, giving an overall picture of how the benchmarks changed. Proportional changes in the geomean reflect proportional changes in the benchmarks. For example, given n benchmarks, if sec/op for one of them increases by a factor of 2, then the sec/op geomean will increase by a factor of ⁿ√2.
I need more help
First off, do not worry if the benchmarks are failing. They are not tests. The intention is for them to be a tool for you to use during development.
If you would like a hand interpreting the results come chat with us in #serverless-agent in the internal DataDog slack or in #serverless in the public DataDog slack. We're happy to help!
Benchmark stats
goos: linux
goarch: amd64
pkg: github.com/DataDog/datadog-agent/pkg/serverless/daemon
cpu: AMD EPYC 7763 64-Core Processor
│ baseline/benchmark.log │ current/benchmark.log │
│ sec/op │ sec/op vs base │
api-gateway-appsec.json 90.20µ ± 4% 86.30µ ± 5% -4.32% (p=0.019 n=10)
api-gateway-kong-appsec.json 71.27µ ± 3% 69.00µ ± 3% -3.18% (p=0.002 n=10)
api-gateway-kong.json 70.41µ ± 2% 66.40µ ± 2% -5.70% (p=0.000 n=10)
api-gateway-non-proxy-async.json 110.8µ ± 1% 107.6µ ± 2% -2.88% (p=0.000 n=10)
api-gateway-non-proxy.json 113.7µ ± 2% 108.8µ ± 3% -4.28% (p=0.004 n=10)
api-gateway-websocket-connect.json 74.96µ ± 1% 72.96µ ± 2% -2.67% (p=0.000 n=10)
api-gateway-websocket-default.json 67.39µ ± 1% 65.49µ ± 3% -2.81% (p=0.009 n=10)
api-gateway-websocket-disconnect.json 67.38µ ± 1% 66.06µ ± 2% -1.96% (p=0.004 n=10)
api-gateway.json 121.9µ ± 1% 120.0µ ± 1% -1.58% (p=0.000 n=10)
application-load-balancer.json 67.79µ ± 1% 65.85µ ± 2% -2.86% (p=0.000 n=10)
cloudfront.json 50.70µ ± 2% 48.33µ ± 1% -4.66% (p=0.000 n=10)
cloudwatch-events.json 40.97µ ± 2% 39.04µ ± 2% -4.70% (p=0.000 n=10)
cloudwatch-logs.json 67.51µ ± 1% 66.34µ ± 1% -1.73% (p=0.007 n=10)
custom.json 33.09µ ± 1% 32.22µ ± 2% -2.63% (p=0.000 n=10)
dynamodb.json 96.86µ ± 2% 96.66µ ± 2% ~ (p=0.579 n=10)
empty.json 31.46µ ± 1% 31.03µ ± 2% ~ (p=0.075 n=10)
eventbridge-custom.json 50.60µ ± 2% 49.00µ ± 3% -3.17% (p=0.009 n=10)
eventbridge-no-bus.json 49.82µ ± 2% 47.98µ ± 3% -3.70% (p=0.000 n=10)
eventbridge-no-timestamp.json 49.58µ ± 3% 47.84µ ± 2% -3.51% (p=0.005 n=10)
http-api.json 77.79µ ± 2% 75.10µ ± 3% -3.47% (p=0.000 n=10)
kinesis-batch.json 74.92µ ± 1% 72.92µ ± 3% -2.67% (p=0.019 n=10)
kinesis.json 57.48µ ± 1% 56.88µ ± 2% ~ (p=0.315 n=10)
s3.json 62.48µ ± 1% 61.64µ ± 2% -1.35% (p=0.035 n=10)
sns-batch.json 95.09µ ± 2% 92.97µ ± 2% -2.22% (p=0.023 n=10)
sns.json 67.75µ ± 1% 66.69µ ± 4% ~ (p=0.218 n=10)
snssqs.json 114.5µ ± 1% 112.2µ ± 1% -1.97% (p=0.002 n=10)
snssqs_no_dd_context.json 103.4µ ± 1% 100.8µ ± 1% -2.52% (p=0.000 n=10)
sqs-aws-header.json 59.05µ ± 1% 58.81µ ± 2% ~ (p=0.739 n=10)
sqs-batch.json 100.43µ ± 2% 96.99µ ± 2% -3.42% (p=0.001 n=10)
sqs.json 72.75µ ± 3% 71.05µ ± 1% -2.34% (p=0.011 n=10)
sqs_no_dd_context.json 65.56µ ± 2% 66.05µ ± 1% ~ (p=0.436 n=10)
stepfunction.json 48.96µ ± 5% 47.15µ ± 3% -3.71% (p=0.007 n=10)
geomean 68.77µ 66.96µ -2.63%
│ baseline/benchmark.log │ current/benchmark.log │
│ B/op │ B/op vs base │
api-gateway-appsec.json 37.34Ki ± 0% 37.33Ki ± 0% ~ (p=0.197 n=10)
api-gateway-kong-appsec.json 26.94Ki ± 0% 26.94Ki ± 0% ~ (p=0.897 n=10)
api-gateway-kong.json 24.44Ki ± 0% 24.43Ki ± 0% ~ (p=0.566 n=10)
api-gateway-non-proxy-async.json 48.15Ki ± 0% 48.14Ki ± 0% ~ (p=0.325 n=10)
api-gateway-non-proxy.json 47.38Ki ± 0% 47.37Ki ± 0% ~ (p=0.382 n=10)
api-gateway-websocket-connect.json 25.55Ki ± 0% 25.55Ki ± 0% ~ (p=0.810 n=10)
api-gateway-websocket-default.json 21.45Ki ± 0% 21.45Ki ± 0% ~ (p=0.616 n=10)
api-gateway-websocket-disconnect.json 21.24Ki ± 0% 21.23Ki ± 0% ~ (p=0.385 n=10)
api-gateway.json 49.63Ki ± 0% 49.62Ki ± 0% ~ (p=0.725 n=10)
application-load-balancer.json 23.33Ki ± 0% 23.32Ki ± 0% ~ (p=0.085 n=10)
cloudfront.json 17.70Ki ± 0% 17.69Ki ± 0% ~ (p=0.072 n=10)
cloudwatch-events.json 11.75Ki ± 0% 11.74Ki ± 0% -0.10% (p=0.008 n=10)
cloudwatch-logs.json 53.39Ki ± 0% 53.39Ki ± 0% ~ (p=0.400 n=10)
custom.json 9.776Ki ± 0% 9.768Ki ± 0% ~ (p=0.362 n=10)
dynamodb.json 40.83Ki ± 0% 40.83Ki ± 0% ~ (p=0.809 n=10)
empty.json 9.339Ki ± 0% 9.362Ki ± 0% +0.25% (p=0.004 n=10)
eventbridge-custom.json 15.06Ki ± 0% 15.05Ki ± 0% ~ (p=0.469 n=10)
eventbridge-no-bus.json 14.04Ki ± 0% 14.02Ki ± 0% ~ (p=0.172 n=10)
eventbridge-no-timestamp.json 14.05Ki ± 0% 14.04Ki ± 0% ~ (p=0.796 n=10)
http-api.json 23.92Ki ± 0% 23.91Ki ± 0% ~ (p=0.927 n=10)
kinesis-batch.json 27.15Ki ± 0% 27.10Ki ± 0% -0.16% (p=0.027 n=10)
kinesis.json 17.92Ki ± 0% 17.95Ki ± 0% ~ (p=0.118 n=10)
s3.json 20.45Ki ± 0% 20.44Ki ± 1% ~ (p=0.529 n=10)
sns-batch.json 38.82Ki ± 0% 38.81Ki ± 0% ~ (p=0.926 n=10)
sns.json 24.09Ki ± 0% 24.09Ki ± 0% ~ (p=0.739 n=10)
snssqs.json 50.77Ki ± 0% 50.74Ki ± 0% ~ (p=0.542 n=10)
snssqs_no_dd_context.json 44.99Ki ± 0% 44.96Ki ± 0% ~ (p=0.123 n=10)
sqs-aws-header.json 18.85Ki ± 1% 18.97Ki ± 0% ~ (p=0.105 n=10)
sqs-batch.json 41.83Ki ± 0% 41.82Ki ± 0% ~ (p=0.811 n=10)
sqs.json 25.67Ki ± 0% 25.71Ki ± 1% ~ (p=0.971 n=10)
sqs_no_dd_context.json 20.78Ki ± 1% 20.84Ki ± 0% ~ (p=0.165 n=10)
stepfunction.json 14.36Ki ± 2% 14.30Ki ± 1% ~ (p=0.739 n=10)
geomean 24.50Ki 24.50Ki +0.00%
│ baseline/benchmark.log │ current/benchmark.log │
│ allocs/op │ allocs/op vs base │
api-gateway-appsec.json 630.0 ± 0% 629.0 ± 0% ~ (p=0.656 n=10)
api-gateway-kong-appsec.json 488.0 ± 0% 488.0 ± 0% ~ (p=1.000 n=10) ¹
api-gateway-kong.json 466.0 ± 0% 466.0 ± 0% ~ (p=1.000 n=10)
api-gateway-non-proxy-async.json 726.0 ± 0% 726.0 ± 0% ~ (p=1.000 n=10)
api-gateway-non-proxy.json 716.5 ± 0% 716.0 ± 0% ~ (p=0.650 n=10)
api-gateway-websocket-connect.json 453.5 ± 0% 453.0 ± 0% ~ (p=0.650 n=10)
api-gateway-websocket-default.json 380.0 ± 0% 379.0 ± 0% ~ (p=0.656 n=10)
api-gateway-websocket-disconnect.json 370.0 ± 0% 370.0 ± 0% ~ (p=1.000 n=10) ¹
api-gateway.json 791.0 ± 0% 791.0 ± 0% ~ (p=1.000 n=10)
application-load-balancer.json 353.0 ± 0% 353.0 ± 0% ~ (p=1.000 n=10)
cloudfront.json 284.0 ± 0% 284.0 ± 0% ~ (p=1.000 n=10) ¹
cloudwatch-events.json 221.0 ± 0% 220.0 ± 0% -0.45% (p=0.020 n=10)
cloudwatch-logs.json 216.0 ± 0% 216.0 ± 0% ~ (p=1.000 n=10)
custom.json 169.0 ± 1% 169.0 ± 1% ~ (p=1.000 n=10)
dynamodb.json 589.0 ± 0% 589.0 ± 0% ~ (p=1.000 n=10)
empty.json 160.0 ± 0% 160.0 ± 1% ~ (p=0.474 n=10)
eventbridge-custom.json 267.0 ± 0% 267.0 ± 0% ~ (p=0.582 n=10)
eventbridge-no-bus.json 258.0 ± 0% 258.0 ± 0% ~ (p=0.365 n=10)
eventbridge-no-timestamp.json 258.0 ± 0% 258.0 ± 0% ~ (p=1.000 n=10)
http-api.json 434.0 ± 0% 434.0 ± 0% ~ (p=0.569 n=10)
kinesis-batch.json 392.0 ± 0% 391.0 ± 1% -0.26% (p=0.016 n=10)
kinesis.json 286.0 ± 0% 287.0 ± 0% ~ (p=0.057 n=10)
s3.json 359.0 ± 1% 358.0 ± 1% ~ (p=0.374 n=10)
sns-batch.json 456.5 ± 0% 456.0 ± 0% ~ (p=0.812 n=10)
sns.json 324.0 ± 0% 324.0 ± 0% ~ (p=0.352 n=10)
snssqs.json 440.0 ± 0% 439.5 ± 0% ~ (p=0.681 n=10)
snssqs_no_dd_context.json 401.0 ± 0% 401.0 ± 0% ~ (p=0.254 n=10)
sqs-aws-header.json 274.0 ± 1% 276.0 ± 0% ~ (p=0.160 n=10)
sqs-batch.json 506.0 ± 0% 506.0 ± 0% ~ (p=0.963 n=10)
sqs.json 352.0 ± 1% 353.0 ± 1% ~ (p=0.812 n=10)
sqs_no_dd_context.json 325.5 ± 1% 326.5 ± 0% ~ (p=0.144 n=10)
stepfunction.json 239.0 ± 2% 237.5 ± 1% ~ (p=0.662 n=10)
geomean 363.7 363.6 -0.02%
¹ all samples are equal
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}