[8.14](backport #39126) Beats DRA pipeline

[mergify[bot]]

Proposed commit message

Beats Packaging pipeline

Migrates packaging.groovy to buildkite. Buildkite packaging pipeline is used to build and publish the packages for the Beats. The pipeline is triggered by a commit to the main or release branches. The pipeline definition is located in the .buildkite/packaging.pipeline.yml

Triggers

Staging packaging DRA is triggered for the main and release branches. Snapshot can be triggered for any branch by the /package comment in the PR. The release-manager dry-run will be used for PR builds.

Pipeline steps

Beats dashboards

Generates build/distributions/dependencies.csv and adds it to the beats-dashboards artifact. The dependencies.csv is required by the release-manager configuration

Packaging snapshot/staging

• Builds the Beats packages for all supported platforms and architectures (mage package, mage ironbank)
• Copies artifacts build/distributions/<beat>/ directory and adds it as an artifact. Where <beat> is the name of the beat
• x-pack artifacts a also copied to build/distributions/<beat>/ directory, where <beat> is the name of the beat. For example, auditbeat, not x-pack/auditbeat

DRA publish

Downloads the artifacts from the packaging snapshot/staging step and publishes them to the Elastic DRA registry.

Snapshot publishing example (Dry run)

Beats-tester and e2e tests are not triggered

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [x] All Snapshot and staging artifacts are built, present, and properly transferred to the corresponding DRA steps
• [x] DRA dry run passes
• [ ] All artifacts have the same size and permissions as on Jenkins. (Need to check one more time)

How to test this PR locally

Related issues

• https://github.com/elastic/ingest-dev/issues/3072
• https://github.com/elastic/ingest-dev/issues/3095

Use cases

Screenshots

Logs

---

This is an automatic backport of pull request #39126 done by [Mergify](https://mergify.com).

[mergify[bot]]

Cherry-pick of c6444db54474441450921fc845a97b148bfaba71 has failed:

On branch mergify/bp/8.14/pr-39126
Your branch is up to date with 'origin/8.14'.

You are currently cherry-picking commit c6444db544.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .buildkite/packaging.pipeline.md
	new file:   .buildkite/scripts/dra.sh
	new file:   .buildkite/scripts/packaging/package-dra.sh
	new file:   .buildkite/scripts/packaging/prepare-release-manager.sh

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   .buildkite/packaging.pipeline.yml

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[elasticmachine]

:green_heart: Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2024-04-26T15:38:19.451+0000

• Duration: 14 min 36 sec

Test stats :test_tube:

Test	Results
Failed	0
Passed	3
Skipped	0
Total	3

:green_heart: Flaky test report

Tests succeeded.

:robot: GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[pazone]

@dliappis @alexsapran this one is ready

[mergify[bot]]

This pull request has not been merged yet. Could you please review and merge it @pazone? 🙏