WIP: Make gradle the root of every dependency graph

[donoghuc]

Do a fist pass at removing circular dependency on rake calling to gradle. Replace every "top level" task with a gradle task. This allows doing all dependency management with gradle, while still allowing us to use rake for leaf node tasks when appropriate.

Release notes

What does this PR do?

Why is it important/What is the impact to the user?

Checklist

• [ ] My code follows the style guidelines of this project
• [ ] I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (and/or docker env variables)
• [ ] I have added tests that prove my fix is effective or that my feature works

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

[github-actions[bot]]

:robot: GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)
• /run exhaustive tests : Run the exhaustive tests Buildkite pipeline.

[mergify[bot]]

This pull request does not have a backport label. Could you fix it @donoghuc? 🙏 To fixup this pull request, you need to add the backport labels for the needed branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.
• If no backport is necessary, please add the backport-skip label

[donoghuc]

Note to self: make sure to re-evaluate https://github.com/elastic/logstash/pull/18340/files#r2488030095

[donoghuc]

run exhaustive tests

[donoghuc]

Got through PR tests, moving on the exhaustive next (Still need to do DRA and ansillary).

[donoghuc]

run exhaustive tests

[donoghuc]

Made more progress on acceptance tests... The fpm gem does not like being called from gradle/jruby 😅 https://github.com/elastic/logstash/pull/18471#discussion_r2612451324

Also, there will be some follow on work with invokingh bunlder directly for acceptance tests. For now, instead of invoking rake i'm invoking bundler, but i have not decided if that is actually a win or not.

[donoghuc]

Today i was able to figure out the fpm issues but the problem i'm playing whack-a-mole with is the management of dowloading jdk. The exhaustive test pipeline provides an example of an issue i'm facing.

Currently as a prep step we run ./gradlew clean bootstrap artifactDeb artifactRpm to prepare artifacts for testing on linux targets which consume .rpm and .deb packages. When run in isolation (for example artifactDeb) we can prepare a complete artifact (in this case one that includes a jdk). This has a dependsOn for copyJdk and is "finalized" by a delete of that jdk. Unfortunately when run together (artifactDeb artifactRpm) the deletion does not inform gradle that copyJdk is needed again and the wrong JDK is used.

I need to figure out exactly which tasks need the copyJdk (and which explicitly need it deleted) in order to figure out how to best represent it in gradle.

[donoghuc]

run exhaustive tests

[donoghuc]

run exhaustive tests

[donoghuc]

run exhaustive tests

[donoghuc]

I'v been building and testing on rhel-8 and ubuntu-2204 gcp resources (in addition to macos). I have the tests all passing etc and i'm fairly sure i've got the DRA stuff working. I'll be looking closer at ways to gain confidence in the DRA step, but this is for sure ready for review.

This is the first step in https://github.com/elastic/logstash/issues/18403 specifically https://github.com/elastic/logstash/issues/18476 which aims to basically just stop calling rake directly. We do still call rake directly for the acceptance tests, but in that case rake does not call back in to gradle. Untangling that bit of rake/gradle will be addressed with https://github.com/elastic/logstash/issues/18477

[donoghuc]

run exhaustive tests

[donoghuc]

I kicked off a DRA snapshot job on this to double check things would build. Obviously it fails to be published (desireable). Just wanted to make sure build is all working. Seems good! https://buildkite.com/elastic/logstash-dra-snapshot-pipeline/builds/4307

[elasticmachine]

:yellow_heart: Build succeeded, but was flaky

• Buildkite Build
• Commit: e83867dccd7bcc3210a7697b31d6825ecf3d3e62

Failed CI Steps

• :lab_coat: IT Persistent Queues / part 4-of-6

History

• :yellow_heart: Build #3965 was flaky e9b688361ba152ee70270e5fb986dfd543802a64
• :green_heart: Build #3963 succeeded edb91db095e47e963d4b25b1e1d94fb6b60851b1
• :green_heart: Build #3960 succeeded 30467c94c3d07c84fc68b6b1d8ec29980f0c6ecf
• :yellow_heart: Build #3959 was flaky 295b9a081cec80f4f171bcca348f52b7531c7a61
• :broken_heart: Build #3958 failed c5387b64900414879be5425db35e4953e2417da4
• :green_heart: Build #3956 succeeded 5c08684b5c0702605bcd160b13598f882584e988

[donoghuc]

Now that the 9.3 branch has been cut, i'm going to get this merged so i can build the PR for removal of unused tasks on this. I'll keep an eye on CI in the main branch to make sure there are no surprises!