Upstream: Build Upgrades

[sgammon]

Summary

Not for merge

This is a massive PR that upgrades the Gradle build in a big way. I did this sort of for fun, over the weekend, while working on other PRs which are reasonable and not enormous.

We are intending to use and extend Pkl so we could maintain a fork, and I have no expectation that this will get merged as-is. That being said, since we've done the leg work on some of this stuff and it could be useful, I wanted to file and offer any of the below as a menu of changes the Pkl team could get upstreamed from us if desired.

There are a handful of test failures and checksums have changed, which I am fixing. Total broken tests number less than 10, I believe, and the breakages seem fixable.

Feedback is of course welcome but since this PR isn't designed to be merged, such feedback would be incorporated in another PR, which is smaller and reviewable.

Thank you to the Pkl team for a very cool project!

Update: Feb 21st

New batch of changes pushed with the following:

• chore: buildSrc refactored to includeBuild("build-logic") (here's why)
• feat: upgrade GraalVM → latest (23.1.2)
• feat: support for --strict-image-heap
• feat: support for GraalVM PGO in native mode
• feat: preinitialization of pkl truffle module
• feat: support for engine sharing/context re-use
• feat: native jpms support (module-info)
• feat: faster native builds with -Ob
• feat: light plugin system for embedded customization of pkl-core engine
• ~~feat: use epsilon gc in native binary~~ note: rolling this back bc pkl server

Initial native binary benchmarks show a modest improvement:

Download URLs to try it out

• macOS aarch64: zip, tar.gz

Update: Split-out PRs (Feb 20th)

• #213
• #212
• #211

Changes enclosed

0) Fun developer ergonomics stuff.

• Pkl icon for IDEA project
• Pkl syntax highlighting in some test fragments using IDEA Language Injections

1) Build upgrades.

• Support for Gradle toolchains
• Support for Gradle's jvm-test-suite plugins
• Separate toolchain and library target JVMs
• Unified test and coverage reporting
• Detekt, PMD, with baseline set
• Local + remote build caching support
• All build warnings fixed
• Transition to property = value syntax instead of property.set(value)
• Kotlin upgrade → 1.9.22 (for build)
• Gradle upgraded → 8.6
• Toolchain JVM → 21

2) Supply chain security.

• OWASP dependency checks
• Gradle dependency verification
• SPDX SBOM (not yet landed)
• Signing of release artifacts with Sigstore (not yet landed)

3) GitHub project integration/ergonomics

• Dependency Review on PRs
• Github Dependency Graph build and submission
• Enforcement of formatting (Spotless) on PRs
• Checking of published Kotlin library APIs for breakages on PRs

4) Library updates / upgrades

• Kotlin upgrade → 1.9.22 (for codegen and scripting)
• Includes #196

Full changelog

Expand to see full changelog

- chore: `buildSrc` refactored to `includeBuild("build-logic")`
- feat: upgrade GraalVM → latest (`23.1.2`)
- feat: support for `--strict-image-heap`
- feat: support for GraalVM PGO in native mode
- feat: preinitialization of `pkl` truffle module
- feat: support for engine sharing/context re-use
- feat: native jpms support (`module-info`)
- feat: faster native builds with `-Ob`
- feat: support for build scans with gradle enterprise
- feat: support for caching with buildless (inert without key)
- feat: support for gradle java toolchains
- feat: support for static analysis with detekt
- feat: support for toolchain vs. runtime target
- feat: support for dynamic provisioning of toolchains
- feat: support for static java checking with pmd
- feat: enable typed project accessors, use them project-wide
- feat: stricter repositories, locking for build classpath
- feat: property to retarget java or kotlin bytecode versions
- feat: parameter name integration between javac and kotlinc
- feat: kotlin coverage support via `kover` plugin
- feat: dependency verification for gradle build
- feat: support for new gradle `jvm-test-suite` plugin
- feat: reasonable local and remote build caching support
- feat: project icon in intellij new ui
- feat: aggregate reporting for tests, coverage, detekt
- fix: repeatable/consistent archives from gradle
- fix: don't list ephemeral spotless configurations in lockfiles
- fix: make version catalog symbols available in `buildSrc`
- fix: specify `rootProject.name` for `buildSrc`
- fix: error when running `gradlew tasks`
- fix: various java or gradle deprecations
- fix: move all tool (linter, etc) versions into version catalog
- chore: add testlogger for clearer test outcomes
- chore: check build configuration with gradle doctor plugin
- chore: generate initial suite of dependency verification material
- chore: transition to property set syntax (`property = xyz`)
- chore: cleanup uses of `buildDir` (becomes `layout.buildDirectory`)
- chore: add Gradle Versions plugin for update checks
- chore: upgrade Gradle → `8.6` (supports Java 21)
- chore: upgrade Kotlin → `1.9.22` (build-time)
- chore: upgrade KotlinX Serialization → `1.6.3`
- chore: upgrade KotlinX HTML → `0.11.0`
- chore: general dependency upgrades, where safe

Not yet completed

• feat: signing of artifacts with sigstore
• feat: embedding of SPDX SBOM in artifacts
• feat: dependency vulnerability checks with owasp
• test: checksum failures

[sgammon]

Some screenshots for your viewing pleasure:

New project icon in IDEA

Highlighting of Pkl with language injections

New Github Action jobs for PR checks

Mocha-style test logging for clear testing outcomes

[sgammon]

I intend to benchmark this against main (the build, and runtime) to make sure it is an actual improvement over the status quo. I'll update this comment with the results of those tests.

Other material with data about this change:

• After: Build scan of a full build/test/check on this PR
• Before: Build scan of current main

[sgammon]

@holzensp You are totally right! The intent is not to merge this PR, but instead to offer a menu of changes for the Pkl team from our fork. I am completely happy to split any of these enclosed changes up into their own PRs.

Edit: It seems to me there are some very sensible changes I can split off first and file separately, for example, the Version Catalog tool fix. I'll clean up my existing PRs and file that first couple for review.

[sgammon]

Okay, I will pause there for now because I'm not sure which of these other changes you guys would like to see filed. Liking comments from the self-tour is a good way to let me know.

The first three are pretty reasonable and don't touch lockfiles at all, so we can avoid conflicts and I can make my lockfile changes more specific.

[sgammon]

@holzensp I have re-requested review with a new batch of changes, with another self-review listed above. This gets into bigger refactors like buildSrc → build-logic, JPMS, and GraalVM → 23.x, with PGO.

Even though these are bigger changes, with the benefit of hindsight on this fork, they should be easy to chop up into reviewable portions. Thanks for the reviews and for merging my PRs so far 😄

cc / @stackoverflow

Edit: Reports and other material

• CI run in GHA: Summary
• JVM build: Build scan
• JVM test: Build scan
• Testsuite: Build scan

[holzensp]

Terribly sorry; I was entirely off-the-grid for a couple of weeks. Circling back to this tomorrow.

[sgammon]

@holzensp No problem! Take your time and hope your trip off-grid was good 😁. There's a lot here, this fork is alive and continues to evolve. The latest summary of features supported ahead of main (I could rebase to clean any of these up and split them out into their own PRs):

• Entirely modular build; this sort of turned into a thing (and a repo)
  • There are some dependencies here aside from Guava that needed modularization; I either have PRs for these or queued changes and JARs.
• GraalVM support is now at latest (23.1.2 / Java 21)
• Pkl can now build with PGO (~10% perf gain) and --strict-image-heap
• Lightweight plugin system for pkl-core based on service loader
• I fixed the issues related to upgrading SnakeYAML

The plug-in system allows me to customize the org.graalvm...Context before it is used by Pkl, so I can do things like inject my own FileSystem.

Please consider me available for anything needed for Pkl, and I'm on the Discord too. @bioball has my cell and knows I'm committed to help.

[netvl]

@sgammon A note on the buildSrc vs includeBuild thing: I believe that in recent Gradle versions, the buildSrc is now automatically treated as an included build: https://github.com/gradle/gradle/issues/2531 https://github.com/gradle/gradle/pull/22540

There are minor differences, but I think they are not related to the main reason why previously it was useful to replace buildSrc with an explicit included build. So I would say that it might be worth switching back to buildSrc since it is more conventional and does not require extra setup.