Jenkins pipeline files for building OpenJDK

Eclipse Adoptium makes use of these scripts to build binaries on the build farm at https://ci.adoptium.net

Repository contents

This repository contains several useful scripts in order to build OpenJDK personally or at build farm scale.

1. The docs folder contains images and utility scripts to produce up to date documentation.
2. The pipelines folder contains the Groovy pipeline scripts for Jenkins (e.g. build | test | checksum | release).
3. The tools folder contains pipelines/ analysis scripts that deliever success/failure trends and build scripts for code-tool dependancies for the build and test process (e.g. asmtools | jcov | jtharness | jtreg | sigtest).

Configuration Files

The pipelines/jobs/configurations directory contains two categories of configuration files that our jenkins pipelines use (Nicknamed #Build Configs and #Nightly Configs for short).

To ensure both configurations are not overridden in a race condition scenario by another job, the job generators ensure they remain in the sync with the repository.

Generally, any new parameters/configurations that effect the jenkins environment directly should be implemented here. If this is not the case, it would likely be better placed in temurin-build/platform-specific-configurations (for OS or make-adopt-build-farm.sh specific use cases) or temurin-build/build.sh (for anyone, including end users and jenkins pipelines).

Build

The build config files are the ones that follow the format jdkxx(u)_pipeline_config.groovy with xx being the version number and an optional u if the Java source code is pulled from an update repository. Each is a groovy class with a single Map<String, Map<String, ?>> property containing node labels, tests and other jenkins parameters/constants that are crucial for allowing different parts of the build pipeline to mesh together.

Each architecture/platform has it's own entry similar to the one below (for JDK8 x64 mac builds). The pipelines use the parent map key (e.g. x64Mac) to retrieve the data. See #Data Fields for the currently available fields you can utilise.

x64Mac        : [
    os                   : 'mac',
    arch                 : 'x64',
    additionalNodeLabels : [
            temurin  : 'macos10.14',
            corretto : 'build-macstadium-macos1010-1',
            openj9   : 'macos10.14'
    ],
    test                 : 'default'
]

Data fields

NOTE: When the type field implies a map, the String key of the inner map is the variant for that field. E.g:

                additionalNodeLabels : [
                        temurin : 'xlc13&&aix710',
                        openj9 :  'xlc13&&aix715'
                ],

Nightly

The nightly or beta/non-release config files are the ones that follow the format jdkxx(u).groovy with xx being the version number and an optional u if the Java source code is pulled from an update repository. Each is a simple groovy script that's contents can be loaded in and accessed by another script.

Evaluation pipeline/jobs

The evaluation config files are the ones that follow the format jdkxx(u)_evaluation.groovy with xx being the version number and an optional u if the Java source code is pulled from an update repository.

targetConfigurations

A single Map<String, Map<String, String>> variable containing what platforms and variants will be run in the nightly builds, evaluation builds and releases (by default, this can be altered in jenkins parameters before executing a user build). If you are creating your own nightly config, you will need to ensure the key values of the upper map are the same as the key values in the corresponding build config file.

Release pipeline/jobs

The release config files are the ones that follow the format jdkxx(u)_release.groovy with xx being the version number and an optional u if the Java source code is pulled from an update repository. jdkxx(u)*.groovy

targetConfigurations = [
        "x64Mac"        : [
                "temurin",
                "openj9"
        ],
        "x64Linux"      : [
                "temurin",
                "openj9",
                "corretto",
                "dragonwell"
        ],
        "x32Windows"    : [
                "temurin",
                "openj9"
        ],
        "x64Windows"    : [
                "temurin",
                "openj9",
                "dragonwell"
        ],
        "ppc64Aix"      : [
                "temurin",
                "openj9"
        ],
        "ppc64leLinux"  : [
                "temurin",
                "openj9"
        ],
        "s390xLinux"    : [
                "temurin",
                "openj9"
        ],
        "aarch64Linux"  : [
                "temurin",
                "openj9",
                "dragonwell"
        ],
        "arm32Linux"  : [
                "temurin"
        ],
        "sparcv9Solaris": [
                "temurin"
        ]
]

disableJob

If this is present, the jenkins generators will still create the top-level pipeline and downstream jobs but will set them as disabled. jdkxx(u).groovy

disableJob = true

triggerSchedule_nightly / triggerSchedule_weekly / triggerSchedule_evaluation / triggerSchedule_weekly_evaluation

All JDK versions now support "beta" EA triggered builds from the publication of upstream build tags. Eclipse Adoptium no longer runs scheduled nightly/weekend builds.

The one exception to this is Oracle managed STS versions, whose builds are managed internal to Oracle and not published until the GA day. For these a triggerSchedule_weekly is required to build the upstream HEAD commits on a regular basis.

Cron expression that defines when (and how often) nightly/evaluation and weekly/weekly-evaluation builds will be executed

in jdkxx(u).groovy

triggerSchedule_nightly="TZ=UTC\n05 18 * * 1,3,5"
triggerSchedule_weekly="TZ=UTC\n05 12 * * 6"

in jdkXX(u)_evaluation.groovy

triggerSchedule_evaluation="TZ=UTC\n15 18 * * 1,3,5"
triggerSchedule_weekly_evaluation="TZ=UTC\n25 12 * * 6"

weekly_release_scmReferences / weekly_evaluation_scmReferences

Source control references (e.g. tags) to use in the scheduled weekly release or weekly evaluation builds in jdkXX(u).groovy Use below two ways can set the job never to run:

• do not set triggerSchedule_nightly or triggerSchedule_weekly in the groovy file
• untick ENABLE_PIPELINE_SCHEDULE option in the Jenkins job which calls pipelines/build/regeneration/build_pipeline_generator.groovy

weekly_release_scmReferences

Source control references (e.g. tags) to use in the scheduled weekly release builds jdkxx(u).groovy

weekly_release_scmReferences = [
        "temurin"        : "jdk8u282-b08"
]

in jdkXX(u)_evaluation.groovy

weekly_evaluation_scmReferences== [
        "temurin"        : "jdk8u282-b07",
        "openj9"         : "v0.24.0-release",
        "corretto"       : "",
        "dragonwell"     : ""
]

Metadata

Alongside the built assets a metadata file will be created with info about the build. This will be a JSON document of the form:

{
    "vendor": "Eclipse Adoptium",
    "os": "mac",
    "arch": "x64",
    "variant": "openj9",
    "version": {
        "minor": 0,
        "patch": null,
        "msi_product_version": "11.0.18.6",
        "security": 0,
        "pre": null,
        "adopt_build_number": 0,
        "major": 15,
        "version": "15+29-202007070926",
        "semver": "15.0.0+29.0.202007070926",
        "build": 29,
        "opt": "202007070926"
    },
    "scmRef": "<output of git describe OR buildConfig.SCM_REF>",
    "buildRef": "<build-repo-name/build-commit-sha>",
    "version_data": "jdk15",
    "binary_type": "debugimage",
    "sha256": "<shasum>",
    "full_version_output": "<output of java --version>",
    "makejdk_any_platform_args": "<output of configure to makejdk-any-platform.sh>",
    "configure_arguments": "<output of bash configure>"
}

The Metadata class is contained in the Metadata.groovy file and the Json is constructed and written in the openjdk_build_pipeline.groovy file.

It is worth noting the additional tags on the SemVer is the Adoptium build number.

Below are all of the keys contained in the metadata file and some example values that can be present.

• vendor: Example values: [Eclipse Adoptium, Alibaba, Huawei]

This tag is used to identify the vendor of the JDK being built, this value is set in the build.sh file and defaults to "Temurin".

• os: Example values: [windows, mac, linux, aix, solaris, alpine-linux]

This tag identifies the operating system the JDK has been built on (and should be used on).

• arch: Example values: [aarch64, ppc64, s390x, x64, x86-32, arm]

This tag identifies the architecture the JDK has been built on and it intended to run on.

• variant: Example values: [hotspot, temurin, openj9, corretto, dragonwell, bisheng, fast_startup]

This tag identifies the JVM being used by the JDK, "dragonwell" itself is not a JVM but is currently considered a variant in its own right.

• variant_version:

This tag is used to identify a version number of the variant being built, it currently is exclusively used by OpenJ9 and has the following keys:

• major: Example values: [0, 1]

• minor: Example values: [22, 23, 24]

• security: Example values: [0, 1]

• tags: Example values: [m1, m2]

• version:

This tag contains the full version information of the JDK built, it uses the VersionInfo.groovy class and the ParseVersion.groovy class.

It contains the following keys:

• minor: Example values: [0]

• security: Example Values: [0, 9, 252 272]

• pre: Example values: [null]

• adopt_build_number: Example values: [0] If the ADOPT_BUILD_NUMBER parameter is used to build te JDK that value will appear here, otherwise a default value of 0 appears.

• major: Example values: [8, 11, 15, 16]

• version: Example values: [1.8.0_272-202010111709-b09, 11.0.9+10-202010122348, 14.0.2+11-202007272039, 16+19-202010120348]

• semver: Example values: [8.0.202+8.0.202008210941, 11.0.9+10.0.202010122348, 14.0.2+11.0.202007272039, 16.0.0+19.0.202010120339] Formed from the major, minor, security, and build number by the formSemver() function.

• build: Example values: [6, 9, 18] The OpenJDK build number for the JDK being built.

• opt: Example values: [202008210941, 202010120348, 202007272039]

• scmRef: Example values: [dragonwell-8.4.4_jdk8u262-b10, jdk-16+19_adopt-61198-g59e3baa94ac, jdk-11.0.9+10_adopt-197-g11f44f68c5, 23f997ca1]

A reference the the base JDK repository being build, usually including a GitHub commit reference, i.e. jdk-16+19_adopt-61198-g59e3baa94ac links to https://github.com/adoptium/jdk/commit/59e3baa94ac via the commit SHA 59e3baa94ac.

Values that only contain a commit reference such as 23f997ca1 are OpenJ9 commits on their respective JDK repositories, for example 23f997ca1 links to the commit https://github.com/ibmruntimes/openj9-openjdk-jdk14/commit/23f997ca1.

• buildRef: Example values: [temurin-build/fe0f2dba, temurin-build/f412a523] A reference to the build tools repository used to create the JDK, uses the format repository-name/commit-SHA.

• version_data: Example values: [jdk8u, jdk11u, jdk14u, jdk]

• binary_type: Example values: [jdk, jre, debugimage, testimage]

• sha256: Example values: [20278aa9459e7636f6237e85fcd68deec1f42fa90c6c541a2dfa127f4156d3e2, 2f9700bd75a807614d6d525fbd8d016c609a9ea71bf1ffd5d4839f3c1c8e4b8e] A SHA to verify the contents of the JDK.

• full_version_output: Example values:

openjdk version \"1.8.0_252\"\nOpenJDK Runtime Environment (Alibaba Dragonwell 8.4.4) (build 1.8.0_252-202010111720-b06)\nOpenJDK 64-Bit Server VM (Alibaba Dragonwell 8.4.4) (build 25.252-b06, mixed mode)\n`

The full output of the command java -version for the JDK.

• configure_arguments: The full output generated by make/autoconf/configure from the JDK built.

Tag driven "beta" EA builds

pipelines/build/common/trigger_beta_build.groovy

Jenkins pipeline to automate triggering of "beta" EA builds from the publication of upstream build tags. Eclipse Adoptium no longer runs scheduled nightly/weekend builds, instead pipeline builds are triggered by this job.

The one exception to this is Oracle managed STS versions, whose builds are managed internal to Oracle and not published until the GA day. For these a triggerSchedule_weekly is required to build the upstream HEAD commits on a regular basis.

pipelines/build/common/trigger_beta_build.groovy job parameters:

• String: JDK_VERSION - JDK version to trigger. (Numerical version number, 8, 11, 17, ...)

• String: MIRROR_REPO - github repository where source mirror is located for the given JDK_VERSION

• String: BINARIES_REPO - github organisation/repo template for where binaries are published for jdk-NN, "_NN_" gets replaced by the version

• CheckBox: FORCE_MAIN - Force the trigger of the "main" pipeline build for the current latest build tag, even if it is already published

• CheckBox: FORCE_EVALUATION - Force the trigger of the "evaluation" pipeline build for the current latest build tag, even if it is already published

• Multi-line Text: OVERRIDE_MAIN_TARGET_CONFIGURATIONS - Override targetConfigurations for FORCE_MAIN, eg: { "x64Linux": [ "temurin" ], "x64Mac": [ "temurin" ] }

• Multi-line Text: OVERRIDE_EVALUATION_TARGET_CONFIGURATIONS - Override targetConfigurations for FORCE_EVALUATION, eg: { "aarch64AlpineLinux": [ "temurin" ] }

Build status

Table generated with generateBuildMatrix.sh

Platform	Java 8	Java 11	Java 17	Java 21	Java HEAD
aix-ppc64-temurin
alpine-linux-aarch64-temurin				N/A
alpine-linux-x64-temurin
linux-aarch64-temurin
linux-arm-temurin
linux-ppc64le-temurin
linux-riscv64-temurin	N/A	N/A	N/A	N/A
linux-riscv64-temurin-cross	N/A	N/A	N/A	N/A
linux-s390x-temurin
linux-x64-temurin
mac-aarch64-temurin	N/A
mac-x64-temurin
solaris-sparcv9-temurin		N/A	N/A	N/A	N/A
solaris-x64-temurin		N/A	N/A	N/A	N/A
windows-aarch64-temurin	N/A			N/A
windows-x64-temurin
windows-x64-temurin_reproduce_compare	N/A	N/A	N/A	N/A	N/A
windows-x86-32-temurin				N/A	N/A