adoptium
Track runtimes of Windows pipelines with 5 releases
This week I have been performing dry runs for the April 2024 release cycle. Since we have five releases in parallel, some platforms will take longer than usual to work through the list they are all initiated. This is the first time we'll have done five, with 4 being initiated previously (8, 11, 17, 21 - 22 is the new one with 21 in LTS alongside all of the others)
I have already removed win32 from the default triggers on 8, 11 and 17 (We do not build win32 for 21+) which should reduce contention (We will start a separate pipeline for those) and we are also in the process of switching off some of our windows machines and replacing them with more up to date versions as per #3238) Most of this issue will involve the gathering of data to show how things are going and determine any improvements we can make going forward.
Builds were kicked off in the order JDK22, 8, 11, 17, and 21 in order to determine the worst-case scenario for the current latest LTS release. In each case the latest ones are the important ones, with previous ones added for comparison where available.
For the "Win64 pipeline" tables, note that the queue+build+smoketest all have to complete before other test jobs are initiated, so the total time for the pipeline will be the sum of those plus the slowest of the others (usually extended.openjdk). Note also that the "sign_verification" step also needs to run on a test machine, and that is done after the others are complete so cn get delayed for extended amounts of time too (I'm seeing 8 hours for some of them - perhaps that could be targetted to build nodes in the future?)
release-openjdk22-pipeline
Win64 JDK22 pipeline:
release-openjdk8-pipeline
Win64 JDK8 pipeline
release-openjdk11-pipeline
Win64 JDK11 pipeline
release-openjdk17-pipeline
Win64 JDK17 pipeline
release-openjdk21-pipeline
Win64 JDK21 pipeline
Noting also that we've got some jobs that appear quite "stuck" as mentioned in slack:
- https://ci.adoptium.net/job/Test_openjdk17_hs_extended.openjdk_x86-32_windows_rerun/10/consoleFull seems to have been stuck on the win2019 machine (running for over 6 hours and not progressing, despite only having two targets)
- https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_x86-64_windows_rerun/10/console is another re-run which seems to be generating quite a lot of exceptions on test-azure-win2022-x64-2
- https://ci.adoptium.net/job/Test_openjdk21_hs_extended.openjdk_x86-64_windows_testList_1/4/console may also be stuck as it started over four hours ago and the log shows it hasn't produced any more output after about the first half hour of execution.
System has mostly quiesced now. Regens done to remove Linux/s390x from JDK22u and win32 from all relevant releases (8, 11, 17). Those plus the usual jdk8/arm32 will be triggered manually in the April cycle. I have also added the test labels to build-azure-2 to increase capacity a little more. The next dry-run (once regenerations are done) will also include the fix to allow the rhel6 test machine to be re-enabled, and the marking offline of all of the equinix_esxi systems which are due to be decommissioned.
Initiated new pipelines:
- 1731UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk22-pipeline/13/
- 1826UTC :https://ci.adoptium.net/job/build-scripts/job/release-openjdk8-pipeline/48
- Windows failed, rerunning it at https://ci.adoptium.net/job/build-scripts/job/jobs/job/release/job/jobs/job/jdk8u/job/jdk8u-release-windows-x64-temurin/18/
- 1850:UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk11-pipeline/44/
- 1914:UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk17-pipeline/61/
- 1918 UTC: https://ci.adoptium.net/job/build-scripts/job/release-openjdk21-pipeline/31/
JDK22
release-openjdk22-pipeline
jdk22u-release-windows-x64-temurin
Test_openjdk22_hs_extended.openjdk_x86-64_windows
JDK8
release-openjdk8-pipeline
jdk8u-release-windows-x64-temurin
Test_openjdk8_hs_extended.openjdk_x86-64_windows
JDK11
release-openjdk11-pipeline
jdk11u-release-windows-x64-temurin
Test_openjdk11_hs_extended.openjdk_x86-64_windows
JDK17
release-openjdk17-pipeline
jdk17u-release-windows-x64-temurin
Test_openjdk17_hs_extended.openjdk_x86-64_windows
JDK21
release-openjdk21-pipeline
jdk21u-release-windows-x64-temurin
Test_openjdk21_hs_extended.openjdk_x86-64_windows
From the release pipelines in April 2024:
| JDK | 64-bit | 32-bit |
|---|---|---|
| JDK8 | 4h29 (0) | 9h32 (3h) |
| JDK11 | 9h30 (0h) | 12h16 (0h) |
| JDK17 | 25h | 13h |
| JDK21 | 14h27 | n/a |
| JDK22 | 25h | n/a |
FYI @steelhead31 perhaps we should look at what these times were like for the October release and then see whether the addition of dynamic machines for additional capacity has made a difference in the January release (or even some of the ea builds)