adoptium
Jenkins: audit of any retained individual build jobs to reduce disk usage
Ref: https://github.com/adoptium/infrastructure/issues/3285
jenkins@jenkins-hetzner-ubuntu2004:~/jobs$ du -sk build-scripts/jobs/jobs/jobs/jdk* build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk* build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk*
38483916 build-scripts/jobs/jobs/jobs/jdk
40709196 build-scripts/jobs/jobs/jobs/jdk11u
31727760 build-scripts/jobs/jobs/jobs/jdk17u
9464336 build-scripts/jobs/jobs/jobs/jdk20u
10528416 build-scripts/jobs/jobs/jobs/jdk21
12237780 build-scripts/jobs/jobs/jobs/jdk21u
13770112 build-scripts/jobs/jobs/jobs/jdk22
25356428 build-scripts/jobs/jobs/jobs/jdk8u
3805584 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk
18372 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk11u
4906184 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk17u
5259572 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk20u
4542752 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21
5025796 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21u
2300000 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk22
557188 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk23
627552 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk8u
16899976 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk11u
13937892 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk17u
5823836 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk19u
8339712 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk20
6462948 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk20u
6126940 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21
8507088 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21u
7039280 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk8u
With about 270Gb of space used by build-scripts/jobs (i.e. the individual platform jobs) we should look at which ones are being retained in there and see if they are still required.
See also https://github.com/adoptium/infrastructure/issues/3442 from the March upgrade cycle
Summary equivalent to the one in the initial description of this issue, in preparation for a clean-up session tomorow between myself and @steelhead31
42915908 build-scripts/jobs/jobs/jobs/jdk
46431992 build-scripts/jobs/jobs/jobs/jdk11u
38206776 build-scripts/jobs/jobs/jobs/jdk17u
1056 build-scripts/jobs/jobs/jobs/jdk20u
39546752 build-scripts/jobs/jobs/jobs/jdk21u
308 build-scripts/jobs/jobs/jobs/jdk22
1888 build-scripts/jobs/jobs/jobs/jdk22u
484 build-scripts/jobs/jobs/jobs/jdk23u
17509360 build-scripts/jobs/jobs/jobs/jdk24u
18523124 build-scripts/jobs/jobs/jobs/jdk25
488 build-scripts/jobs/jobs/jobs/jdk26
319665968 build-scripts/jobs/jobs/jobs/jdk8u
4913852 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk
1457116 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk11u
5785860 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk17u
5671880 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk21u
966084 build-scripts/jobs/jobs/jobs/evaluation/jobs/jobs/jobs/jdk8u
17057984 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk11u
16290960 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk17u
12174160 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk21u
9153732 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk24u
8116900 build-scripts/jobs/jobs/jobs/release/jobs/jobs/jobs/jdk8u
The jdk8u is over 300GiB but that is mostly from the Solaris simple jobs which are retaining more things than they should so the job can be easily reconfigured:
26313284 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simple
19242656 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simplepipe
109826440 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin-simpletest
576 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-sparcv9-temurin_SmokeTests
397104 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-hotspot
592 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-hotspot_SmokeTests
1139896 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin
60536000 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simple
36985096 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipe
24 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipebackup
28 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplepipestatic
68 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simplestatic
38687888 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simpletest
24 build-scripts/jobs/jobs/jobs/jdk8u/jobs/jdk8u-solaris-x64-temurin-simpleteststatic
Also noting that within the SPARC ones there are some large core files (>1GiB) being generated by the system tests - this example has 5 large core files in it making the archive for 91 end up as 91GiB in total. Noting that the x64 ones aren't producing cores but are still coming in at 1.3GiB per job archive and contain over 60,000 files. At the very least we should modify the process to compress them going forward (the 1.3Gib directory goes down to about 50M with xz and 170M with gzip
53876835 4 drwxr-xr-x 2 jenkins jenkins 4096 Jun 18 14:03 91/archive/output_17502006059704/langtools_all_0/work/classes/tools/javac/annotations/6365854/test/core
53664120 785116 -rw------- 1 jenkins jenkins 803954242 Jun 18 05:51 91/archive/output_17502144506801/MauveMultiThrdLoad_5m_0/20250617-225104-MauveMultiThrdLoad/results/core
53668107 1144776 -rw------- 1 jenkins jenkins 1172238666 Jun 18 06:33 91/archive/output_1750220679405/MiniMix_5m_0/20250617-232947-MixedLoadTest/results/core
53634418 1278776 -rw------- 1 jenkins jenkins 1309460234 Jun 18 06:37 91/archive/output_1750220679405/MiniMix_10m_0/20250617-233306-MixedLoadTest/results/core
53646157 3671740 -rw------- 1 jenkins jenkins 3759843010 Jun 18 06:45 91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-233725-MixedLoadTest/results/core
53648916 2681144 -rw------- 1 jenkins jenkins 2745476898 Jun 18 06:50 91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-234527-MixedLoadTest/results/core
Removing solaris core files has increased the space in the jobs file system from 36G to 111G - list of remove core files is in the twisty
jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441582146568/MiniMix_5m_0/20250408-192340-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441582146568/MiniMix_10m_0/20250408-192702-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/75/archive/output_17441519076259/MauveMultiThrdLoad_5m_0/20250408-175509-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_17502144506801/MauveMultiThrdLoad_5m_0/20250617-225104-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_5m_0/20250617-232947-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_10m_0/20250617-233306-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-233725-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/91/archive/output_1750220679405/MiniMix_aot_5m_0/20250617-234527-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_5m_0/20250609-191926-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_10m_0/20250609-192303-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_aot_5m_0/20250609-192949-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495147644198/MiniMix_aot_5m_0/20250609-193510-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/89/archive/output_17495085158373/MauveMultiThrdLoad_5m_0/20250609-181234-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/60/archive/output_17371492448118/MauveMultiThrdLoad_5m_0/20250117-163900-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/79/archive/output_17461573923340/MauveMultiThrdLoad_5m_0/20250501-225330-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375768463742/MauveMultiThrdLoad_5m_0/20250122-154200-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_5m_0/20250122-155907-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_10m_0/20250122-160233-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/62/archive/output_17375831371306/MiniMix_aot_5m_0/20250122-160704-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17483998949170/MauveMultiThrdLoad_5m_0/20250527-215856-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_5m_0/20250527-232729-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_10m_0/20250527-233213-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/81/archive/output_17484061333494/MiniMix_aot_5m_0/20250527-233704-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/59/archive/output_17371284426562/MauveMultiThrdLoad_5m_0/20250117-095106-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_5m_0/20250520-192338-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_10m_0/20250520-192935-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_1747787009862/MiniMix_aot_5m_0/20250520-193445-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/80/archive/output_17477807639826/MauveMultiThrdLoad_5m_0/20250520-190018-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/93/archive/output_17526835348670/MauveMultiThrdLoad_5m_0/20250716-125906-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/93/archive/output_17526897767603/MiniMix_10m_0/20250716-132702-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_5m_0/20250611-125046-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_10m_0/20250611-125524-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496639255764/MiniMix_aot_5m_0/20250611-130244-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/90/archive/output_17496576858763/MauveMultiThrdLoad_5m_0/20250611-122837-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_1748631151270/MauveMultiThrdLoad_5m_0/20250530-145850-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_17486373876362/MiniMix_5m_0/20250530-154149-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/83/archive/output_17486373876362/MiniMix_10m_0/20250530-154619-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_17371794076987/MauveMultiThrdLoad_5m_0/20250118-001654-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_5m_0/20250118-013903-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_10m_0/20250118-014217-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/61/archive/output_1737185626472/MiniMix_aot_5m_0/20250118-014652-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_5m_0/20250721-161420-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_10m_0/20250721-161856-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_aot_5m_0/20250721-163129-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531324597660/MiniMix_aot_5m_0/20250721-162337-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/95/archive/output_17531262467452/MauveMultiThrdLoad_5m_0/20250721-155749-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448304533232/MauveMultiThrdLoad_5m_0/20250416-151815-MauveMultiThrdLoad/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_5m_0/20250416-155139-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_10m_0/20250416-155720-MixedLoadTest/results/core
jdk8u-solaris-sparcv9-temurin-simpletest/builds/78/archive/output_17448366901769/MiniMix_aot_5m_0/20250416-160854-MixedLoadTest/results/core
FYI @smlambert - I'm not sure if we've got anything covering this or plan to take any action on the number of core files being created during the Solaris/sparcv9 system test runs that are showing in the above list?
Space increased from 116G to 166G after enabling discarding of old builds (20 jobs being retained for now - 10 with artifacts):
jenkins@jenkins-hetzner-ubuntu2004:~/jobs/build-scripts/jobs/jobs/jobs/jdk8u/jobs$ du -sk *simpletest
5448780 jdk8u-solaris-sparcv9-temurin-simpletest
7807268 jdk8u-solaris-x64-temurin-simpletest
Noting that as mentioned in https://github.com/adoptium/infrastructure/issues/3028 there are a significant number of old jobs - especially openjdk11-pipeline which has over 50GiB - which are explicitly locked from deletion. A number of them are ones from the AdoptOpenJDK days (including some from other variants such as OpenJ9) and the expectation is that they can be cleared too.
Note: job runs that have been explicitly marked as "keep forever" have <keepLog>true</keepLog> in their build.xml. At the time of eriting there at 805 such jobs - 242 of which have release in their name. I've split them into release jobs, other build jobs, test jobs and everything else:
keeplog=true.release.txt keeplog=true.builds.txt keeplog=true.test.txt keeplog=true.others.txt
discard options have also been added to all Solaris simple jobs which will retain 20 builds with a max of 3 artefacts for now. The last two GA builds have been set to keep permanently for now. This will take effect on the next build triggered from a new jdk8u tag. Current space usage before that cleanup showing there's a good 150GiB of space to be recovered here:
26313284 jdk8u-solaris-sparcv9-temurin-simple
19242656 jdk8u-solaris-sparcv9-temurin-simplepipe
5452176 jdk8u-solaris-sparcv9-temurin-simpletest
60536000 jdk8u-solaris-x64-temurin-simple
36985096 jdk8u-solaris-x64-temurin-simplepipe
9036828 jdk8u-solaris-x64-temurin-simpletest
Space increased from 116G to 166G after enabling discarding of old builds (20 jobs being retained for now - 10 with artifacts):
There is 246G today after the Solaris x64 jobs have done some clearup with the discard options now enabled (noting that this took effect without explicitly running the x64 jobs):
27054444 jdk8u-solaris-sparcv9-temurin-simple
19982568 jdk8u-solaris-sparcv9-temurin-simplepipe
5452176 jdk8u-solaris-sparcv9-temurin-simpletest
1841880 jdk8u-solaris-x64-temurin-simple
1833068 jdk8u-solaris-x64-temurin-simplepipe
9036828 jdk8u-solaris-x64-temurin-simpletest