Build pipelines are keeping too many builds

[sxa]

Despite the following from :

create_job_from_template.groovy:        numToKeep(10)
create_job_from_template.groovy:        artifactNumToKeep(1)

We appear to be keeping far more builds than that in the pipelines:

root@Ubuntu-1604-xenial-64-minimal /home/jenkins/.jenkins/jobs/build-scripts/jobs # for A in open*; do echo ==== $A; ls -d $A/builds/[0-9]* | wc; done
==== openjdk10-pipeline
     10      10     300
==== openjdk11-pipeline
     32      32     960
==== openjdk12-pipeline
     23      23     690
==== openjdk13-pipeline
     14      14     406
==== openjdk8-pipeline
     34      34     986
==== openjdk9-pipeline
     10      10     290

This needs to be resolved as these have chewed up an additional 50Gb of space on the jenkins master node over the last week.

[sxa]

@johnoliver are you able to look into this? Is it just because of all the locked GA builds being retained?

[sxa]

Current numbers:

==== openjdk10-pipeline
      4       4     120
==== openjdk11-pipeline
     47      47    1410
==== openjdk12-pipeline
     33      33     990
==== openjdk13-pipeline
     24      24     715
==== openjdk8-pipeline
     47      47    1363
==== openjdk9-pipeline
      3       3      87

[johnoliver]

I think most of the issue is probably the ga builds, looking at the 8 pipeline we currently have 30 builds retained between GA and nightly, although that does still leave about 17 dirs unaccounted for why they are there. long term we do just need to find a way to offload our builds so they dont accumulate on ci.

[sxa]

The alternate question is whether they need to be retained on the CI for so long

[karianna]

We've increased disk space and @johnoliver is exploring S3 and Azure Storage and Artifactory storage options.