Clear up old workspace directories on build machines

[sxa]

The more we leave around the larger the risk of hitting an out of disk space condition. Here is an example where many now-obsolete directories are on the first Linux/ppc64le build machine:

[jenkins@build-osuosl-centos74-ppc64le-2 ~]$ du -sk /home/jenkins/workspace/build-scripts/jobs/*
2269036	/home/jenkins/workspace/build-scripts/jobs/jdk
4	/home/jenkins/workspace/build-scripts/jobs/jdk10u
4389292	/home/jenkins/workspace/build-scripts/jobs/jdk11u
4	/home/jenkins/workspace/build-scripts/jobs/jdk12u
4	/home/jenkins/workspace/build-scripts/jobs/jdk13u
12832376	/home/jenkins/workspace/build-scripts/jobs/jdk14
5723684	/home/jenkins/workspace/build-scripts/jobs/jdk14u
13602956	/home/jenkins/workspace/build-scripts/jobs/jdk15
4	/home/jenkins/workspace/build-scripts/jobs/jdk15u
6496820	/home/jenkins/workspace/build-scripts/jobs/jdk16
3691776	/home/jenkins/workspace/build-scripts/jobs/jdk16u
7051580	/home/jenkins/workspace/build-scripts/jobs/jdk17
8925568	/home/jenkins/workspace/build-scripts/jobs/jdk17u
2265320	/home/jenkins/workspace/build-scripts/jobs/jdk18
8673852	/home/jenkins/workspace/build-scripts/jobs/jdk8u
[jenkins@build-osuosl-centos74-ppc64le-2 ~]$ 

JDK14, 15 and 16 are no longer required. The non-u version of 17 is no longer required, and with the recent changes anything with hotspot will no longer be required as the jobs have been changed to have temurin in the name. Thinking about it, it may just be worth wiping out the whole of ~jenkins/workspace/build-scripts/jobs on every machine ...

[aixtools]

Doing that (removing all jobs/* directories) - weekly, or otherwise before a release cycle (just so it is done regularly) might be a simple solution.

A git checkout action is not the worst thing in the world - certainly compared to a systemdown status due to lack of freespace.

[sxa]

This is going to need a re-evaluation in light of the release and evaluation pipelines that we now have which has the potential to duplicate the space required for builds

[sxa]

This will likely be exacerbated by having slpit out the release pipelines, and switching some platforms to experimental. Marking critical

[steelhead31]

Im going to take a look at this tomorrow as my day of learning exercise!

[steelhead31]

Having done some investigating, I believe the best/simplest way to do this is via a groovy script run on the jenkins master, this may need a groovy plugin for jenkins.. @sxa do you have any thoughts on installing this plugin for example ?

[sxa]

Hmmm I'm curious as to why it needs to run specifically on the jenkins master? Can jenkins pipelines not let you do the same as the groovy plugin? Are you looking at something more complex than just running an equvalent of "Delete build-scripts/jobs/${VERSION}" from the top level jenkins workspace directory across all machines labelled as build

[steelhead31]

Probably,  just digging around various solutions... I could use the same plugin to run on selected nodes too, ie those with the build label... I've written a script, but I can only test on 2 vms ... so running on the master appears to work... running from the master will clear workspaces across all nodes, which is what I've done in dev... should be easy to adapt to run using labels though

def deleted = []
def oneWeekAgo = new Date() - 7
jenkins.model.Jenkins.instance.nodes.each { hudson.model.Node node ->
  node.workspaceRoot.listDirectories().each { hudson.FilePath path ->
    def pathName = path.getRemote()
    if (path.name.startsWith(".")) {
      println "Skipping internal dir $node.displayName:$pathName"
    } else {
        def lastModified = new Date(path.lastModified())
      if (lastModified < oneWeekAgo) {
        println "Deleting workspace at $node.displayName:$pathName (last modified $lastModified)"
        path.deleteRecursive()
        deleted << "$node.displayName:$pathName"
      } else {
        println "Skipping workspace at $node.displayName:$pathName (last modified $lastModified)"
      }
    }
  }
}
"Deleted workspaces: \n\t" + deleted.sort().join("\n\t")

I'll try converting this into a pipeline script, I was trying to do it as a standalone job, but that doesn't let me use groovy, only shell!

[steelhead31]

Ok, version 1 is here.. https://ci.adoptium.net/view/Tooling/job/DeleteBuilderWorkspaces/ it clears down all workspace directories, working on an improved version to only delete directories over 7 days old.

[sxa]

@steelhead31 has this been run on all the machines successfully now?

[steelhead31]

@steelhead31 has this been run on all the machines successfully now?

Yes, this has, so Im going to close this.