Should we disable auto-update of system packages?

[richardlau]

Originally posted by @richardlau in https://github.com/nodejs/build/issues/3754#issuecomment-2152673030

Should we disable auto-update of system packages?

That's a good question. On the one hand it would mean no unexpected surprises but the onus is then back on us to keep the systems reasonably up-to-date. Maybe we should schedule regular (monthly?) maintenance windows for applying routine package updates?

@nodejs/build-infra thoughts?

This manifests as Jenkins being unable to spawn git/bash after an automatic Java Runtime Environment update until the Jenkins agent is restarted, e.g.

09:26:33 ERROR: Error cloning remote repo 'origin'
09:26:33 hudson.plugins.git.GitException: Could not init /home/iojs/build/workspace/node-test-commit
09:26:33 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1073)
...
09:26:33 Caused by: java.io.IOException: Cannot run program "git" (in directory "/home/iojs/build/workspace/node-test-commit"): error=0, Failed to exec spawn helper: pid: 1881683, exit value: 1
09:26:33 	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1143)
...
09:26:33 Caused by: java.io.IOException: error=0, Failed to exec spawn helper: pid: 1881683, exit value: 1
09:26:33 	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)

This has affected the Ubuntu machines:

• https://github.com/nodejs/build/issues/3754
• https://github.com/nodejs/build/issues/3643

and also Debian:

• https://github.com/nodejs/build/issues/3941
• https://github.com/nodejs/build/issues/3625

Alternatively maybe there's a way to have the Systemd service running the Jenkins agent to restart if Java gets updated.