No space left on test-digitalocean-ubuntu1804-x64-1

[targos]

https://ci.nodejs.org/job/citgm-smoker-nobuild/1103/nodes=ubuntu1804-64/ https://ci.nodejs.org/job/citgm-smoker-nobuild/1102/nodes=ubuntu1804-64/ https://ci.nodejs.org/job/citgm-smoker-nobuild/1101/nodes=ubuntu1804-64/

[richardlau]

We have around 3G free:

iojs@test-digitalocean-ubuntu1804-x64-1:~$ df -h  .
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   21G  3.1G  88% /

Space usage:

iojs@test-digitalocean-ubuntu1804-x64-1:~$ du -hs /home/iojs/
15G     /home/iojs/
iojs@test-digitalocean-ubuntu1804-x64-1:~$ du -hs .[^.]* *
134M    .babel.json
4.0K    .bash_history
4.0K    .bash_logout
4.0K    .bashrc
5.4G    .cache
1.7G    .ccache
0       .cloud-locale-test.skip
56K     .config
4.0K    .gitconfig
482M    .jenkins
20K     .local
12K     .node-clinic-rc
4.0K    .node-gyp
4.0K    .node-spawn-wrap-
0       .node_repl_history
2.8M    .npm
4.0K    .npmrc
4.0K    .profile
4.0K    .rnd
8.0K    .ssh
24M     .standard-cache
75M     .standard-v12-cache
3.6M    .standard-v13-cache
5.5M    .standard-v14-cache
8.0K    .viminfo
5.1M    .xo-cache
8.0K    .yarn
4.0K    .yo-rc-global.json
231M    acmeair
39M     acmeair-zips-now.tar
6.6G    build
64K     hs_err_pid830.log
20M     jenkins_diagnostics.txt
4.0K    node-icu
468K    node-tmp
1.5M    slave.jar
39M     tmp
iojs@test-digitalocean-ubuntu1804-x64-1:~$

5.4 G in .cache seems a lot:

iojs@test-digitalocean-ubuntu1804-x64-1:~$ du -hs .cache/*
8.0K    .cache/JNA
180M    .cache/electron
36K     .cache/fontconfig
76K     .cache/js-v8flags
4.0G    .cache/node-gyp
1.2G    .cache/yarn
iojs@test-digitalocean-ubuntu1804-x64-1:~$

.cache/node-gyp is full of directories for nightly builds going back as far as Node.js 13.

I've removed all of .cache.

iojs@test-digitalocean-ubuntu1804-x64-1:~$ rm -rf /home/iojs/.cache/*
iojs@test-digitalocean-ubuntu1804-x64-1:~$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        25G   16G  8.5G  65% /
iojs@test-digitalocean-ubuntu1804-x64-1:~$

[richardlau]

Given the nightlies in .cache/node-gyp I suspect it'd be something like the node-test-node-addon-api* jobs in https://ci.nodejs.org/view/x%20-%20Abi%20stable%20module%20API/

I think it's perfectly valid to cache headers for recent nightlies, but we probably need a cleanup script of some kind to clean up older ones.

[richardlau]

To clarify, it's not only the node-addon-api jobs that build addons/cause node-gyp to cache headers, but they are the ones that run most frequently (i.e. daily against the nightlies). CITGM will also do so but is run much less often (and hardly at all against nightlies).

Metrics: Each headers package for a version of Node is around 7 MB (a large proportion of that is openssl headers for the various archstectures). At one nightly per day that's (7×365) 2555 MB or 2.5 G per year.

[richardlau]

Seems to have reoccurred https://github.com/nodejs/build/issues/2877.

3.8G    .cache

Also of note it appears that the headers package is now much larger:

7.0M    .cache/node-gyp/17.0.0-nightly20211008dbc91de011
7.0M    .cache/node-gyp/17.0.0-nightly20211009879ff775d0
7.0M    .cache/node-gyp/17.0.0-nightly202110108068f40313
67M     .cache/node-gyp/17.0.0-nightly202110110cc37209fa
67M     .cache/node-gyp/17.0.0-nightly20211012a784258444
67M     .cache/node-gyp/17.0.0-nightly20211013f570109094

A cursory glance at the changes https://github.com/nodejs/node/compare/8068f40313...0cc37209fa would suggest that the large increase in size was the OpenSSL 3 switchover.

[rvagg]

@danbev I think you might be the right person to help deal with this:

root@test-digitalocean-ubuntu1804-x64-1:/home/iojs/.cache/node-gyp/18.0.0-nightly20220222c071bd581a/include/node/openssl/archs# ls
BSD-x86     VC-WIN64-ARM  aix64-gcc       darwin64-arm64-cc   linux-armv4  linux-ppc64    linux32-s390x    linux64-s390x
BSD-x86_64  VC-WIN64A     aix64-gcc-as    darwin64-x86_64-cc  linux-elf    linux-ppc64le  linux64-mips64   solaris-x86-gcc
VC-WIN32    aix-gcc       darwin-i386-cc  linux-aarch64       linux-ppc    linux-x86_64   linux64-riscv64  solaris64-x86_64-gcc
root@test-digitalocean-ubuntu1804-x64-1:/home/iojs/.cache/node-gyp/18.0.0-nightly20220222c071bd581a/include/node/openssl/archs# du -sh .
61M     .

Also https://nodejs.org/download/release/latest-v16.x/

node-v16.14.0-headers.tar.gz                       08-Feb-2022 12:46              560771

vs

https://nodejs.org/download/release/latest-v17.x/

node-v17.6.0-headers.tar.gz                        22-Feb-2022 22:38             9923593

on the latest 18.x nightly it's even up to 10028309 bytes now! These files were supposed to be small and lean for node-gyp to download and use to compile addons. They're now the size of 0.12 binary tarballs!

[danbev]

@danbev I think you might be the right person to help deal with this:

I'm currently on a leave of absence and will be back working in April, and would be happy to take a closer look at this then (unless someone else beats me to it).

[richardlau]

Hit the problem on test-digitalocean-fedora34-x64-2 (reported by @BethGriggs) during https://ci.nodejs.org/job/citgm-smoker/nodes=fedora-latest-x64/2935/. e.g. https://ci.nodejs.org/job/citgm-smoker/nodes=fedora-latest-x64/2935/testReport/junit/(root)/citgm/npm_v8_10_0/

 npm ERR! code ENOSPC
 npm ERR! syscall write
 npm ERR! errno -28
 npm ERR! nospc ENOSPC: no space left on device, write
 npm ERR! nospc There appears to be insufficient space on your system to finish.
 npm ERR! nospc Clear up some disk space and try again.
 npm ERR! A complete log of this run can be found in:
 npm ERR!     /home/iojs/tmp/citgm_tmp/567e7901-60a6-45c5-9c5a-9b07d9f3dc40/home/.npm/_logs/2022-05-16T16_41_35_684Z-debug-0.log

/home/iojs/.cache/node-gyp was 7G, mostly from nightlies.

[root@test-digitalocean-fedora34-x64-2 iojs]# du -hs /home/iojs/.cache/node-gyp
7.0G    /home/iojs/.cache/node-gyp
[root@test-digitalocean-fedora34-x64-2 iojs]# find /home/iojs/.cache/node-gyp/ -type d -name "*nightly*" | xargs rm -rf
[root@test-digitalocean-fedora34-x64-2 iojs]# du -hs /home/iojs/.cache/node-gyp
461M    /home/iojs/.cache/node-gyp
[root@test-digitalocean-fedora34-x64-2 iojs]#

To reiterate https://github.com/nodejs/build/issues/2695#issuecomment-870534806, we'll need to come up with a script/cleanup job to periodically clear out the cached headers.

[github-actions[bot]]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.