Out of space on www server

[richardlau]

Reported by @RafaelGSS when trying to promote 19.5.0:

# Promoting v19.5.0...
... Checking assets
... Expecting a total of 39 assets for v19.x
... 39 assets waiting in staging
... 0 assets already promoted
 ✓  Complete set of expected assets in place for v19.x
Are you sure you want to promote the v19.5.0 assets? [y/n] y
Promoting v19.5.0/docs...
...
Promoting v19.5.0/win-x86...
/home/nodejs/queue-cdn-purge.sh: line 13: echo: write error: No space left on device

This is a different disk (/dev/vda1) to https://github.com/nodejs/build/issues/3125 and is not the disk that holds the downloads.

Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  3.8M  1.6G   1% /run
/dev/vda1       315G  302G     0 100% /
tmpfs           7.9G   76K  7.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda        3.9T  1.7T  2.0T  47% /home
/dev/sdb        2.0T  1.8T   27G  99% /mnt/nodejs_org_home_old
tmpfs           1.6G     0  1.6G   0% /run/user/0

[richardlau]

I'm guessing we might want to look at /var/log:

# du -hs /var/*
12M     /var/backups
123M    /var/cache
4.0K    /var/crash
101G    /var/lib
4.0K    /var/local
0       /var/lock
181G    /var/log
4.0K    /var/mail
4.0K    /var/opt
0       /var/run
4.0K    /var/snap
28K     /var/spool
36K     /var/tmp
12K     /var/www

[RafaelGSS]

Even after freeing it, I think we might need to clean up all v19.5.0 artifacts present. Otherwise, the release script will return:

# Checking for releases ...
No releases to promote!

[richardlau]

@RafaelGSS Once we fix the space problem I think you can rerun the release script with -s:

# To _only_ sign an existing release, use:
#  release.sh -s vx.y.z

The binaries were moved from staging to the downloads, https://nodejs.org/download/release/v19.5.0/ exists (but has no shasums).

[RafaelGSS]

Are all the binaries there? from the logs looks like some of them might be conflicted

[richardlau]

The largest logs are nginx:

# du -hs /var/log/*/
108K    /var/log/apt/
4.0K    /var/log/cloudflare/
4.0K    /var/log/dist-upgrade/
4.0K    /var/log/fsck/
4.0K    /var/log/iptraf/
4.1G    /var/log/journal/
160K    /var/log/letsencrypt/
4.0K    /var/log/lxd/
176G    /var/log/nginx/
4.0K    /var/log/telegraf/
12K     /var/log/unattended-upgrades/

of which

# du -hs /var/log/nginx/*/
35M     /var/log/nginx/benchmarking/
449M    /var/log/nginx/coverage/
33G     /var/log/nginx/iojs/
128M    /var/log/nginx/libuv/
136G    /var/log/nginx/nodejs/
5.0G    /var/log/nginx/nodejs-backup/
2.3G    /var/log/nginx/old/

[RafaelGSS]

Are all the binaries there? from the logs looks like some of them might be conflicted

I've checked the content size of the binaries and it looks ok. So, when it's freed let me know to sign the release. Thank you!

[richardlau]

@RafaelGSS They're no longer in staging. Both staging and downloads are under /home which has 2 TB of free space.

[richardlau]

I've removed /root/nightly (https://github.com/nodejs/build/issues/3125#issuecomment-1358036749) which has recovered 16G while we work out what to do about /var/log/.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.9G     0  7.9G   0% /dev
tmpfs           1.6G  3.7M  1.6G   1% /run
/dev/vda1       315G  287G   16G  95% /
tmpfs           7.9G   76K  7.9G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           7.9G     0  7.9G   0% /sys/fs/cgroup
/dev/sda        3.9T  1.7T  2.0T  47% /home
/dev/sdb        2.0T  1.8T   27G  99% /mnt/nodejs_org_home_old
tmpfs           1.6G     0  1.6G   0% /run/user/0

[RafaelGSS]

I was able to perform the release, thanks again @richardlau!

[richardlau]

I'll keep this issue open so we can decide how to handle the nginx logs.

[github-actions[bot]]

This issue is stale because it has been open many days with no activity. It will be closed soon unless the stale label is removed or a comment is made.

[sxa]

Ping to keep this alive