No space left on macOS host?

[targos]

trafficstars

See https://ci.nodejs.org/job/node-test-commit-osx/44401/nodes=osx1015/#showFailuresLink

[richardlau]

🤷 There have been three node-test-commit-osx builds on the same machine, https://ci.nodejs.org/computer/test-nearform-macos10.15-x64-3/builds, since then which all passed:

And

test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   25Gi   15Gi    63%  309782 536799578    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

[richardlau]

hmm. https://ci.nodejs.org/job/node-test-commit-osx/44429/nodes=osx1015/console also ran out of space on the same machine, test-nearform-macos10.15-x64-3.

13:54:44 fatal error: /Library/Developer/CommandLineTools/usr/bin/libtool: can't write to output file (No space left on device)
13:55:20 make[2]: *** [/Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/out/Release/libv8_initializers.a] Error 1

Again we have around 13G of free space.

test-nearform-macos10:~ iojs$ df -h
Filesystem      Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s5    51Gi   10Gi   13Gi    44%  487478 536621882    0%   /
devfs          186Ki  186Ki    0Bi   100%     642         0  100%   /dev
/dev/disk1s1    51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
/dev/disk1s4    51Gi  1.0Gi   13Gi     8%       2 537109358    0%   /private/var/vm
map auto_home    0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
test-nearform-macos10:~ iojs$ df -h /Users/iojs/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /Users/iojs/build/workspace/
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$ df -h /tmp
Filesystem     Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk1s1   51Gi   26Gi   13Gi    67%  357543 536751817    0%   /System/Volumes/Data
test-nearform-macos10:~ iojs$

The test-commit job does a git clean at the end. So somehow the job is chewing through 13G while running? (!)

[richardlau]

@AshCripps do you know if it's normal for /dev and /System/Volumes/Data/home to be 100% used on macs?

[AshCripps]

im going to say no, /System/Volumes/Data is the map of the user directory IIRC so its likely is a full disk. Catalina changed this reporting into a whole weird mess.

[tniessen]

So far, all three CI runs on https://github.com/nodejs/node/pull/42628 failed with ENOSPC on osx.

[richardlau]

I'm just more and more confused:

[sxa]

FYI the builds really are using about 14GB while in flight: During build (while running tests):

/dev/disk1s1    53710936 39596968   1474908    97%  284892 536824468    0%   /System/Volumes/Data

After build (Workspace cleared up):

Filesystem    1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s5     53710936 10873180  14855648    43%  487478 536621882    0%   /

Some of the files in out/Release are pretty big - these two account for 5GB alone;

3004068	libv8_base_without_compiler.a
2001752	libv8_initializers.a

and (probably obviously from the above) obj.target is quite large at over 6Gb

So we need to either get these expanded or find something else to delete. The disks are showing as around 51Gb at the moment. /var/db is quite large at about 8.7Gb but I don't know whether anything can be done about that. ccache for the iojs user is using 4.5Gb

[sxa]

Note - the -1 machine has a slightly larger disk than -2 and -3:

[sxa@sainz .ssh]$ for A in 1 2 3; do ssh test-nearform-macos10.15-x64-$A df -k .; done
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    68359372 26111720  29608124    47%  293179 683300541    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 24257204  16802816    60%  296611 536812749    0%   /System/Volumes/Data
Filesystem   1024-blocks     Used Available Capacity iused     ifree %iused  Mounted on
/dev/disk1s1    53710936 39309828   1762048    96%  284926 536824434    0%   /System/Volumes/Data
[sxa@sainz .ssh]$ 

[AshCripps]

Note these machines cant really be any bigger as the host they are on started hitting issues when the VMs were a bigger size. (This was the roughly the largest size that was stable)

[sxa]

issues when the VMs were a bigger size.

Is that disk size specifically? Any idea what the issues were? I could understand if RAM/CPU allocation was overdone it would lead to issues but disk seems a little odd.

[AshCripps]

I think it was due to apples autoupdate feature meaning the host needed enough headroom other it crashes - im sure there was an issue on this in the build repo.

[richardlau]

Jenkins has automatically taken test-nearform-macos10.15-x64-3 offline due to low disk space.

We're now also seeing disk space issues on test-orka-macos11-x64-2, e.g. https://ci.nodejs.org/job/node-test-commit-osx/44542/nodes=osx11-x64/console

10:55:10 not ok 3236 pummel/test-fs-readfile-tostring-fail
10:55:10   ---
10:55:10   duration_ms: 0.203
10:55:10   severity: fail
10:55:10   exitcode: 7
10:55:10   stack: |-
10:55:10     /Users/iojs/build/workspace/node-test-commit-osx/nodes/osx11-x64/test/pummel/test-fs-readfile-tostring-fail.js:69
10:55:10       throw err;
10:55:10       ^
10:55:10     
10:55:10     [Error: ENOSPC: no space left on device, write] {
10:55:10       errno: -28,
10:55:10       code: 'ENOSPC',
10:55:10       syscall: 'write'
10:55:10     }
10:55:10     
10:55:10     Node.js v18.0.0-pre
10:55:10   ...

[richardlau]

For test-orka-macos11-x64-2 I've removed /Users/iojs/build/workspace/node-test-commit-osx-as-shared-mdawson which was 1.5GB.

[richardlau]

test-orka-macos11-x64-2:

administrator@test-orka-macos11-x64-2 ~ % df -h
Filesystem       Size   Used  Avail Capacity iused     ifree %iused  Mounted on
/dev/disk2s5s1   90Gi   14Gi  1.8Gi    89%  553788 941116212    0%   /
devfs           189Ki  189Ki    0Bi   100%     653         0  100%   /dev
/dev/disk2s4     90Gi  1.0Gi  1.8Gi    36%       2 941669998    0%   /System/Volumes/VM
/dev/disk2s2     90Gi  305Mi  1.8Gi    15%    1038 941668962    0%   /System/Volumes/Preboot
/dev/disk2s6     90Gi  440Ki  1.8Gi     1%      18 941669982    0%   /System/Volumes/Update
/dev/disk2s1     90Gi   72Gi  1.8Gi    98%  455861 941214139    0%   /System/Volumes/Data
map auto_home     0Bi    0Bi    0Bi   100%       0         0  100%   /System/Volumes/Data/home
administrator@test-orka-macos11-x64-2 ~ %

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /System/Volumes/Data/*
 18M    /System/Volumes/Data/Applications
du: /System/Volumes/Data/Library/Caches/com.apple.aned: Operation not permitted
2.5G    /System/Volumes/Data/Library
271M    /System/Volumes/Data/Previous Content
3.2G    /System/Volumes/Data/System
 21G    /System/Volumes/Data/Users
  0B    /System/Volumes/Data/Volumes
266M    /System/Volumes/Data/boot
  0B    /System/Volumes/Data/cores
1.0K    /System/Volumes/Data/home
  0B    /System/Volumes/Data/mnt
  0B    /System/Volumes/Data/opt
du: /System/Volumes/Data/private/var/networkd/db: Operation not permitted
du: /System/Volumes/Data/private/var/db/appinstalld: Operation not permitted
du: /System/Volumes/Data/private/var/db/fpsd/dvp: Operation not permitted
du: /System/Volumes/Data/private/var/db/installcoordinationd: Operation not permitted
du: /System/Volumes/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/private/var/db/lockdown: Operation not permitted
du: /System/Volumes/Data/private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /System/Volumes/Data/private/var/db/DifferentialPrivacy: Operation not permitted
du: /System/Volumes/Data/private/var/db/MobileIdentityService: Operation not permitted
du: /System/Volumes/Data/private/var/db/searchparty: Operation not permitted
du: /System/Volumes/Data/private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /System/Volumes/Data/private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/protected/trustd/private: Operation not permitted
3.8G    /System/Volumes/Data/private
  0B    /System/Volumes/Data/sw
943M    /System/Volumes/Data/usr
administrator@test-orka-macos11-x64-2 ~ %

21G in /System/Volumes/Data/Users is mostly the builds (workspace and ccache):

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/iojs/build/workspace/node-test-commit-osx
 16G    /Users/iojs/build/workspace/node-test-commit-osx
administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/iojs/Library/Caches/ccache
4.1G    /Users/iojs/Library/Caches/ccache

but that leaves ~50G unaccounted for 😞.

[AshCripps]

you are probably best off not looking in /System/Volumes/Data as it should be just a mirror that you cant access of filesystem, just regular checks under / and /Users should help to find the data. Besides that I wonder if its started downloading an os update as they can be fairly hefty.

[richardlau]

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /*
 18M    /Applications
du: /Library/Caches/com.apple.aned: Operation not permitted
2.5G    /Library
du: /System/Library/Templates/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/Library/Caches/com.apple.aned: Operation not permitted
du: /System/Volumes/Data/private/var/networkd/db: Operation not permitted
du: /System/Volumes/Data/private/var/db/appinstalld: Operation not permitted
du: /System/Volumes/Data/private/var/db/fpsd/dvp: Operation not permitted
du: /System/Volumes/Data/private/var/db/installcoordinationd: Operation not permitted
du: /System/Volumes/Data/private/var/db/oah: Operation not permitted
du: /System/Volumes/Data/private/var/db/lockdown: Operation not permitted
du: /System/Volumes/Data/private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /System/Volumes/Data/private/var/db/DifferentialPrivacy: Operation not permitted
du: /System/Volumes/Data/private/var/db/MobileIdentityService: Operation not permitted
du: /System/Volumes/Data/private/var/db/searchparty: Operation not permitted
du: /System/Volumes/Data/private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /System/Volumes/Data/private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /System/Volumes/Data/private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /System/Volumes/Data/private/var/protected/trustd/private: Operation not permitted
 38G    /System
7.0G    /Users
  0B    /Volumes
4.6M    /bin
  0B    /cores
4.5K    /dev
  0B    /etc
  0B    /home
  0B    /opt
du: /private/var/networkd/db: Operation not permitted
du: /private/var/db/appinstalld: Operation not permitted
du: /private/var/db/fpsd/dvp: Operation not permitted
du: /private/var/db/installcoordinationd: Operation not permitted
du: /private/var/db/oah: Operation not permitted
du: /private/var/db/lockdown: Operation not permitted
du: /private/var/db/KernelExtensionManagement/Staging: Operation not permitted
du: /private/var/db/DifferentialPrivacy: Operation not permitted
du: /private/var/db/MobileIdentityService: Operation not permitted
du: /private/var/db/searchparty: Operation not permitted
du: /private/var/db/CoreDuet/Knowledge: Operation not permitted
du: /private/var/db/ConfigurationProfiles/Store: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.progressd/ClassKit: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/dmd: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.SharedWebCredentials: Operation not permitted
du: /private/var/folders/hb/6bmy20jj5tz9ws4l4c81_nxc0000gn/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.progressd/ClassKit: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.exchangesync: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/dmd: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.SharedWebCredentials: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/lj/xrn01fc14_3cy6rk6mq9bvb80000gq/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /private/var/folders/7l/fb92_ds12k19tryhky7zbfnw0000gp/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/0: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000sm00006d/C: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000y800007k/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.ScreenTimeAgent/Store: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.lockoutagent: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/dmd: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.nsurlsessiond: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/0/com.apple.routined/dv: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n00000z000007r/C/com.apple.quicklook.ThumbnailsAgent/com.apple.QuickLook.thumbnailcache: Operation not permitted
du: /private/var/folders/zz/zyxvpxvq6csfxvn_n0000000000000/0/com.apple.LaunchServices.dv: Operation not permitted
du: /private/var/protected/trustd/private: Operation not permitted
3.8G    /private
2.0M    /sbin
  0B    /tmp
1.5G    /usr
  0B    /var
administrator@test-orka-macos11-x64-2 ~ %

administrator@test-orka-macos11-x64-2 ~ % sudo du -hs /Users/*
1.5M    /Users/Shared
568M    /Users/administrator
6.4G    /Users/iojs
administrator@test-orka-macos11-x64-2 ~ %

🤷

[targos]

Can you use the GUI somehow? Maybe macOS tools (for example the "Storage" tab, in "About This Mac") will be able to find what's taking the remaining space.

[richardlau]

I have no idea. The internal Build docs say the Orka machines have VNC disabled but then give a username/password for screen sharing. I do not have a mac device and only have access to Windows and Linux locally.

[joyeecheung]

If we are unable to find what's occupying the disk space, is it possible to just erase the disk and set up the machines from scratch again?

[sxa]

Yeah the mac VNC client is a bit differnet from the others. I've been able to use realvnc to connect to a mac from another platform but in general it's a bit easier from a mac client :-)

Having said that I cannot establish a connection to the VNC server on those machines even when on the private network.

[sxa]

By the time I logged into to have a look there seemed to be 16Gb of space free so it's probably ok now again.

[richardlau]

By the time I logged into to have a look there seemed to be 16Gb of space free so it's probably ok now again.

I think that's because https://github.com/nodejs/build/issues/2917#issuecomment-1099098543 freed up enough space for node-test-commit-osx to run successfully (seems to need around the 16G mark to run). node-test-commit-osx runs make distclean at the end to reclaim disk space used for the compiled artifacts.

[tniessen]

osx1015 is failing tests with ENOSPC again: https://ci.nodejs.org/job/node-test-commit-osx/44792/nodes=osx1015/

[tniessen]

Again: https://ci.nodejs.org/job/node-test-commit-osx/45454/nodes=osx11-x64/

[targos]

Again: https://ci.nodejs.org/job/node-test-commit-osx/47635/nodes=osx1015/

[AshCripps]

So ive had a little poke around on nearform-2 (the problem machine targos linked above) and have managed to free up some space with a simple reboot. I believe the root of our issue is the bloat of /System/Volumes/Data/private which is a temporary cache of sorts mac uses (hence why a reboot free a GB or so) but its still at 6GB. This needs some investigation into why its getting so bloated - it seems you can just blow away the folder but that runs the risk of bricking the machine depending on whats using it so would need to be done when there is time to rebuild it with no issue.

The big problem with the nearform x64 machines is they are 2 vms hosted on a physical mac mini which means we need space for 3 os's all our tools and builds on only 200GB of disk space - there is around 40GB of head room on the host itself so we can give some leeway but if /private is just going to keep bloating thats not much help.

I wil try to find a solution for this file bloat in the meantime but if this keeps happening I will increase the size of the VMs as a stop gap.

[targos]

Again:

• https://ci.nodejs.org/job/node-test-commit-osx/47902/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/
• https://ci.nodejs.org/job/node-test-commit-osx/47899/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[targos]

https://ci.nodejs.org/job/node-test-commit-osx/47935/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[aduh95]

https://ci.nodejs.org/job/node-test-commit-osx/48075/nodes=osx11-x64/testReport/junit/(root)/test/pummel_test_fs_readfile_tostring_fail/

[targos]

https://ci.nodejs.org/job/node-test-commit-osx/48110/nodes=osx1015/testReport/junit/(root)/test/pummel_test_fs_largefile/