google
No space left on device
In OpenSSL CI, we're regularly getting errors because of "No space left on device". An example log: https://github.com/openssl/openssl/actions/runs/7811610255/job/21307017321?pr=23502
I believe the main reason is that even that we run the CIFuzz on each and every pull request the oss-fuzz project is set up in such way that it builds and runs fuzzers on all the active release branches of OpenSSL.
I understand that doing so is useful for the OSS-Fuzz runs but it is definitely not for our GH CI CIFuzz workflow. We need to somehow fix this but I have no idea how to do so. I.e., if the Dockerfile and build.sh can have some conditionals that would skip cloning and building the branches and just build and run against the GH commit from the pull request.
Sorry for the late reply. Is this still a problem for OpenSSL? The latest actions seem passing?
Hmm, yeah, that was fixed by my recent PR #11618 That saves a lot of disk space by removing build artefacts that are no longer needed any more. Now the dist usage stays in the range 80-90% for each build step. Meanwhile I found the reason why this issue started: The github actions runner have changed their configuration around 6th of February, I can see that by looking at old CI runs on oss-fuzz: new runs: e.g. https://github.com/google/oss-fuzz/actions/runs/7800673195/job/21274044506
2024-02-06T13:46:04.3265761Z Filesystem Size Used Avail Use% Mounted on
2024-02-06T13:46:04.3266631Z /dev/root 73G 53G 20G 73% /
2024-02-06T13:46:04.3267091Z tmpfs 7.9G 172K 7.9G 1% /dev/shm
2024-02-06T13:46:04.3267557Z tmpfs 3.2G 1.1M 3.2G 1% /run
2024-02-06T13:46:04.3268121Z tmpfs 5.0M 0 5.0M 0% /run/lock
2024-02-06T13:46:04.3268787Z /dev/sdb15 105M 6.1M 99M 6% /boot/efi
2024-02-06T13:46:04.3269272Z /dev/sda1 74G 4.1G 66G 6% /mnt
2024-02-06T13:46:04.3269815Z tmpfs 1.6G 12K 1.6G 1% /run/user/1001
old runs: e.g. https://github.com/google/oss-fuzz/actions/runs/7761860320/job/21171166951
2024-02-02T21:42:14.9390652Z Filesystem Size Used Avail Use% Mounted on
2024-02-02T21:42:14.9391571Z /dev/root 84G 54G 31G 64% /
2024-02-02T21:42:14.9392692Z tmpfs 7.9G 172K 7.9G 1% /dev/shm
2024-02-02T21:42:14.9393689Z tmpfs 3.2G 1.1M 3.2G 1% /run
2024-02-02T21:42:14.9394754Z tmpfs 5.0M 0 5.0M 0% /run/lock
2024-02-02T21:42:14.9395639Z /dev/sdb15 105M 6.1M 99M 6% /boot/efi
2024-02-02T21:42:14.9396363Z /dev/sda1 63G 4.1G 56G 7% /mnt
2024-02-02T21:42:14.9397080Z tmpfs 1.6G 12K 1.6G 1% /run/user/1001
but the /mnt folder only holds a swapfile and I dont see any reason why that volume size had to be increased at all.
Seems you are using clusterfuzzlite on Github free runners. If it is so, you can cleanup space on runner before running fuzzing. I recently encountered the same problem and here is a commit ^1 with a fix. TL;DR: remove unused preinstalled packages.
CIFuzz has been disabled for Suricata due to this problem cf https://github.com/google/oss-fuzz/pull/11797 to try to get it back
You can also look if this Cleanup step here helps: https://github.com/bernd-edlinger/openssl/commits/debug_cifuzz_workflow/
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 76026616 58037916 17972316 77% /
tmpfs 8182296 172 8182124 1% /dev/shm
tmpfs 3272920 1112 3271808 1% /run
tmpfs 5120 0 5120 0% /run/lock
/dev/sdb15 106832 6186 100646 6% /boot/efi
/dev/sda1 76829444 4194336 68686668 6% /mnt
tmpfs 1636456 12 1636444 1% /run/user/1001
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/root 76026616 36810956 39199276 49% /
tmpfs 8182296 172 8182124 1% /dev/shm
tmpfs 3272920 1112 3271808 1% /run
tmpfs 5120 0 5120 0% /run/lock
/dev/sdb15 106832 6186 100646 6% /boot/efi
/dev/sda1 76829444 4194336 68686668 6% /mnt
tmpfs 1636456 12 1636444 1% /run/user/1001
It delayed the workflow for a minute or so to delete all this useless stuff, but maybe that is also an alternative way out of the current situation.