Building derivation fails when remote machine has already built it

[basvandijk]

My hydra server is configured to use rpibuild3 (a Raspberry Pi 3) as a build machine. When hydra attempts to perform a remote build on that machine I see the following repeating in my journal:

Dec 11 14:32:44 hera hydra-queue-runner[27440]: performing step ‘/nix/store/l7iibqga4l9rk7zr9nvdhccyyvd0g236-smb-dummy.conf.drv’ 1 times on ‘root@rpibuild3’ (needed by build 14 and 0 others)
Dec 11 14:32:45 hera hydra-queue-runner[27440]: outputs of ‘/nix/store/l7iibqga4l9rk7zr9nvdhccyyvd0g236-smb-dummy.conf.drv’ substituted or already valid on ‘root@rpibuild3’
Dec 11 14:32:45 hera hydra-queue-runner[27440]: killing process 28795
Dec 11 14:32:45 hera hydra-queue-runner[27440]: will disable machine ‘root@rpibuild3’ for 87s
Dec 11 14:32:45 hera hydra-queue-runner[27440]: possibly transient failure building ‘/nix/store/l7iibqga4l9rk7zr9nvdhccyyvd0g236-smb-dummy.conf.drv’ on ‘root@rpibuild3’: unexpected end-of-file
Dec 11 14:32:45 hera hydra-queue-runner[27440]: will retry ‘/nix/store/l7iibqga4l9rk7zr9nvdhccyyvd0g236-smb-dummy.conf.drv’ after 1620s

What I understand from this is that rpibuild3 has already built /nix/store/l7iibqga4l9rk7zr9nvdhccyyvd0g236-smb-dummy.conf.drv. What I don't understand is why process 28795 got killed.

Is this a bug?

[domenkozar]

unexpected end-of-file: full disk?

[basvandijk]

@domenkozar unfortunately it's not that easy. I still have 110G available.

[npry]

For anyone in the future coming across this, to debug: on the hydra machine, sudo -u hydra-queue-runner nix build -L -j0 <your_derivation>. You'll be able to see whatever underlying error occurred on the remote builder.