compress_build_logs and upload_logs_to_binary_cache are mutually exclusive and badly conflict with one another

[delroth]

Describe the bug

Both the compress_build_logs and upload_logs_to_binary_cache config options try to exclusively handle log files fetched by Hydra from its builders. Both do different things with the log file, but both unlink() the original file once they're done, so it should be pretty obvious they can never work together.

This is especially bad since compress_build_logs is enabled by default.

To Reproduce Enable both settings, notice that some logs are not uploaded to the cache, and notice that hydra-notify fails to bzip2 a large percentage of logs.

Expected behavior Either make those not conflict, or error out when this configuration is found.

Screenshots n/a

Hydra Server:

hydra.nixos.org

Additional context This is likely the reason why hydra.nixos.org has had randomly missing logs for years. We've found 72M stranded .bz2 log files on the local drive of the Hydra server, which are all log files that likely haven't been uploaded to S3...