Artifact downloading from Hydra buffers the whole artifact into RAM before sending it

[delroth]

https://github.com/NixOS/hydra/blob/c1bd50a80db45930cdfc1975903d1f76cca1053a/src/lib/Hydra/Controller/Build.pm#L239

The way Hydra artifacts downloads works is that it runs "nix store cat", reads the whole contents into RAM, and then sends it as HTTP response. This has multiple issues for larger files (e.g. disk images):

• Very long time to first byte, which could cause some clients to time out.
• Very high RAM usage by the Hydra web workers.
• Significant decrease in throughput, since the transfer time happens ~ 2x.

We noticed it a few weeks ago on #infra:nixos.org but then this got rediscovered again today on #hydra:nixos.org so I thought it might be worth filing a bug to actually track this.

[delroth]

This sometimes ends up hogging all workers on h.n.o, causing the website to become unavailable and channel scripts to fail.