nix
nix copied to clipboard
NixOS
SSL certificate path does not propagate to builds during `nix-build`
Describe the bug
Fetchers such as fetchzip rely on curl which by default will try to verify SSL certificates. When running behind an intercepting proxy, curl must either be run with --insecure, or it must be given a certificate bundle containing the self-signed certificate, e.g. through the CURL_CA_BUNDLE environment variable. It seems this information does not propagate to the build process through the nix daemon.
The documentation suggests the path to the SSL certificate bundle should propagate properly when setting NIX_SSL_CERT_FILE. I accordingly adjusted environment.variables. Indeed, systemctl show nix-daemon | grep Environment now shows correct entries for NIX_SSL_CERT_FILE and even CURL_CA_BUNDLE, but the build fails as described below.
Steps To Reproduce
- Run behind an intercepting corporate proxy, and add self-signed certificates to
security.pki.certificates - Run cntlm
users.users.cntlm.group = "cntlm"; users.users.cntlm.isSystemUser = true; users.groups.cntlm = {}; services.cntlm = { enable = true; configText = <-- snip config and credentials -->; }; networking.proxy.default = "http://127.0.0.1:3128/"; networking.proxy.noProxy = "127.0.0.*,localhost,<-- snip internal sites -->"; - Verify that your proxy works, and curl succeeds if and only if
CURL_CA_BUNDLEis set properly> nix-shell --pure -p curl [nix-shell:~/projects/nix-pkg-test]$ curl -I https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz HTTP/1.1 200 Connection established Proxy-agent: netentsec Connection: close curl: (77) error setting certificate file: /no-cert-file.crt [nix-shell:~/projects/nix-pkg-test]$ CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt curl -I https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz HTTP/1.1 200 Connection established Proxy-agent: netentsec Connection: close HTTP/1.1 200 OK Date: Mon, 27 May 2024 11:59:24 GMT Server: Apache/2.4.29 (Trisquel_GNU/Linux) Strict-Transport-Security: max-age=63072000 Last-Modified: Sun, 29 May 2022 23:05:08 GMT ETag: "fc451-5e02e9229e6eb" Accept-Ranges: bytes Content-Length: 1033297 Content-Security-Policy: default-src 'self'; img-src 'self' https://static.fsf.org https://static.gnu.org https://gnu.org http://static.fsf.org http://static.gnu.org http://gnu.org http://gnu.org; object-src 'none'; frame-ancestors 'none'; child-src 'self' https://static.gnu.org http://static.fsf.org http://static.gnu.org http://gnu.org; X-Frame-Options: DENY X-Content-Type-Options: nosniff Content-Type: application/x-gzip - Build a basic package using
fetchzip,fetchFromGitHub, or similar (nofetchTarball)let pkgs = import <nixpkgs> {}; in { myhello = pkgs.stdenv.mkDerivation { pname = "hello"; version = "0.5"; src = pkgs.fetchzip { url = "https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz"; sha256 = ""; }; }; } - See error
> nix-build -A myhello warning: found empty hash, assuming 'sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=' these 2 derivations will be built: /nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv /nix/store/mj82j0ljhn3ibs0lyqzy571igq881yia-hello-0.5.drv building '/nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv'... error checking the existence of https://tarballs.nixos.org/sha256/: curl: (60) SSL certificate problem: self-signed certificate in certificate chain More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. trying https://ftp.gnu.org/gnu/hello/hello-2.12.1.tar.gz % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 curl: (60) SSL certificate problem: self-signed certificate in certificate chain More details here: https://curl.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. error: cannot download source from any mirror error: builder for '/nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv' failed with exit code 1; last 10 log lines: > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 > curl: (60) SSL certificate problem: self-signed certificate in certificate chain > More details here: https://curl.se/docs/sslcerts.html > > curl failed to verify the legitimacy of the server and therefore could not > establish a secure connection to it. To learn more about this situation and > how to fix it, please visit the web page mentioned above. > error: cannot download source from any mirror For full logs, run 'nix log /nix/store/8n9l4kbrbjgn8b6f1j0aksi180zyfbgk-source.drv'. error: 1 dependencies of derivation '/nix/store/mj82j0ljhn3ibs0lyqzy571igq881yia-hello-0.5.drv' failed to build
Expected behavior
Succeeding build.
nix-env --version output
nix-env (Nix) 2.18.1
Additional context
The fact that the fetchers don't consider system certificates IMO is a bug in itself. They wrongly use ${cacert}/etc/ssl/certs/ca-bundle.crt which gives rise to this issue in the first place (explanation). Nevertheless, NIX_SSL_CERT_FILE should still override it.
Priorities
Add :+1: to issues you find important.
I wonder if behavior changes with --option ssl-cert-file /path/to/bundle (it might not, since I think it takes effect in the same way NIX_SSL_CERT_FILE does, but worth a shot...)
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/nix-build-curl-issues-behind-proxy/45911/5
Fetchers such as
fetchzip
since fetchzip is not a nix builtin but implemented in nixpkgs (based on fetchurl), this issue should perhaps be moved to the nixpkgs repo instead?
this issue should perhaps be moved to the nixpkgs repo instead?
I thought it might also be a Nix (daemon?) issue that it kills NIX_SSL_CERT_FILE or something, because I experienced this with multiple fetchers, but I don't actually know. What do you think?
What do you think?
Hmm, indeed curl in nixpkgs does reference NIX_SSL_CERT_FILE in a comment - not sure how this is intended to fit together.
https://github.com/NixOS/nix/issues/3155 and https://discourse.nixos.org/t/nix-cannot-find-my-custom-ssl-certificate/27361/5 might also be relevant.
yeah, I saw those issues and already applied the Environment fix, by adding NIX_SSL_CERT_FILE to systemd.services.nix-daemon.serviceConfig.Environment, and checking systemctl show nix-daemon | grep Environment, whose output correctly includes
CURL_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt
NIX_SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
A workaround when using curl through fetchurl is to set NIX_CURL_FLAGS in the nix-daemon environment [1] [2]. On NixOS in configuration.nix you can do something like this:
systemd.services.nix-daemon = {
enable = true;
serviceConfig = {
Environment = [
"NIX_CURL_FLAGS=\"--cacert /etc/ssl/certs/ca-certificates.crt\""
];
};
};
which seems to work in my setup. Still, I think NIX_SSL_CERT_FILE should be fixed and used instead.
edit: this workaround seems to work with nix-build but not with nixos-rebuild on NixOS which runs into the original error
Triaged in Nix maintainers meeting:
- This is a documentation issue, partly in the Nix manual, largely in the Nixpkgs manual.
This issue has been mentioned on NixOS Discourse. There might be relevant details there:
https://discourse.nixos.org/t/2024-05-29-nix-team-meeting-minutes-148/46195/1
Another hacky workaround is utilize the extra certificate arguments of the cacert package. This way, your certificates will also end up in ${cacert}/etc/ssl/certs/ca-bundle.crt. On NixOS, you can do
nixpkgs.overlays = [
(final: prev: {
cacert = prev.cacert.overrideAttrs (old: {
extraCertificateStrings=[your_certificates];
});
})
];
Since almost everything on your system transitively depends on cacert, nixos-rebuild will re-compile most of NixOS from scratch. I had to pass --max-jobs 1 in order to not run out of memory, and this took more than a day.
Still, I think NIX_SSL_CERT_FILE should be fixed and used instead.
I'm bumping into this issue and I wholeheartedly agree. Currently not running NixOS, but a Nix-based developer environment on Fedora with a custom certificate and I am bumping into the same issue described here. A fix for both NixOS and non-NixOS users would be ideal.
Did anybody find a solution how this works for git?
In my gitconfig outside of the Nix world, I would put somehting like:
[http]
sslCAInfo = /etc/nix/macos-keychain.crt
How would I propagate this behavior to the git fetcher in nix? I am already using the determinate systems nix installer, which does a great job to make nix more enterprise friendly setting:
NIX_SSL_CERT_FILE=/etc/nix/macos-keychain.crt
SSL_CERT_FILE=/etc/nix/macos-keychain.crt
Also just happened to me when using fetchFromGitea with a local server that has a custom cert.
The CURL_CA_BUNDLE workaround helped. Deployment is started with colmena
