nix icon indicating copy to clipboard operation
nix copied to clipboard

Download error 28 while updating channels or installing packages

Open LinuxUser393 opened this issue 7 months ago • 10 comments

Description of the bug

While updating the nix channel on my machine it fails when trying to resolve the cache. I then tried installing a package and the same error for a different cache.nixos.org url was returned. I then left it for a while to see if it was either an issue with my connection or an update on my OS, but I am trying to upgrade to 25.05 from 24.11 and I am encountering the same issue on two different connections.

Steps to reproduce

I am using the following:

  • Ubuntu 22.04
  • Kernel 6.8.0
  • Lenovo ThinkPad T14
  • Nix 2.29.0 (Multi-User Install)
  • System version of cURL: 7.81.0
Result of `nix-channel --update`

    warning: ignoring the client-specified setting 'download-attempts', because it is a restricted setting and you are not a trusted user
    warning: ignoring the client-specified setting 'stalled-download-timeout', because it is a restricted setting and you are not a trusted user
    unpacking 1 channels...
    warning: ignoring the client-specified setting 'download-attempts', because it is a restricted setting and you are not a trusted user
    warning: ignoring the client-specified setting 'stalled-download-timeout', because it is a restricted setting and you are not a trusted user
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 260 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 583 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 1119 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 2059 ms
    error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds
    error: program '/nix/store/3qh0pcp0wvqc2gd5qzir4mlfcgwmdfyy-nix-2.29.0/bin/nix-env' failed with exit code 1
Result of `nix-channel -vvvvvvvvvv --update`

    using cache entry 'file:{"name":"nixpkgs-25.05-darwin","store":"/nix/store","url":"https://nixos.org/channels/nixpkgs-25.05-darwin"}' -> '{"etag":"\"64f5583ce38603173c0d41607c8d7261\"","storePath":"690c25wh3r41zxzvzw5gmb61wn6js28b-nixpkgs-25.05-darwin","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae"}'
    warning: ignoring the client-specified setting 'download-attempts', because it is a restricted setting and you are not a trusted user
    warning: ignoring the client-specified setting 'stalled-download-timeout', because it is a restricted setting and you are not a trusted user
    performing daemon worker op: 11
    acquiring write lock on '/nix/var/nix/temproots/13199'
    performing daemon worker op: 1
    using cache entry 'file:{"name":"nixpkgs-25.05-darwin","store":"/nix/store","url":"https://nixos.org/channels/nixpkgs-25.05-darwin"}' -> '{"etag":"\"64f5583ce38603173c0d41607c8d7261\"","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae"}', '/nix/store/690c25wh3r41zxzvzw5gmb61wn6js28b-nixpkgs-25.05-darwin'
    using cache entry 'file:{"name":"nixexprs.tar.xz","store":"/nix/store","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae/nixexprs.tar.xz"}' -> '{"etag":"\"34f07d1799419295ce257603f71ec870\"","storePath":"gg7dayixpdi4hf9agjc14zm8f8bqxqh0-nixexprs.tar.xz","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae/nixexprs.tar.xz"}'
    performing daemon worker op: 11
    performing daemon worker op: 1
    using cache entry 'file:{"name":"nixexprs.tar.xz","store":"/nix/store","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae/nixexprs.tar.xz"}' -> '{"etag":"\"34f07d1799419295ce257603f71ec870\"","url":"https://releases.nixos.org/nixpkgs/25.05-darwin/nixpkgs-darwin-25.05pre803654.5ac14523b6ae/nixexprs.tar.xz"}', '/nix/store/gg7dayixpdi4hf9agjc14zm8f8bqxqh0-nixexprs.tar.xz'
    unpacking 1 channels...
    warning: ignoring the client-specified setting 'download-attempts', because it is a restricted setting and you are not a trusted user
    warning: ignoring the client-specified setting 'stalled-download-timeout', because it is a restricted setting and you are not a trusted user
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 297 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 681 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 1072 ms
    warning: error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds; retrying in 2379 ms
    error: unable to download 'https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo': Timeout was reached (28) Resolving timed out after 5000 milliseconds
    error: program '/nix/store/3qh0pcp0wvqc2gd5qzir4mlfcgwmdfyy-nix-2.29.0/bin/nix-env' failed with exit code 1
Contents of `~/.config/nix/nix.conf`

    experimental-features = nix-command
    connect-timeout = 60000
    stalled-download-timeout = 60000
    download-attempts = 10
Result of `curl "https://cache.nixos.org/grg9cnxl08hi6s82hjrghlrzwvcnj0gz.narinfo"`

    404
Result of `time ping cache.nixos.org -c 1`

    PING dualstack.v2.shared.global.fastly.net (199.232.90.217) 56(84) bytes of data.
    64 bytes from 199.232.90.217 (199.232.90.217): icmp_seq=1 ttl=53 time=19.0 ms

    --- dualstack.v2.shared.global.fastly.net ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 19.025/19.025/19.025/0.000 ms

    real	0m5.117s
    user	0m0.003s
    sys	0m0.005s

Expected Behavior

I expected setting connect-timeout in my nix.conf to something larger than 5 seconds would increase the timeout duration for resolving the links, but it would appear not. I also tried it in /etc/nix/nix.conf I added stalled-download-timeout and download-attempts to see if they would have any affect, but they didn't other than confirming that the config file was loaded by adding the warnings at the top. I would think that nix uses it's own bundled downloader rather than the one on the OS, but I tested curl anyways and the result is the same as what shows up on my browser. I tried to figure out how long it takes to resolve cache.nixos.org, and I never saw a time under 0m5.088s with a ping of about 20ms. This would make sense as it is slightly longer than the 5 second default for connect-timeout, but changing that value in the config doesn't affect it. Lastly, I was trying to install MPV using nix-env -iA nixpkgs.mpv while I was diagnosing this, and it failed at least 50 times as I had it on a loop to see if it was just random network issues, and it eventually resolved the connection and was able to download just fine, so this appears to be an issue with just the timeout for resolving cache.nixos.org.

LinuxUser393 avatar Jun 10 '25 02:06 LinuxUser393

I'm noticing this as well. Seems to have changed recently, probably between https://github.com/NixOS/nixpkgs/commit/46e634be05ce and https://github.com/NixOS/nixpkgs/commit/9807714d6944.

(My DNS resolution takes longer when the dynamic IP assigned to me by Verizon changes, until I apply a manual update via nixops. Today's IP change ran into this, but the last one (on May 22) didn't.)

jerith666 avatar Jul 16 '25 18:07 jerith666

I have a fresh arch linux install, i was trying to install nix as usual but i get

warning: error: unable to download 'https://nixos.org/channels/nixpkgs-unstable': Timeout was reached (28) Resolving timed out after 5001 milliseconds; retrying in 264 ms

when i do nix-channel --update

is this related?

i am also having the same time out problem when i try to ping nixos.org (ipv4 and ipv6)

rnoba avatar Jul 18 '25 20:07 rnoba

Getting the same issue when I tried to run nixos-rebuild, which I only got after upgrading my nixpkgs flake input when dealing bcachefs issue with Linux 6.16.

RadxaYuntian avatar Jul 29 '25 03:07 RadxaYuntian

I'm noticing this as well. Seems to have changed recently, probably between NixOS/nixpkgs@46e634be05ce and NixOS/nixpkgs@9807714d6944.

At least NixOS/nixpkgs@3a69ec5f156ead261e57a377fb03e105e9a6fe0c on nixos-25.05 branch is still good.

RadxaYuntian avatar Jul 29 '25 09:07 RadxaYuntian

Ran into this issue with Ubuntu under WSL2. Setting the connect-timeout = 60000 in /etc/nix/nix.conf resolved the issue for me.

xfthhxk avatar Aug 01 '25 17:08 xfthhxk

Might be the same thing as #13466

dramforever avatar Sep 14 '25 11:09 dramforever

Seemingly the OP has tried increasing connect-timeout, so something else may be going on

lzcunt avatar Sep 15 '25 10:09 lzcunt

This means that the timeout was still 5 seconds on the daemon side, which ~/.config/nix/nix.conf doesn't affect.

Timeout was reached (28) Resolving timed out after 5000 milliseconds

I don't know if this part is intended but see https://github.com/NixOS/nix/issues/13466#issuecomment-3288919951 for a temporary fix for the daemon on NixOS

On non-NixOS just edit /etc/nix/nix.conf and restart daemon

dramforever avatar Sep 15 '25 11:09 dramforever

Please test 2.31.2. You can use the workaround here while updating: https://github.com/NixOS/nix/issues/13466#issuecomment-3288919951. Both may be required, since some network operations happen on CLI side and some on daemon side.

If you see this and the value is close to 5000, the config hasn't been set correctly.

Resolving timed out after 5001 milliseconds

dramforever avatar Sep 23 '25 15:09 dramforever

Thanks for the help! I tried the workaround you mentioned, which worked well enough to upgrade to 2.31.2. I had a bit of trouble figuring out how to upgrade to 2.31.2 as that is not the latest stable version, but I eventually followed https://nix.dev/manual/nix/2.28/installation/upgrading.html replacing -p nix in nix-shell with -p nixVersions.latest and replacing --attr nix cacert with --attr nixVersions.latest cacert in the nix-env command. If anyone else is doing this in the future, remember to check the version of the nix package and the nixVersions.latest packages.

LinuxUser393 avatar Sep 23 '25 23:09 LinuxUser393