Tar download very slow

[shyim]

Describe the bug

When running nix flake update, it downloads the tar.gz file, which takes a lot of time. This usually takes me a minute, but for my colleagues, it takes some time up to half an hour.

The same download using curl takes just a second:

❯ time curl -L -o foo.tar.gz 'https://github.com/NixOS/nixpkgs/archive/e4e7a43a9db7e22613accfeb1005cca1b2b1ee0d.tar.gz'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 40.4M  100 40.4M    0     0  24.2M      0  0:00:01  0:00:01 --:--:-- 28.5M

________________________________________________________
Executed in    1.68 secs      fish           external
   usr time  225.63 millis   92.00 micros  225.54 millis
   sys time  129.86 millis  978.00 micros  128.89 millis

I suspect that Nix does a range download and GitHub does not like that and have to export the tar always again from GIt 🤔

Steps To Reproduce

1. Run a nix flake update so nixpkgs fetches the new tarball

Expected behavior

A clear and concise description of what you expected to happen.

nix-env --version output

❯ nix-env --version
nix-env (Nix) 2.22.0

Additional context

Add any other context about the problem here.

Priorities

Add :+1: to issues you find important.

[JayanSmart]

I can reproduce this on a 1Gbps connection, I am taking over 10 min's to download the tarball when I run the following command on a fresh nix install: echo "Hello world" | nix run "nixpkgs#cowsay"

[edolstra]

Can you check if the slow performance also happens with Nix 2.20 (nix shell nix/2.20.6)? It might be related to the Git-based tarball cache that we started using in 2.21.

[Rucadi]

I was having the same problem with centos7, (inside a container), works well in an ubuntu host, using the latest nix from unstable.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2024-05-15-nix-team-meeting-minutes-146/45491/1

[shyim]

2.20.6 is A LOT faster for me. With 2.21 it's slow again

[JayanSmart]

I can confirm, 2.20.6 is a lot faster for me as well. Same issue persists on verison 2.21.2.

[Bird87ZA]

I have the same problem on a 1Gbps connection.

[rbutera]

also facing this issue on a 1gbps connection in the UK

[ccalhoun1999]

I am also dealing with horribly slow downloads in Japan.

[Aneeqasif]

is there a fix or workaround guyz?

[dkolgrre]

I'm also experiencing this issue on an M1 macbook pro. Downloading any tar manually/with wget takes seconds, but using Nix it takes FOREVER.

[zimbatm]

Here is a workaround. We're still testing it but so far it looks like a good trade-off (notice the shallow=1):

-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
+    nixpkgs.url = "git+https://github.com/NixOS/nixpkgs?shallow=1&ref=nixpkgs-unstable";

[shishkin]

Thanks @zimbatm! This does improve download speed for nixpkgs to the point where I didn't even notice the download message for nixpkgs during nix flake update.

[onny]

Should this issue be moved to the Nix repository?

[NobbZ]

This is the nix repository…

[byrnedo]

FWIW, on Mac M3, slow as hell if downloading in nix (I'm using devenv), but if I try and download the archive from the browser, it's lightning fast. I read somewhere that it might be something to do with authenticated vs unauthenticated requests to github?

[Mic92]

@byrnedo If you think this is an issue you can try adding a GITHUB_TOKEN to your $HOME/.config/nix/nix.conf: access-tokens = github.com=ghp_000000000000000000000000000000000000

[ccalhoun1999]

@byrnedo If you think this is an issue you can try adding a GITHUB_TOKEN to your $HOME/.config/nix/nix.conf: access-tokens = github.com=ghp_000000000000000000000000000000000000

Is a github token something we could set in our nix configuration? I'm struggling to find the option for it.

[hey2022]

Is a github token something we could set in our nix configuration? I'm struggling to find the option for it.

You can use the nix.settings NixOS option. You can set the access-tokens like this:

nix.settings = {
  access-tokens = "github.com=ghp_000000000000000000000000000000000000";
}

[NobbZ]

Here is a workaround. We're still testing it but so far it looks like a good trade-off (notice the shallow=1):

• nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";

• nixpkgs.url = "git+https://github.com/NixOS/nixpkgs?shallow=1&ref=nixpkgs-unstable";

According to the fetchTree docs, shallow=1 is the default, do you pass a redundant arg or are the docs wrong?

[Mic92]

fetchTree

This only applies to builtins.fetchTree, not the not for flake urls. Shallow clone don't compute a revision count so we cannot easily turn shallow clones on without breaking backwards compatibility.

[byrnedo]

@byrnedo If you think this is an issue you can try adding a GITHUB_TOKEN to your $HOME/.config/nix/nix.conf: access-tokens = github.com=ghp_000000000000000000000000000000000000

This didn't seem to help any

[Mic92]

This is what we do for NFS: https://github.com/TUM-DSE/doctor-cluster-config/blob/0e1a7021c17326d92a76913feea253145242ef3b/modules/nfs/client.nix#L44 It also fixes home-manager.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2025-05-07-nix-team-meeting-minutes-224/63973/1

[byrnedo]

FWIW, I notice I get slow github downloads, in the same speed range, if I download a build artifact (from a random git repo, not devenv) from github actions builds. So, not release artifacts, but build artifacts.