Could not read output of nix-prefetch-url when "unknown compression method" occurs

[chriswarbo]

I just got this output from niv:

$ niv add skulpture-qt4 -v 0.2.4 -t 'http://skulpture.maxiom.de/releases/skulpture-<version>.tar.gz'
Adding package skulpture-qt4
  FATAL: Could not read the output of 'nix-prefetch-url'. This is a bug. Please create a
  ticket:

    https://github.com/nmattia/niv/issues/new

  Thanks! I'll buy you a beer.
  command:
  nix-prefetch-url--unpack http://skulpture.maxiom.de/releases/skulpture-0.2.4.tar.gz --name skulpture-0.2.4.tar.gz
  stdout:

  stderr:
  error: unknown compression method 'x-gzip'

  FATAL: One or more packages failed to update:
  skulpture-qt4: ExitFailure 1
ERROR: ExitFailure 1

This was on an i686-linux install of NixOS 19.09, with this version of niv:

"niv": {
        "branch": "master",
        "description": "Easy dependency management for Nix projects",
        "homepage": "https://github.com/nmattia/niv",
        "owner": "nmattia",
        "repo": "niv",
        "rev": "d13bf5ff11850f49f4282d59b25661c45a851936",
        "sha256": "06hg0i173dia99bcv8si8qqwdw0ymbngsilz2s182d4afsnylq5z",
        "type": "tarball",
        "url": "https://github.com/nmattia/niv/archive/89ae775e9dfc2571f912156dd2f8627e14d4d507.tar.gz",
        "url_template": "https://github.com/<owner>/<repo>/archive/<rev>.tar.gz"
    },

Looks like a problem with nix-prefetch-url (with or without --unpack) and this server's response headers:

$ curl -I "http://skulpture.maxiom.de/releases/skulpture-0.2.4.tar.gz"
HTTP/1.1 200 OK
Date: Tue, 25 Aug 2020 10:42:36 GMT
Server: Apache/2.4.43 (Unix)
Last-Modified: Mon, 02 Apr 2018 18:22:19 GMT
ETag: "1d9ca-568e1ab957357"
Accept-Ranges: bytes
Content-Length: 121290
Content-Type: application/x-tar
Content-Encoding: x-gzip

It works with fetchurl from nixpkgs:

$ nix repl
Welcome to Nix version 2.3.2. Type :? for help.

nix-repl> :b with import <nixpkgs> {}; fetchurl {
        url    = "http://skulpture.maxiom.de/releases/skulpture-0.2.4.tar.gz";
        sha256 = "1s27xqd32ck09r1nnjp1pyxwi0js7a7rg2ppkvq2mk78nfcl6sk0";
      }
:b with import <nixpkgs> {}; fetchurl {
                  url    = "http://skulpture.maxiom.de/releases/skulpture-0.2.4.tar.gz";
                  sha256 = "1s27xqd32ck09r1nnjp1pyxwi0js7a7rg2ppkvq2mk78nfcl6sk0";
                }

this derivation produced the following outputs:
  out -> /nix/store/…-skulpture-0.2.4.tar.gz

[nmattia]

That's annoying. According to @edolstra, it's a server error: https://github.com/NixOS/nix/issues/1568

Looks like setting accept-encoding will correctly return a gzip, with Content-Encoding: gzip header:

$ curl -v -H 'Accept-Encoding: gzip' http://skulpture.maxiom.de/releases/skulpture-0.2.4.tar.gz | tar -z --list | head
* Connected to skulpture.maxiom.de (2a01:238:20a:202:1092::) port 80 (#0)
> GET /releases/skulpture-0.2.4.tar.gz HTTP/1.1
> Host: skulpture.maxiom.de
> User-Agent: curl/7.70.0
> Accept: */*
> Accept-Encoding: gzip
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Wed, 26 Aug 2020 09:42:57 GMT
< Server: Apache/2.4.43 (Unix)
< Last-Modified: Mon, 02 Apr 2018 18:22:19 GMT
< ETag: "1d9ca-568e1ab957357"
< Accept-Ranges: bytes
< Content-Length: 121290
< Content-Type: application/x-tar
< Content-Encoding: gzip
< 
{ [4020 bytes data]
skulpture-0.2.4/
skulpture-0.2.4/README.html
skulpture-0.2.4/README
skulpture-0.2.4/CMakeLists.txt
skulpture-0.2.4/NEWS
skulpture-0.2.4/INSTALL
skulpture-0.2.4/src/
skulpture-0.2.4/src/CMakeLists.txt
skulpture-0.2.4/src/.directory
skulpture-0.2.4/src/config/
...

Unfortunately it doesn't look like nix-prefetch-url allows setting headers. Might be worth opening an issue on the Nix tracker!