bad (null) hash for package from extra pypi index

[abathur]

Ran into a hurdle around extra pypi indexes.

background

I'm looking at converting a python project from mach-nix to dream2nix via the pip module. It needs one private package from an extra index.

The key bits of how I worked around this with mach-nix are:

• use builtins.fetchurl to fetch the https:///` page (it's a "simple" index if I'm remembering the term right).
• use a runCommand to extract the specified version number from the requirements script
• use xmllint with an xpath expression to extract the corresponding https://<index>/<package>-<version>.tar.gz url
• use mach-nix.buildPythonPackage (builtins.readFile tarball_url) to build this and layer it in to overridesPre.
• filter the '--extra-index-url' out of the requirements passed to mach-nix (IIRC it wasn't compatible)

What I'm trying

Since it sounds like dream2nix's proxy approach should at least theoretically be compatible with the extra index, I tried to drop these extra steps.

It didn't quite work in practice, but it does look close.

Here's a boiled-down form of what I see during refresh:

...
[14:33:36.311] Addon error: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/nix/store/65n0vxmj4511m0c1xvz6pnygzlafbfby-filter-pypi-responses.py", line 92, in response
    badFiles = get_files_to_hide(pname, max_ts)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/65n0vxmj4511m0c1xvz6pnygzlafbfby-filter-pypi-responses.py", line 40, in get_files_to_hide
    with urlopen(req, context=context) as response:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "/nix/store/d0lpzyjad99g4rm84d4mj3mw1iq8hl03-python3-3.11.9/lib/python3.11/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
[::1]:61217: GET https://pypi.org/simple/<package>/
          << 404 Not Found 13b
[::1]:61220: GET https://<index>/<package>/
          << 200 OK 2.7k
Collecting <package>==<version> (from -r /nix/store/1d3mzch7iy720lp9r3pyajpdhd85810q-combined_requirements_dev.txt (line 99))
[::1]:61220: GET https://<index>/<package>-<version>.tar.gz
          << 302 Found 419b
[14:33:36.728][[::1]:61252] client connect
[14:33:36.948][[::1]:61252] server connect <hostname-the-index-resolved-to>:443 (<ip-the-index-resolved-to>:443)
[::1]:61252: GET https://<url-the-index-resolved-to>
          << 200 OK 991k
  Downloading https://<index>/<package>-<version>.tar.gz (1.0 MB)
  Preparing metadata (setup.py) ... done
...

When I try to nix build this, however, I get this slightly-cryptic error:

… while calling 'apply'

 at /nix/store/rnrl3q3v6fmfdd6civyh8k478n533wyq-source/modules/dream2nix/mkDerivation/interface.nix:21:13:

   20|     type = t.nullOr dreamTypes.drvPartOrPackage;
   21|     apply = drv: drv.public or drv;
     |             ^
   22|     default = null;

… while calling anonymous lambda

 at /nix/store/0vyi8f8l8cya10dmgfrj0df2iqxlhiyi-source/lib/modules.nix:834:19:

  833|           # Avoid sorting if we don't have to.
  834|           if any (def: def.value._type or "" == "order") defs''.values
     |                   ^
  835|           then sortProperties defs''.values

… from call site

 at /nix/store/rnrl3q3v6fmfdd6civyh8k478n533wyq-source/modules/dream2nix/pip/default.nix:102:28:

  101|       mkDerivation = {
  102|         src = l.mkDefault (fetchers.${metadata.sources.${config.name}.type} metadata.sources.${config.name});
     |                            ^
  103|         doCheck = l.mkDefault false;

… while calling 'url'

 at /nix/store/rnrl3q3v6fmfdd6civyh8k478n533wyq-source/modules/dream2nix/pip/default.nix:71:11:

   70|   fetchers = {
   71|     url = info: l.fetchurl {inherit (info) url sha256;};
     |           ^
   72|     git = info: config.deps.fetchgit {inherit (info) url sha256 rev;};

error: value is null while a string was expected

The index only includes an md5, so no sha256 is getting picked up for the lock.json:

{
  "fetchPipMetadata": {
    "sources": {
      // ...
      "<package>": {
        "sha256": null,
        "type": "url",
        "url": "https://<index>/<package>-<version>.tar.gz",
        "version": "<version>"
      },
      // ...
    },
    "targets": {
      "default": {
        // ...
      }
    }
  },
  "invalidationHash": "..."
}

nix build works fine if I manually edit lock.json to replace the null with the correct hash, which I was able to compute by doing something like:

$ pip download <package> --extra-index-url https://<index> --no-deps
Looking in indexes: https://pypi.org/simple, https://<index>/
Collecting <package>
  Downloading https://<index>/<package>-<version>.tar.gz (1.0 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.0/1.0 MB 2.6 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Saved ./<package>-<version>.tar.gz
Successfully downloaded <package>

$ pip hash --algorithm sha256 <package>-<version>.tar.gz
<package>-<version>.tar.gz:
--hash=sha256:...

Notes

Less confident about these bits, but I'll context dump just in case they help or draw out any corrections :)

• I may be missing it, but I didn't see a lever to control this, so I'm imagining an operationalized version of my workaround will look like a post-refresh cleanup script to calculate the right value and rewrite it.

• I'm assuming from the source that the proxy-based publication date filtering isn't applying to extra indexes--and it doesn't look like there's an override/option to make it do so? (Just confirming understanding; this isn't a blocker for my usecase, since I'm explicitly pinning package versions for anything in this index anyways.

• It looks like the hash comes from running pip install --dry-run --report, so I tried that for this package and confirmed that ["download_info"]["archive_info"]["hash"] for this entry looks like "md5=..."

• I didn't dig very deep into pip's source to see if artifacts downloaded during the install report step end up at a stable location, but I don't see any obvious levers here to either make it re-compute hashes or keep the artifact around so that we can readily do so.

  It looks like the pip module used to use pip download, but it sounds like there were good reasons to switch to install+report--so I imagine the fix would look like a fallback procedure that tries to download and hash these if the report didn't produce a usable hash? (Unless perhaps there's a way to lean on the pip cache? When I manually played around with pip download it seems to be re-downloading the files as long as they don't already exist in the current directory. I do see similar cache entries, but I suspect they're from the download and not the report?)