No way to customize / override wheel build process

[9999years]

There's no way to customize the way wheels are built with pypi2nix. In specific, pycorpora wants to download the corpora sources when it's installed (see setup.py lines 20-37).

pycorpora fails with an SSL error when running setup.py, but that's beside the point -- I want to be able to use Nix to pick a specific revision of corpora. With this function I can pick an arbitrary revision:

{ pkgs ? import <nixpkgs> { }, ... }:
let rev = "8d0d939d3ebc868ce576fdc87b02e8b93961b734";
in pkgs.fetchurl {
  name = "corpora.zip";
  url = "https://github.com/dariusk/corpora/archive/${rev}.zip";
  sha512 =
    "0xljh4hk63abba38gq9bsl14zzr4wif591d5ifqgglw2a2fr8nsp3wcvar2nbc7sz8szjas7d3wl3476xl6zvi21n5dv9q1n403yfhx";
}

Then, I'd like to use overrides to add the proper arguments:

{ pkgs, python }:
self: super: {
  "pycorpora" = python.overrideDerivation super."pycorpora" (old: {
    setupPyGlobalFlags = [
      "--corpora-zip-url=${import ./build/corpora.nix pkgs}"
    ];
  });
}

This doesn't work, of course, because the derivation I want to override doesn't exist yet.

Is there any way / can we add some way to customize the way wheels are built? If I could specify an overlay for pythonPackages in the generated requirements.nix or something that would be ideal.

Log for pypi2nix -e pycorpora==0.1.2

$ pypi2nix --version
2.0.4
$ pypi2nix --verbose -e pycorpora==0.1.2
WARNING: You are using pip version 20.0.dev0; however, version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
  ERROR: Command errored out with exit status 1:
   command: /nix/store/ks2vr8r8jlz20bkiqfdykmz6x06zin5v-python3-3.7.6-env/bin/python3.7 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/run/user/1000/tmpbaohyujb/pip-wheel-2o10_2o8/pycorpora/setup.py'"'"'; __file__='"'"'/run/user/1000/tmpbaohyujb/pip-wheel-2o10_2o8/pycorpora/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /run/user/1000/tmpbaohyujb/pip-wheel-fx6moxev
       cwd: /run/user/1000/tmpbaohyujb/pip-wheel-2o10_2o8/pycorpora/
  Complete output (18 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib
  creating build/lib/pycorpora
  copying pycorpora/__init__.py -> build/lib/pycorpora
  creating build/lib/pycorpora/data
  creating build/lib/pycorpora/data/pycorpora_test
  copying pycorpora/data/pycorpora_test/test.json -> build/lib/pycorpora/data/pycorpora_test
  copying pycorpora/data/pycorpora_test/test-filename.json -> build/lib/pycorpora/data/pycorpora_test
  creating build/lib/pycorpora/data/pycorpora_test/subdir
  copying pycorpora/data/pycorpora_test/subdir/another_test.json -> build/lib/pycorpora/data/pycorpora_test/subdir
  installing to build/bdist.linux-x86_64/wheel
  running install
  Installing corpora data from https://github.com/dariusk/corpora/archive/master.zip
  creating corpora-download
  error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1076)>
  ----------------------------------------
  ERROR: Failed building wheel for pycorpora
ERROR: Failed to build one or more wheels
DEBUG: |-> nix-shell --pure /run/user/1000/tmpzrw1ll24 --command 'python -c '"'"'import json;from setuptools._vendor.packaging.markers import default_environment;print(json.dumps(default_environment()))'"'"''
DEBUG:     {"implementation_name": "cpython", "implementation_version": "3.7.6", "os_name": "posix", "platform_machine": "x86_64", "platform_release": "5.4.17", "platform_system": "Linux", "platform_version": "#1-NixOS SMP Sat Feb 1 09:34:53 UTC 2020", "python_full_version": "3.7.6", "platform_python_implementation": "CPython", "python_version": "3.7", "sys_platform": "linux"}
INFO: pypi2nix v2.0.4 running ...
INFO: Downloading wheels and creating wheelhouse ...
DEBUG: |-> nix-instantiate --eval --expr 'let pkgs = import <nixpkgs> {}; in ""'
DEBUG:     ""
INFO: Downloading runtime requirements
DEBUG: Created requirements file /run/user/1000/tmpbaohyujb/e3b4b0e95fb10b2badecc7b5c86e3677.txt
DEBUG: Created requirements file /run/user/1000/tmpbaohyujb/670933a1582dcbbdf4184b337fae3245.txt
DEBUG: |-> nix-shell --argstr download_cache_dir /run/user/1000/tmpbaohyujb/cache --arg extra_build_inputs '[  ]' --argstr project_dir /run/user/1000/tmpbaohyujb --argstr python_version python3 --argstr extra_env '' --arg requirements_files '[ "/run/user/1000/tmpbaohyujb/670933a1582dcbbdf4184b337fae3245.txt" ]' --argstr destination_directory /run/user/1000/tmpbaohyujb/download --argstr editable_sources_directory /run/user/1000/tmpbaohyujb/editable_sources --argstr build_directory /run/user/1000/tmpbaohyujb/build --pure /nix/store/qqdnqwymdx6whkj5i48vx4az1flliy5l-pypi2nix-2.0.4/lib/python3.7/site-packages/pypi2nix/pip/download.nix --command exit
DEBUG:     Collecting pycorpora==0.1.2
DEBUG:       Downloading pycorpora-0.1.2.tar.gz (6.7 kB)
DEBUG:       Saved /run/user/1000/tmpbaohyujb/download/pycorpora-0.1.2.tar.gz
DEBUG:     Successfully downloaded pycorpora
INFO: Build wheels of setup and runtime requirements
DEBUG: Created requirements file /run/user/1000/tmpbaohyujb/d02e01d82f4cec5e0695dacd2c3b6979.txt
DEBUG: Created requirements file /run/user/1000/tmpbaohyujb/eff243ba4ff425dd06ba4e6f6797269d.txt
DEBUG: |-> nix-shell --argstr download_cache_dir /run/user/1000/tmpbaohyujb/cache --arg extra_build_inputs '[  ]' --argstr project_dir /run/user/1000/tmpbaohyujb --argstr python_version python3 --argstr extra_env '' --arg wheels_cache '[  ]' --arg requirements_files '[ "/run/user/1000/tmpbaohyujb/eff243ba4ff425dd06ba4e6f6797269d.txt" ]' --argstr editable_sources_directory /run/user/1000/tmpbaohyujb/editable_sources --argstr build_directory /run/user/1000/tmpbaohyujb/build --argstr wheels_dir /run/user/1000/tmpbaohyujb/wheel --arg sources '[ "/run/user/1000/tmpbaohyujb/download" ]' --pure /nix/store/qqdnqwymdx6whkj5i48vx4az1flliy5l-pypi2nix-2.0.4/lib/python3.7/site-packages/pypi2nix/pip/wheel.nix --command exit
DEBUG:     Looking in links: file:///run/user/1000/tmpbaohyujb/download
DEBUG:     Processing /run/user/1000/tmpbaohyujb/download/pycorpora-0.1.2.tar.gz
DEBUG:     Building wheels for collected packages: pycorpora
DEBUG:       Building wheel for pycorpora (setup.py): started
DEBUG:       Building wheel for pycorpora (setup.py): finished with status 'error'
DEBUG:       Running setup.py clean for pycorpora
DEBUG:     Failed to build pycorpora
ERROR: Collecting pycorpora==0.1.2
ERROR:   Downloading pycorpora-0.1.2.tar.gz (6.7 kB)
ERROR:   Saved /run/user/1000/tmpbaohyujb/download/pycorpora-0.1.2.tar.gz
ERROR: Successfully downloaded pycorpora
ERROR: Looking in links: file:///run/user/1000/tmpbaohyujb/download
ERROR: Processing /run/user/1000/tmpbaohyujb/download/pycorpora-0.1.2.tar.gz
ERROR: Building wheels for collected packages: pycorpora
ERROR:   Building wheel for pycorpora (setup.py): started
ERROR:   Building wheel for pycorpora (setup.py): finished with status 'error'
ERROR:   Running setup.py clean for pycorpora
ERROR: Failed to build pycorpora
Do you want to report above issue (a browser will open with prefilled details of issue)? [y/N]:

[seppeljordan]

Hi, thanks for reporting the issue. I am currently the only one maintaining this project. Therefore it is hard for me to implement all the suggestions from the issue tracker. I agree that users should be able to customize the build process. The only way to influence the build process currently is by setting environment environment variables for pip and setup.py. Check out https://pypi2nix.readthedocs.io/en/latest/index.html#additional-environment-variables for more info. If you feel like implementing this feature for pypi2nix go ahead, it would be a service to "the community" and I would be happy to assist via code reviews or answer any questions about the code as good as I can. There is also a rudimentary section about developing pypi2nix in the manual. This offer stands of course for anybody reading this :)