mach-nix
mach-nix copied to clipboard
DavHau
Unable to install pygmo with cloudpickle
I'm trying to set up a custom Python environment:
{
inputs = {
#requires master branch in order to include fixes that unbreak pagmo (pygmo dependency) on mac
nixpkgs.url = "github:nixos/nixpkgs";
flake-utils.url = "github:numtide/flake-utils";
mach-nix = {
url = "github:DavHau/mach-nix";
inputs.nixpkgs.follows = "nixpkgs";
inputs.flake-utils.follows = "flake-utils";
inputs.pypi-deps-db.follows = "pypi-deps-db";
};
pypi-deps-db = {
url = "github:DavHau/mach-nix";
};
};
outputs = { self, nixpkgs, flake-utils, mach-nix, pypi-deps-db }:
flake-utils.lib.eachDefaultSystem (system:
let
python = "python39";
pkgs = import nixpkgs { inherit system; };
mach = import mach-nix { inherit pkgs python; };
fullPython = mach.mkPython {
requirements = ''
pygmo
cloudpickle
'';
};
in {
devShell = pkgs.mkShell {
nativeBuildInputs = [ fullPython ];
};
});
}
This gives the following error:
error: builder for '/nix/store/dvazzy9mkfxfhrpac3i97rki6axbvkzj-python3.9-pyzmq-23.2.0.drv' failed with exit code 1;
> Executing setuptoolsBuildPhase
> Traceback (most recent call last):
> File "/private/tmp/nix-build-python3.9-pyzmq-23.2.0.drv-0/pyzmq-23.2.0/nix_run_setup", line 8, in <module>
> exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
> File "setup.py", line 34, in <module>
> from packaging.version import Version as V
> ModuleNotFoundError: No module named 'packaging'
So since this seems like a missing dependency for pyzmq, so let's try adding this line to the definition of fullPython: _.pyzmq.buildInputs.add = [ mach.nixpkgs.python39Packages.packaging ];
At this point, the following collision occurs:
error: builder for '/nix/store/l1pzi2mwdiz9vs1dg5gmhk8ackv9panc-python3-3.9.13-env.drv' failed with exit code 255;
last 1 log lines:
> error: collision between `/nix/store/zksixc5s5l874idyb5h24kvw7hrx8667-python3.9-cloudpickle-2.1.0/lib/python3.9/site-packages/cloudpickle/__pycache__/__init__.cpython-39.pyc' and `/nix/store/qqqpp1ndpf75f6hs0l309rq77bzfiisl-python3.9-cloudpickle-2.1.0/lib/python3.9/site-packages/cloudpickle/__pycache__/__init__.cpython-39.pyc'
For full logs, run 'nix log /nix/store/l1pzi2mwdiz9vs1dg5gmhk8ackv9panc-python3-3.9.13-env.drv'.
error: 1 dependencies of derivation '/nix/store/sl59bwrhcg0il2l6493krrayl940fxik-python3-3.9.13-env.drv' failed to build
error: 1 dependencies of derivation '/nix/store/vx4is3zgvwg6v1bdjln4jh6nvy5hay2m-nix-shell-env.drv' failed to build
Interestingly, it's between two identical versions of cloudpickle!
My suspicion is that some of the packages are being installed from nixpkgs (because I manually specified pyzmq, without going through mach-nix), and some from other sources, and this is causing conflict. The first thing this suggests is that maybe forcing cloudpickle to be installed from nixpkgs is the solution, but setting its provider to "nixpkgs" does nothing. So let's try forcing everything to go through nixpkgs by adding this to fullpython
providers = {
_default = "nixpkgs,wheel,sdist,conda,conda-forge";
};
whereupon we get this:
error: builder for '/nix/store/9z58lh3mg0avyqmgdb1mbfdxybf1ns5s-python3.9-pyparsing-3.0.9.drv' failed with exit code 2;
> ModuleNotFoundError: No module named 'flit_core'
Adding the requisite _.pyparsing.buildInputs.add = [ mach.nixpkgs.python39Packages.flit-core ];we get a similar Cython error:
error: builder for '/nix/store/gpb3lb3r2dns22fk90s0w2wzyhgwk7nv-python3.9-numpy-1.23.1.drv' failed with exit code 1;
> File "setup.py", line 258, in generate_cython
> raise OSError(msg) from e
> OSError: Cython needs to be installed in Python as a module
Trying to fix this with _.numpy.buildInputs.add = [ mach.nixpkgs.python39Packages.cython ];:
> Executing setuptoolsBuildPhase
> Error: 'pybind11' must be installed before running the build.
All right, let's add pybind11: _.scipy.buildInputs.add = [ mach.nixpkgs.python39Packages.pybind11 ];... at which point we get
error: builder for '/nix/store/40hwmp6m2vkacd0fhqanjf94xs8lb5wd-python3.9-scipy-1.8.1.drv' failed with exit code 1;
last 10 log lines:
> Traceback (most recent call last):
> File "/private/tmp/nix-build-python3.9-scipy-1.8.1.drv-1/scipy-1.8.1/nix_run_setup", line 8, in <module>
> exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
> File "setup.py", line 628, in <module>
> setup_package()
> File "setup.py", line 612, in setup_package
> generate_cython()
> File "setup.py", line 361, in generate_cython
> raise RuntimeError("Running cythonize failed!")
> RuntimeError: Running cythonize failed!
which is well out of my depth, but I suspect that the python package isn't being brought in correctly.
So perhaps let's try another approach: install pygmo from conda-forge, which is how they recommend doing it in their docs:
error: builder for '/nix/store/n8h9bl05y6ayi6myvk7357cdi1h4dxdr-mach_nix_file.drv' failed with exit code 1;
last 9 log lines:
>
> The Package 'pypy' (build: ()) is not available from any of the selected providers ['conda/conda-forge', 'conda/main', 'conda/r', 'nixpkgs', 'sdist', 'wheel']
> for the selected python version
> The required package might just not (yet) be part of the dependency DB currently used.
> The DB can be updated by specifying 'pypiDataRev' when importing mach-nix.
> For examples see: https://github.com/DavHau/mach-nix/blob/master/examples.md
> If it still doesn't work, there might have been an error while building the DB.
> Please open an issue at: https://github.com/DavHau/mach-nix/issues/new
>
For full logs, run 'nix log /nix/store/n8h9bl05y6ayi6myvk7357cdi1h4dxdr-mach_nix_file.drv'.
which is the subject of #199.
Unfortunately, there's no easy way for me to remove cloudpickle from the the requirements list - the full flake I'm trying to write has a ton of other packages that depennd on cloudpickle, so just removing it from requirements and using the one pulled in by pygmo is not an option.
I'm at my wit's end right now - is there a simple-ish way to sort out this dependency hell? Thanks!
System info: aarch64-darwin, nix 2.8.1.
Personally, I have seen weirdness when doing import mach-nix in a flake; I generally just use the flake input directly eg. mach-nix.lib.${system}.mkPython etc. I also maintain a bunch of overrides to get packages working as expected, thought this was due to the python310 upgrade but seems like you are experiencing it as well
providers =
{
aiohttp = "nixpkgs";
aiosignal = "nixpkgs";
atpublic = "nixpkgs";
distributed = "nixpkgs";
jeepney = "nixpkgs";
psutil = "nixpkgs";
py-spy = "nixpkgs";
tomli = "nixpkgs";
torch = "nixpkgs";
typing-extension = "wheel";
virtualenv = "nixpkgs";
};
requirements_py10 = # seems to be neccessary for pretty much every environment
requirements
+ ''
flit-core
tomli
setuptools
typing-extensions >= 4.1.0
'';
Adding the extra providers and requirements didn't seem to have an effect.
Interestingly enough, going from an import to direct call to mach-nix appears to expose another bug.
Flake:
{
inputs = {
#requires master branch in order to include fixes that unbreak pagmo (pygmo dependency) on mac
nixpkgs.url = "github:nixos/nixpkgs/nixpkgs-unstable";
flake-utils.url = "github:numtide/flake-utils";
mach-nix = {
url = "github:DavHau/mach-nix/3.5.0";
inputs.nixpkgs.follows = "nixpkgs";
inputs.flake-utils.follows = "flake-utils";
inputs.pypi-deps-db.follows = "pypi-deps-db";
};
pypi-deps-db = {
url = "github:DavHau/mach-nix/3.5.0";
};
};
outputs = { self, nixpkgs, flake-utils, mach-nix, pypi-deps-db }:
flake-utils.lib.eachDefaultSystem (system:
let
pkgs = import nixpkgs { inherit system; };
fullPython = mach-nix.lib.${system}.mkPython {
requirements = ''
flit-core
'';
};
in {
devShell = pkgs.mkShell {
nativeBuildInputs = [ fullPython ];
};
});
}
gives this mysterious message:
error: opening file '/nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/UNIX_TIMESTAMP': No such file or directory
… while evaluating 'toInt'
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/lib/strings.nix:761:11:
760| # Obviously, it is a bit hacky to use fromJSON this way.
761| toInt = str:
| ^
762| let may_be_int = fromJSON str; in
… from call site
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/flake.nix:15:26:
14| let
15| dataLastModified = toInt (readFile "${inp.pypi-deps-db}/UNIX_TIMESTAMP");
| ^
16| dataOutdated =
… while evaluating 'throwOnOutdatedData'
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:39:25:
38|
39| throwOnOutdatedData = args:
| ^
40| if dataOutdated && ! (args.ignoreDataOutdated or false) then
… from call site
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:64:48:
63|
64| __mkPython = caller: args: _mkPython caller (throwOnOutdatedData args);
| ^
65|
… while evaluating '_mkPython'
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:67:23:
66| # (High level API) generates a python environment with minimal user effort
67| _mkPython = caller: args:
| ^
68| let
… from call site
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:64:30:
63|
64| __mkPython = caller: args: _mkPython caller (throwOnOutdatedData args);
| ^
65|
… while evaluating '__mkPython'
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:64:24:
63|
64| __mkPython = caller: args: _mkPython caller (throwOnOutdatedData args);
| ^
65|
… from call site
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:99:20:
98| # the main functions
99| mkPython = args: __mkPython "mkPython" args;
| ^
100| mkPythonShell = args: (__mkPython "mkPythonShell" args).env;
… while evaluating 'mkPython'
at /nix/store/n6hsk44fw5x8kqkff2mxyd5wd6663ai5-source/default.nix:99:14:
98| # the main functions
99| mkPython = args: __mkPython "mkPython" args;
| ^
100| mkPythonShell = args: (__mkPython "mkPythonShell" args).env;
… from call site
at /nix/store/icany4j5fc357h2dys5wqg4i0zck12d3-source/flake.nix:26:22:
25|
26| fullPython = mach-nix.lib.${system}.mkPython {
| ^
27| requirements = ''
… while evaluating 'isDerivation'
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/lib/attrsets.nix:427:18:
426| */
427| isDerivation = x: x.type or null == "derivation";
| ^
428|
… from call site
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/pkgs/stdenv/generic/make-derivation.nix:193:8:
192| checkDependencyList' = positions: name: deps: lib.flip lib.imap1 deps (index: dep:
193| if lib.isDerivation dep || isNull dep || builtins.typeOf dep == "string" || builtins.typeOf dep == "path" then dep
| ^
194| else if lib.isList dep then checkDependencyList' ([index] ++ positions) name dep
… while evaluating anonymous lambda
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/pkgs/stdenv/generic/make-derivation.nix:192:81:
191| checkDependencyList = checkDependencyList' [];
192| checkDependencyList' = positions: name: deps: lib.flip lib.imap1 deps (index: dep:
| ^
193| if lib.isDerivation dep || isNull dep || builtins.typeOf dep == "string" || builtins.typeOf dep == "path" then dep
… from call site
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/lib/lists.nix:117:32:
116| */
117| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
| ^
118|
… while evaluating anonymous lambda
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/lib/lists.nix:117:29:
116| */
117| imap1 = f: list: genList (n: f (n + 1) (elemAt list n)) (length list);
| ^
118|
… from call site
… while evaluating anonymous lambda
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/pkgs/stdenv/generic/make-derivation.nix:212:13:
211| (map (drv: drv.__spliced.buildBuild or drv) (checkDependencyList "depsBuildBuild" depsBuildBuild))
212| (map (drv: drv.nativeDrv or drv) (checkDependencyList "nativeBuildInputs" nativeBuildInputs
| ^
213| ++ lib.optional separateDebugInfo' ../../build-support/setup-hooks/separate-debug-info.sh
… from call site
… while evaluating 'getOutput'
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/lib/attrsets.nix:598:23:
597| */
598| getOutput = output: pkg:
| ^
599| if ! pkg ? outputSpecified || ! pkg.outputSpecified
… from call site
… while evaluating the attribute 'nativeBuildInputs' of the derivation 'nix-shell'
at /nix/store/c0ds5sgga2zzgm9m0r4v7d1y5yy4ax1w-source/pkgs/stdenv/generic/make-derivation.nix:270:7:
269| // (lib.optionalAttrs (attrs ? name || (attrs ? pname && attrs ? version)) {
270| name =
| ^
271| let
Your pypi deps db looks suspect - I generally use
pypi-deps-db = {
flake = false;
url = "github:DavHau/pypi-deps-db";
};
to get the latest deps; if you want the version shipped with 3.5.0 you shouldn't need to set it (and as I understand it, it won't be exposed by the mach-nix repo)
How can you specify a python version without an import? All the examples given in the documentation rely on passing a python parameter to mach-nix during the import.
Also, what is the function of flake = false in the above example?
Thanks!
flake = false just fetches the raw repository, instead of evaluating it as a flake (mach nix just needs the source tree of pypi-deps-db to do its resolution; it doesn't use its flake outputs)
You can pass the same python = "python39"; attribute to mkPython(or any of the mach build functions as far as I understand)