Local installation with external deps fails with pip

[salotz-sitx]

Describe the bug

When attempting to install locally with pip I can't get the --use-pkg-config to take effect.

To reproduce

$ pip install . --install-option='--use-pkg-config'
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Looking in indexes: https://__token__:****@gitlab.com/api/v4/groups/10836188/-/packages/pypi/simple
Processing /home/salotz/local/work/python-igraph
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting texttable>=1.6.2
  Using cached texttable-1.6.4.tar.gz (12 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for texttable, due to binaries being disabled for it.
Building wheels for collected packages: igraph
  Building wheel for igraph (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for igraph (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [40 lines of output]
      DEBUGGINGGGGGGG: sys.argv ['/home/salotz/local/work/python-igraph/.venv/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py', 'bdist_wheel', '--dist-dir', '/tmp/pip-wheel-4rd_60ld/tmpd5o0yfkh']
      running bdist_wheel
      running build
      running build_py
      running build_ext
      DEBUGGING: use_pkgconfig: False
      running build_c_core
      Cannot find vendored igraph source in vendor/source/igraph
      
      WARNING: we were not able to detect where igraph is installed on
      your machine (if it is installed at all). We will use the fallback
      library and include paths hardcoded in setup.py and hope that the
      C core of igraph is installed there.
      
      If the compilation fails and you are sure that igraph is installed
      on your machine, adjust the following two variables in setup.py
      accordingly and try again:
      
      - LIBIGRAPH_FALLBACK_INCLUDE_DIRS
      - LIBIGRAPH_FALLBACK_LIBRARY_DIRS
      
      
      Continuing in 10 seconds; press Enter to continue immediately.
      
      Build type: dynamic extension
      Include path: /usr/include/igraph /usr/local/include/igraph
      Library path:
      Runtime library path:
      Linked dynamic libraries: igraph
      Linked static libraries:
      Extra compiler options:
      Extra linker options:
      building 'igraph._igraph' extension
      /usr/bin/gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -fPIC -I/usr/include/igraph -I/usr/local/include/igraph -I/home/salotz/local/work/python-igraph/.venv/include -I/home/salotz/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.5.0/python-3.9.12-z5q67bdfbpf7bete4ipj5rkwwdbi75kt/include/python3.9 -c src/_igraph/arpackobject.c -o build/temp.linux-x86_64-cpython-39/src/_igraph/arpackobject.o
      In file included from src/_igraph/arpackobject.c:23:0:
      src/_igraph/arpackobject.h:28:10: fatal error: igraph_arpack.h: No such file or directory
       #include <igraph_arpack.h>
                ^~~~~~~~~~~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for igraph
Failed to build igraph
ERROR: Could not build wheels for igraph, which is required to install pyproject.toml-based projects

When I run with python setup.py install --use-pkg-config it seems to be fine (although I hit other probably unrelated errors).

Running some debugging statements in the setup.py I can see that in the def process_args_from_command_line that the sys.argv there does not contain the flag.

This might be an issue with pip upstream since I believe it worked when running pip install --install-option='--use-pkg-config', but I can't be sure since this is just broken and tries to apply the option to dependency installs like texttable which failed.

Version information

python-igraph version 0.10.1 I obtained from Github, also with igraph 0.10.1, which is installed via Spack (i.e. from the source archive on the release).

pip is version 22.0.4 and setuptools is 58.1.0.

Python 3.9.12

[salotz-sitx]

I was able to work around this mostly by running:

CFLAGS=$(pkg-config --cflags --libs igraph) pip install .

However this has the potential side effect of still allowing the fallback directories to be added to my gcc calls. Which shouldn't be a problem for me, but also shouldn't happen. So changes to the way that the fallback directories are activated should be implemented. And I would recommend you just skip having --use-pkg-config and just set it as a variable like I did above as I think the reason for the bug report with pip is because the setup.py and pip are doing funky things with arguments and they are colliding.

[vtraag]

This is presumably related to what is said here: https://github.com/igraph/python-igraph/issues/574#issuecomment-1258022997. Presumably, pip does not pass the option --use-pkgconfig during both build and install phases.

FYI, @ntamas, in order to make this work with an external igraph C library, I also introduced a --external flag, instead of --use-pkgconfig (also because pkg-config is not available on Windows) here. It might perhaps be easier indeed to simply do that and leave it to the user to make sure that all necessary flags are correctly setup, as @salotz-sitx also suggests.

[ntamas]

Okay, I've read the thread; let me summarize what's being proposed here so we can agree that we are on the same page.

Proposal nr. 1 is to stop using command line arguments and start relying on environment variables instead as they cannot be swallowed by pip. So we would have something like IGRAPH_USE_PKGCONFIG=1 and then the setup.py script would look for that, right?

Proposal nr. 2 is to prevent setup.py from making educated guesses about igraph's location by default and hide the entire --use-pkgconfig logic behind an --external flag (so if --external is not present, setup.py always uses the vendored copy of igraph, and if --external is present, then setup.py always uses an external, already installed copy).

We would also have to handle the combination when --use-pkgconfig is specified without --external; I believe we should print an error in this case and ask the user to add the --external flag, right?

[vtraag]

We would also have to handle the combination when --use-pkgconfig is specified without --external; I believe we should print an error in this case and ask the user to add the --external flag, right?

My proposal would be to remove the --use-pkgconfig all together. Users who want to install the Python interface of igraph and link to their own locally installed igraph C core should be able to set the necessary flags themselves. This could for instance be done using

CFLAGS=$(pkg-config --cflags --libs igraph) pip install .

as suggested, if people want to rely on pkg-config.

[szhorvat]

My proposal would be to remove the --use-pkgconfig all together.

Very likely, a lot of py-igraph recipes in various package managers depend on this already.

If you remove this, I hope you will help me fix the MacPorts port when it breaks ...

[salotz-sitx]

Try using --global-option instead of --install-option and let us know if that worked.

Same problem.

[vtraag]

Try using --global-option instead of --install-option and let us know if that worked.

Same problem.

Yes, indeed, I also tried this and also didn't work for me.

[szhorvat]

I deleted that soon after posting, you must have gotten it in email. I didn't look carefully, just checked the recipe here: https://github.com/macports/macports-ports/blob/master/python/py-igraph/Portfile#L48 The --global-option there is something different though.

[salotz-sitx]

Just for your information I am making a package for Spack.

My only suggestion would be to support an environment variable as an alternative to any flags you want to keep. So if you do add --external please add an equivalent environment variable.

That said I think the "modern" way to pass arguments to the build backend is to use the config_settings as described in PEP 517: https://peps.python.org/pep-0517/#config-settings

E.g. in pip after 22.1:

pip install --config-settings external=True .

I think this would alleviate the other issues with running --global-option and --install-option giving this warning:

WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.

I don't think its reasonable yet to expect a pip version that high for every installation, but just wanted you to be aware.

[salotz-sitx]

And I guess as a general wish, flags that turn off all "guessing" on the part of the build script is really great for packagers :) I do understand the desire for other use cases to have the magic discovery, but it makes a packager's job much harder.

[ntamas]

See the commit referenced above; this one tweaks the setup script so you can now use IGRAPH_USE_PKG_CONFIG=1 instead of --use-pkg-config.

I'm about to release a new version soon so I was wondering if you could test this with your setup in Spack to see if it satisfies your needs.

[salotz-sitx]

I will test it today

[salotz-sitx]

My initial test is failing, and is not finding the libraries. I will do some more debugging as time allows.

[salotz-sitx]

Actually the problem is on my end, but leads to another question on version compatibility between python-igraph and igraph itself. Is there any compatibility matrix anywhere?

[salotz-sitx]

Okay I can confirm that the new option works, but I would appreciate an answer to the compatibility matrix for writing my packaging recipe. Before I was just using the same version number.

[iosonofabio]

I don't think we advertise an official matrix. If you pair commit tags in this repo with the hash of the C core git submodule you can get a pretty extensive matrix though.

[szhorvat]

For version numbers of the form 0.x.y, x must match between python-igraph and the C library.

Additionally, try to use the same version of C igraph that is bundled with each python-igraph. There are experimental functions in igraph which may change between 0.10.y releases, and break compatibility. We try to avoid breakages, but sometimes it becomes necessary, and in the next release (0.10.3) it will very likely happen. Additionally, new 0.10.y releases may introduce entirely new functions. Therefore I strongly suggest that you always update py-igraph and igraph at the same time in Spack.

[salotz-sitx]

I think this can be closed unless there were other features to be implemented.

[szhorvat]

but I would appreciate an answer to the compatibility matrix

The answer to this is now given in the README. See the "warning" at the end of this section: https://github.com/igraph/python-igraph/#linking-to-an-existing-igraph-installation

[salotz-sitx]

I ended up just pinning them 1:1. You usually don't want to do that for general dependencies, but I figured in this case it was okay since they are so tightly coupled. In fact I'm guessing this is why a lot of projects develop them in the same repo.

[szhorvat]

I ended up just pinning them 1:1.

That sounds reasonable.

Is there any issue left blocking the update in Spack, @salotz-sitx ?