pip_install fails with `error: command '/usr/bin/gcc' failed with exit code 1`

[davians12]

🐞 bug report

Affected Rule

The issue is caused by the rule: pip_install

Is this a regression?

Yes, the previous version in which this bug was not present was: Not a regression

Description

pip_install fails for only one requirement `tree-sitter` from `requirements.txt`

🔥 Exception or Error

Failed to build tree-sitter
 (  ERROR: Command errored out with exit status 1:
   command: /home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/python_bin /home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/pypi__pip/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpxxrp34vx
       cwd: /tmp/pip-wheel-hw8ksvfm/tree-sitter
  Complete output (20 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/tree_sitter
  copying tree_sitter/__init__.py -> build/lib.linux-x86_64-3.7/tree_sitter
  running build_ext
  building 'tree_sitter.binding' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/tree_sitter
  creating build/temp.linux-x86_64-3.7/tree_sitter/core
  creating build/temp.linux-x86_64-3.7/tree_sitter/core/lib
  creating build/temp.linux-x86_64-3.7/tree_sitter/core/lib/src
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g0 -fPIC -Itree_sitter/core/lib/include -Itree_sitter/core/lib/src -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/config-3.7m-x86_64-linux-gnu/Include -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter -c tree_sitter/binding.c -o build/temp.linux-x86_64-3.7/tree_sitter/binding.o -std=c99 -Wno-unused-variable
tree_sitter/binding.c:2:10: fatal error: Python.h: No such file or directory
   #include "Python.h"
            ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for tree-sitter
ERROR: Failed to build one or more wheels
Traceback (most recent call last):
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/rules_python/python/pip_install/extract_wheels/__main__.py", line 5, in 
    main()
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/rules_python/python/pip_install/extract_wheels/__init__.py", line 87, in main
    subprocess.run(pip_args, check=True)
  File "/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/python_bin', '-m', 'pip', 'wheel', '-r', '/home/user/natural-cubert/requirements.txt']' returned non-zero exit status 1.
)

🌍 Your Environment

Operating System:

  
   Ubuntu
  

Output of bazel version:

  
Build label: 4.0.0
Build target: bazel-out/k8-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Thu Jan 21 07:33:24 2021 (1611214404)
Build timestamp: 1611214404
  

Rules_python version:

  
0.1.0
  

Anything else relevant? The bazel setup used to build and run, not sure what changed.

[hrfuller]

I believe the issue lies with the hermetic interpreters which are now used, apparently by default. The installed interpreters libs don't seem have the correct header files present.

[hrfuller]

@davians12 I can't repro this exact issue because I develop on macos. Can you put together a repro of the WORKSPACE which create this issue. In particular are you using a custom python interpreter toolchain target or the built in hermetic toolchains?

[thundergolfer]

@hrfuller Yep this is totally expected with the python-build-standalone intepreter.

pip_install fails for only one requirement tree-sitter from requirements.txt

It's failed to find a wheel for tree-sitter so fell back to compiling it. It appears tree-sitter doesn't distribute a Linux wheel. https://pypi.org/project/tree-sitter/#files

[hrfuller]

So this issue is basically the linux equivalent of bazelbuild/rules_python#646 but in the integration test I added to #679 I can't repro this failure with linux. To me that implies that the interpreter setup here is slightly different. In particular this include path -I/home/user/.cache/bazel/_bazel_user/1ced21babfe30b5e39b8b34411e58735/external/python_interpreter/bazel_install/lib/python3.7/config-3.7m-x86_64-linux-gnu/Include doesn't look like it comes from the standalone interpreters because the external repo name in the path doesn't have the standard form that rules python creates e.g something like `python39_x86_64-apple-darwin. Thats why I wanted to see a repro.

[aaliddell]

I am seeing the same thing, using a custom Python toolchain and the hiredis and python-prctl packages. The build works with 0.7.0 and fails with 0.8.0, so I have bisected the issue back to fe30f1561e3afef59e3f8be6339c88444100de25 where a set of dependencies were updated. It appears that new combination of packages has changed how the include paths are being found when creating the gcc compile args, resulting in the -I flags pointing to the wrong locations.

I've tried looking through the issues on those packages for relevant changes, but not found anything obvious yet. The simple test would be to update these packages one-by-one or just bulk upgrade all of these to their latest version to see if there was a bug that's been resolved.

This works (commit prior to fe30f1561e3afef59e3f8be6339c88444100de25):

http_archive(
    name = "rules_python",
    sha256 = "00360543865915c7492371b631e6824e1bd6976c88062242a65da0e6b18618c0",
    strip_prefix = "rules_python-0f189979467de61cb0512cc0a133d2122e041e90",
    url = "https://github.com/bazelbuild/rules_python/archive/0f189979467de61cb0512cc0a133d2122e041e90.zip",
)

This fails:

http_archive(
    name = "rules_python",
    sha256 = "18a8d97f706d768169f93e251d02eea6f8e1810bd90c2fe78cfdf036aa1772ef",
    strip_prefix = "rules_python-fe30f1561e3afef59e3f8be6339c88444100de25",
    url = "https://github.com/bazelbuild/rules_python/archive/fe30f1561e3afef59e3f8be6339c88444100de25.zip",
)

@davians12: Do you see the same behaviour when you use the above rules_python versions?

I'll try to extract a repro repo too.

[aaliddell]

So, this has been a rabbit hole into Python builds. The repo I have that reproduces this is similar in setup to the demo code you have here in this repo, where Python is pulled and built in patch commands. Given the .../bazel_install/... path segment shown in the top comment, I would guess @davians12 has the same setup, as this was a common-ish setup prior to the hermetic Python introduction recently. This has always been a bit of a hack and it seems we are finally seeing a consequence of that.

The package update that breaks the build is setuptools between v60.1.1 and v60.2.0. Looking at the diff, there are changes to the way Python build config files are detected, which appears to have a knock-on effect in how the include paths are generated in build_ext when building the wheel.

The section of interest appears to be in how distutils checks if a Python executable is part of a 'non-installed' build and changes behaviour if so. The critical piece here is the final Python binary is symlinked back into the source tree, which makes distutils think we are in an uninstalled Python build due to the presence of the source files alongside the binary.

So, as far as I can tell, this is not a bug in rules_python nor any of the upstream Python packages, but rather a latent problem in mangling Python build directories that hasn't mattered until an unrelated cleanup in distutils exposed it.

The fix for users with this build setup is to ensure that the final binary/symlink that is given to your toolchain/pip_install/pip_parse is not positioned alongside the source files, to prevent distutils getting confused. I have done this by separating the source and build dirs in the @python repo. Alternatively, switch to the new hermetic toolchains.

[groodt]

Coincidentally running into the same issue. The setuptools upgrade was introduced here: https://github.com/bazelbuild/rules_python/pull/661

[groodt]

I'm currently winding back setuptools versions to find out when it breaks in my environment. I'm not sure how necessary #661 was. I think we need to be more careful with setuptools upgrades and stay on old versions as long as possible.

[aaliddell]

Are you running with the new built-in toolchains or a custom Python toolchain?

Rolling back setuptools may be a short-term solution, but that only hides the issue and we should be able to upgrade it freely in the future (particularly as new Python releases come out that could start to have issues with a very old setuptools version). In this particular case for this issue: there's nothing that rules_python can fix (besides maybe updating the demo code I linked), as it's my local custom toolchain layout that was breaking distutils. It just so happens that a fair number of us have adopted and adapted that original demo code, so are seeing similar bugs.

[groodt]

My issue is now resolved after upgrading rules_python to the latest and upgrading the failing dependency to a wheel, rather than relying on the sdist. Relying on the sdist was fragile to setuptools and distutils changes as you mention.

I no longer believe I was seeing the same issue, so I think we still need to see a repro from the original poster to be certain (although I do think @aaliddell has the best explanation so far).

[hrfuller]

Agreed the @aaliddell bisect makes sense to me.

[groodt]

@davians12 Can you confirm whether @aaliddell fix solves your issue?

[aaliddell]

A useful closure here could be to update the example python build toolchain here to fix the symlink being inside the source tree.

[garymm]

For me this was fixed on MacOS by version 0.12.0. From the release notes, I'm guessing fixed by https://github.com/bazelbuild/rules_python/pull/809

[github-actions[bot]]

This issue was automatically closed because it went 30 days without a reply since it was labeled "Can Close?"