pip icon indicating copy to clipboard operation
pip copied to clipboard
verified publisher icon pypa

Installed dependency could not be found in when built on Alpine Linux amd64

Open notxvilka opened this issue 9 months ago • 5 comments

Description

When installing Angr from the sources on the Alpine Linux container I met this very weird bug, where one of the dependencies pyvex wasn't recognized by pip despite it being installed and appearing in the pip3 list and pip3 show, the exact same version that is in Angr requirements.

Expected behavior

pyvex recognized as installed and the build of angr proceed normally

pip version

pip 25.0.1 from /home/piptest/.local/lib/python3.12/site-packages/pip (python 3.12)

Python version

Python 3.12.9

OS

Alpine Linux 3.21 amd64

How to Reproduce

Create this Dockerfile:

FROM amd64/alpine:3.21
RUN apk update && apk add --no-cache python3 py3-pip cmake make gcc g++ git
RUN adduser --disabled-password piptest
USER piptest
WORKDIR /home/piptest

RUN git clone --recursive https://github.com/angr/pyvex \
	&& git clone https://github.com/angr/claripy \
	&& git clone https://github.com/angr/archinfo \
	&& git clone https://github.com/angr/cle \
	&& git clone https://github.com/angr/ailment \
	&& git clone https://github.com/angr/angr 

RUN (cd pyvex && git checkout f66825b4ff7c7a6343116b579f61fa538e4e33b9) \
	&& (cd claripy && git checkout 589fbc0ee1fe0c4cea645592d295cf69cc82207f) \
	&& (cd archinfo && git checkout 2683af884255f3593b134a0ddfadc4e2c7c1c35f) \
	&& (cd cle && git checkout b0765b88dee76839bc7a230f878c4e4a2cc9fe9a) \
	&& (cd ailment && git checkout 1602ffce331d6c43761c7e4ed73cc7b293b697af) \
	&& (cd angr && git checkout 2dbb3df3bbb0f9aea10a699d3c7686ffb23fdffd)

ENV PIP_BREAK_SYSTEM_PACKAGES=1

RUN pip3 install --user --upgrade pip

RUN (cd pyvex && pip3 install --user .) \
	&& (cd archinfo && pip3 install --user .) \
	&& (cd cle && pip3 install --user .)

RUN pip3 list && pip3 show pyvex
RUN python3 --version && pip3 --version

RUN (cd ailment && pip3 install --user .) \
	&& (cd angr && pip3 install --user .)

RUN pip3 list && pip3 show angr

Then build it with either podman or docker: docker build .

Output

Processing /home/piptest/angr
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  error: subprocess-exited-with-error
  
  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Collecting setuptools>=61.2
        Using cached setuptools-75.8.2-py3-none-any.whl.metadata (6.7 kB)
      Collecting wheel
        Using cached wheel-0.45.1-py3-none-any.whl.metadata (2.3 kB)
      ERROR: Could not find a version that satisfies the requirement pyvex==9.2.146.dev0 (from versions: 3.
1, 3.11, 3.12, 4.5.9.9b0, 4.5.9.9, 4.5.9.13, 4.5.9.14, 4.5.9.29, 4.5.10.14, 4.5.11.23, 4.5.12.12, 4.5.12.21
, 4.6.1.4b0, 4.6.1.4, 4.6.1.4.post1, 4.6.1.4.post2, 4.6.1.4.post3, 4.6.1.5, 4.6.1.27, 4.6.3.15, 4.6.3.28, 4
.6.4.28, 4.6.5.27, 4.6.6.28, 5.6.8.22, 5.6.8.25, 5.6.10.5, 5.6.12.3, 6.7.1.13, 6.7.1.13.post1, 6.7.1.13.pos
t2, 6.7.1.31, 6.7.3.26, 6.7.4.12, 6.7.6.9, 6.7.7.27, 7.7.9.8, 7.7.9.14, 7.7.12.16, 7.8.2.21, 7.8.6.16, 7.8.
6.23, 7.8.7.1, 7.8.8.1, 8.18.10.5, 8.18.10.25, 8.19.2.4, 8.19.4.5, 8.19.7.25, 8.19.10.29, 8.19.10.30, 8.20.
1.7, 8.20.5.27, 8.20.6.1, 8.20.6.8, 8.20.7.6, 8.20.7.27, 9.0.4378, 9.0.4446, 9.0.4495, 9.0.4663, 9.0.4885, 
9.0.4940, 9.0.5002, 9.0.5034, 9.0.5171, 9.0.5326, 9.0.5327, 9.0.5376, 9.0.5405, 9.0.5450, 9.0.5610, 9.0.573
9, 9.0.5811, 9.0.5903, 9.0.6136, 9.0.6281, 9.0.6421, 9.0.6488, 9.0.6588, 9.0.6642, 9.0.6790, 9.0.6852, 9.0.
6885, 9.0.7293, 9.0.7491, 9.0.7833, 9.0.7912, 9.0.8021, 9.0.8761, 9.0.9031, 9.0.9166, 9.0.9297, 9.0.9355, 9
.0.9438, 9.0.9506, 9.0.9572, 9.0.9684, 9.0.9792, 9.0.9947, 9.0.10010, 9.0.10055, 9.0.10072, 9.0.10159, 9.0.
10281, 9.0.10339, 9.0.10409, 9.0.10534, 9.0.10576, 9.0.10651, 9.0.10689, 9.0.10730, 9.1.10913, 9.1.11508, 9
.1.11611, 9.1.11752, 9.1.12332, 9.2.1, 9.2.2, 9.2.3, 9.2.4, 9.2.5, 9.2.6, 9.2.7, 9.2.8, 9.2.9, 9.2.10, 9.2.
11, 9.2.12, 9.2.13, 9.2.14, 9.2.15, 9.2.16, 9.2.17, 9.2.18, 9.2.19, 9.2.20, 9.2.21, 9.2.22, 9.2.23, 9.2.24,
 9.2.25, 9.2.26, 9.2.27, 9.2.28, 9.2.29, 9.2.30, 9.2.31, 9.2.32, 9.2.33, 9.2.34, 9.2.35, 9.2.36, 9.2.37, 9.
2.38, 9.2.39, 9.2.40, 9.2.41, 9.2.42, 9.2.43, 9.2.44, 9.2.45, 9.2.46, 9.2.47, 9.2.48, 9.2.49, 9.2.50, 9.2.5
1, 9.2.52, 9.2.53, 9.2.54, 9.2.55, 9.2.56, 9.2.57, 9.2.58, 9.2.59, 9.2.60, 9.2.61, 9.2.62, 9.2.63, 9.2.64, 
9.2.65, 9.2.66, 9.2.67, 9.2.68, 9.2.69, 9.2.70, 9.2.71, 9.2.72, 9.2.73, 9.2.74, 9.2.75, 9.2.76, 9.2.77, 9.2
.78, 9.2.79, 9.2.80, 9.2.81, 9.2.82, 9.2.83, 9.2.84, 9.2.85, 9.2.86, 9.2.87, 9.2.88, 9.2.89, 9.2.90, 9.2.91
, 9.2.92, 9.2.93, 9.2.94, 9.2.95, 9.2.96, 9.2.97, 9.2.98, 9.2.99, 9.2.100, 9.2.101, 9.2.102, 9.2.103, 9.2.1
04, 9.2.105, 9.2.106, 9.2.107, 9.2.108, 9.2.109, 9.2.110, 9.2.111, 9.2.112, 9.2.113, 9.2.114, 9.2.115, 9.2.
116, 9.2.117, 9.2.118, 9.2.119, 9.2.120, 9.2.121, 9.2.122, 9.2.123, 9.2.124, 9.2.125, 9.2.126, 9.2.127, 9.2
.128, 9.2.129, 9.2.130, 9.2.131, 9.2.132, 9.2.133, 9.2.134, 9.2.135, 9.2.136, 9.2.137, 9.2.138, 9.2.139, 9.
2.140, 9.2.141, 9.2.142, 9.2.143, 9.2.144, 9.2.145)
      ERROR: No matching distribution found for pyvex==9.2.146.dev0
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

At the same time, pip3 show and pip3 list show pyvex is installed and is of right version:

STEP 11/14: RUN pip3 list && pip3 show pyvex
Package          Version
---------------- ------------
archinfo         9.2.146.dev0
bitarray         3.0.0
bitstring        4.3.0
cart             1.2.3
cffi             1.17.1
cle              9.2.146.dev0
packaging        24.2
pefile           2024.8.26
pip              25.0.1
pycparser        2.22
pycryptodome     3.21.0
pyelftools       0.32
pyparsing        3.1.4
pyvex            9.2.146.dev0
setuptools       70.3.0
sortedcontainers 2.4.0
Name: pyvex
Version: 9.2.146.dev0
Summary: A Python interface to libVEX and VEX IR
Home-page: https://github.com/angr/pyvex
Author: 
Author-email: 
License: BSD-2-Clause
Location: /home/piptest/.local/lib/python3.12/site-packages
Requires: bitstring, cffi
Required-by: cle

Code of Conduct

notxvilka avatar Mar 07 '25 14:03 notxvilka

Can you reproduce this without using --break-system-packages? This would require using a virtual environment rather than the system Python, but should otherwise be more or less the same reproducer.

pfmoore avatar Mar 07 '25 15:03 pfmoore

where one of the dependencies pyvex wasn't recognized by pip despite it being installed and appearing in the pip3 list and pip3 show, the exact same version that is in Angr requirements.

By default, when building from source, pip does not build using your current environment, it creates a new isolated environment and uses the specified build dependencies. The error you got:

ERROR: No matching distribution found for pyvex==9.2.146.dev0

Makes sense, because you can not upload a dev version of a package PyPI. You have the following options:

  • Change the build requirements of the package being built to a version of pyvex that can be installed from PyPI
  • Point pip to an index that contains pyvex==9.2.146.dev0
  • Use --no-build-isolation when installing the package that requires pyvex to build, you will be responsible for making sure build requirements are installed into your environment

notatallshaw avatar Mar 07 '25 15:03 notatallshaw

Can you reproduce this without using --break-system-packages? This would require using a virtual environment rather than the system Python, but should otherwise be more or less the same reproducer.

Yes, it happens with venv too, using this Dockerfile:

FROM amd64/alpine:3.21
RUN apk update && apk add --no-cache python3 py3-pip cmake make gcc g++ git
RUN adduser --disabled-password piptest
USER piptest
WORKDIR /home/piptest

RUN git clone --recursive https://github.com/angr/pyvex \
	&& git clone https://github.com/angr/claripy \
	&& git clone https://github.com/angr/archinfo \
	&& git clone https://github.com/angr/cle \
	&& git clone https://github.com/angr/ailment \
	&& git clone https://github.com/angr/angr 

RUN (cd pyvex && git checkout f66825b4ff7c7a6343116b579f61fa538e4e33b9) \
	&& (cd claripy && git checkout 589fbc0ee1fe0c4cea645592d295cf69cc82207f) \
	&& (cd archinfo && git checkout 2683af884255f3593b134a0ddfadc4e2c7c1c35f) \
	&& (cd cle && git checkout b0765b88dee76839bc7a230f878c4e4a2cc9fe9a) \
	&& (cd ailment && git checkout 1602ffce331d6c43761c7e4ed73cc7b293b697af) \
	&& (cd angr && git checkout 2dbb3df3bbb0f9aea10a699d3c7686ffb23fdffd)

RUN python3 -m venv testvenv && source testvenv/bin/activate && pip3 install --upgrade pip

RUN source testvenv/bin/activate && (cd pyvex && pip3 install .) \
	&& (cd archinfo && pip3 install .) \
	&& (cd cle && pip3 install .)

RUN source testvenv/bin/activate && pip3 list && pip3 show pyvex
RUN source testvenv/bin/activate && python3 --version && pip3 --version

RUN source testvenv/bin/activate && (cd ailment && pip3 install .) \
	&& (cd angr && pip3 install .)

RUN source testvenv/bin/activate && pip3 list && pip3 show angr

notxvilka avatar Mar 07 '25 15:03 notxvilka

Change the build requirements of the package being built to a version of pyvex that can be installed from PyPI

Unsuitable for me, due to the testing and development needs

Point pip to an index that contains pyvex==9.2.146.dev0

I tried using with building wheels locally with pip3 wheel -w $WHEELSDIR and then using pip3 install --find-links $WHEELSDIR but the exactly same error happens

Use --no-build-isolation when installing the package that requires pyvex to build

If I try this approach, using these lines:

RUN source testvenv/bin/activate && pip3 install setuptools && (cd ailment && pip3 install .) \
	&& (cd angr && pip3 install --no-build-isolation .)

then it can't find previously installed claripy module:

Requirement already satisfied: cffi>=1.14.0 in /home/piptest/testvenv/lib/python3.12/site-packages (from an
gr==9.2.146.dev0) (1.17.1)
INFO: pip is looking at multiple versions of angr to determine which version is compatible with other requi
rements. This could take a while.
ERROR: Could not find a version that satisfies the requirement claripy==9.2.146.dev0 (from angr) (from vers
ions: 0.9.0, 0.9.0.1, 0.9.1, 4.5.9.9b0, 4.5.9.9, 4.5.9.13, 4.5.9.14, 4.5.9.29, 4.5.10.14, 4.5.11.23, 4.5.12
.12, 4.5.12.21, 4.6.1.18, 4.6.1.27, 4.6.2.25, 4.6.3.15, 4.6.3.18, 4.6.3.28, 4.6.4.28, 4.6.5.27, 4.6.6.4, 4.
6.6.4.post1, 4.6.6.10, 4.6.6.15, 4.6.6.16, 4.6.6.28, 4.6.6.29, 5.6.8.22, 5.6.10.5, 5.6.10.12, 5.6.12.3, 6.7
.1.13, 6.7.1.13.post2, 6.7.1.31, 6.7.3.26, 6.7.4.12, 6.7.6.9, 6.7.7.27, 7.7.9.8, 7.7.9.14, 7.7.9.21, 7.7.12
.16, 7.8.2.21, 7.8.6.16, 7.8.8.1, 8.18.10.5, 8.18.10.25, 8.19.2.4, 8.19.4.5, 8.19.7.25, 8.19.10.29, 8.19.10
.30, 8.20.1.7, 8.20.5.27, 8.20.6.1, 8.20.6.8, 8.20.7.6, 8.20.7.27, 9.0.4378, 9.0.4446, 9.0.4495, 9.0.4663, 
9.0.4885, 9.0.4940, 9.0.5002, 9.0.5034, 9.0.5171, 9.0.5326, 9.0.5327, 9.0.5376, 9.0.5405, 9.0.5450, 9.0.561
0, 9.0.5739, 9.0.5811, 9.0.5903, 9.0.6136, 9.0.6281, 9.0.6421, 9.0.6488, 9.0.6588, 9.0.6642, 9.0.6790, 9.0.
6852, 9.0.6885, 9.0.7293, 9.0.7491, 9.0.7833, 9.0.7912, 9.0.8021, 9.0.8761, 9.0.9031, 9.0.9166, 9.0.9297, 9
.0.9355, 9.0.9438, 9.0.9506, 9.0.9572, 9.0.9684, 9.0.9792, 9.0.9947, 9.0.10010, 9.0.10055, 9.0.10072, 9.0.1
0159, 9.0.10281, 9.0.10339, 9.0.10409, 9.0.10534, 9.0.10576, 9.0.10651, 9.0.10689, 9.0.10730, 9.1.10913, 9.
1.11508, 9.1.11611, 9.1.11752, 9.1.12332, 9.2.1, 9.2.2, 9.2.3, 9.2.4, 9.2.5, 9.2.6, 9.2.7, 9.2.8, 9.2.9, 9.
2.10, 9.2.11, 9.2.12, 9.2.13, 9.2.14, 9.2.15, 9.2.16, 9.2.17, 9.2.18, 9.2.19, 9.2.20, 9.2.21, 9.2.22, 9.2.2
3, 9.2.24, 9.2.25, 9.2.26, 9.2.27, 9.2.28, 9.2.29, 9.2.30, 9.2.31, 9.2.32, 9.2.33, 9.2.34, 9.2.35, 9.2.36, 
9.2.37, 9.2.38, 9.2.39, 9.2.40, 9.2.41, 9.2.42, 9.2.43, 9.2.44, 9.2.45, 9.2.46, 9.2.47, 9.2.48, 9.2.49, 9.2
.50, 9.2.51, 9.2.52, 9.2.53, 9.2.54, 9.2.55, 9.2.56, 9.2.57, 9.2.58, 9.2.59, 9.2.60, 9.2.61, 9.2.62, 9.2.63
, 9.2.64, 9.2.65, 9.2.66, 9.2.67, 9.2.68, 9.2.69, 9.2.70, 9.2.71, 9.2.72, 9.2.73, 9.2.74, 9.2.75, 9.2.76, 9
.2.77, 9.2.78, 9.2.79, 9.2.80, 9.2.81, 9.2.82, 9.2.83, 9.2.84, 9.2.85, 9.2.86, 9.2.87, 9.2.88, 9.2.89, 9.2.
90, 9.2.91, 9.2.92, 9.2.93, 9.2.94, 9.2.95, 9.2.96, 9.2.97, 9.2.98, 9.2.99, 9.2.100, 9.2.101, 9.2.102, 9.2.
103, 9.2.104, 9.2.105, 9.2.106, 9.2.107, 9.2.108, 9.2.109, 9.2.110, 9.2.111, 9.2.112, 9.2.113, 9.2.114, 9.2
.115, 9.2.116, 9.2.117, 9.2.118, 9.2.119, 9.2.120, 9.2.121, 9.2.122, 9.2.123, 9.2.124, 9.2.125, 9.2.126, 9.
2.127, 9.2.128, 9.2.129, 9.2.130, 9.2.131, 9.2.132, 9.2.133, 9.2.134, 9.2.135, 9.2.136, 9.2.137, 9.2.138, 9
.2.139, 9.2.140, 9.2.141, 9.2.142, 9.2.143, 9.2.144, 9.2.145)
ERROR: No matching distribution found for claripy==9.2.146.dev0
Error: building at STEP "RUN source testvenv/bin/activate && pip3 install setuptools && (cd ailment && pip3
 install .)     && (cd angr && pip3 install --no-build-isolation .)": while running runtime: exit status 1

notxvilka avatar Mar 07 '25 15:03 notxvilka

I tried using with building wheels locally with pip3 wheel -w $WHEELSDIR and then using pip3 install --find-links $WHEELSDIR but the exactly same error happens

Some options are or aren't passed to the isolated build environment, I would have assumed find-links is, but I've not looked at the code and there may be a reason if it's not.

You can always set an environmental variable ENV PIP_FIND_LINKS=$WHEELSDIR to make sure that the pip build subprocess pick this up.

then it can't find previously installed claripy module

As I said, when you use --no-build-isolation you are responsible for making sure the environment has the build requirements, if you take this route I would suggest doing it strategically for only packages that you know you need to build and you know all the build requirements ahead of time.

notatallshaw avatar Mar 07 '25 15:03 notatallshaw