zeromq
BUG: Fails to build on termux
This is a pyzmq bug
- [X] This is a pyzmq-specific bug, not an issue of zmq socket behavior. Don't worry if you're not sure! We'll figure it out together.
What pyzmq version?
master
What libzmq version?
master
Python version (and how it was installed)
3.11
OS
termux/android
What happened?
When I try to install via pip, it spews out cmake related errors.
~ $ pip install zmq
Collecting zmq
Using cached zmq-0.0.0.zip (2.2 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting pyzmq (from zmq)
Using cached pyzmq-26.0.3.tar.gz (267 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Installing backend dependencies ... error
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> [88 lines of output]
Collecting cmake>=3.14
Using cached cmake-3.29.3.tar.gz (30 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'done'
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status 'done'
Installing backend dependencies: started
Installing backend dependencies: finished with status 'error'
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 2
╰─> [65 lines of output]
Collecting cmake
Using cached cmake-3.29.3.tar.gz (30 kB)
ERROR: Exception:
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
status = run_func(*args)
^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
return func(self, options, args)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/commands/install.py", line 391, in run
requirement_set = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
result = self._result = resolver.resolve(
^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve
state = resolution.resolve(requirements, max_rounds=max_rounds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 397, in resolve
self._add_to_criteria(self.state.criteria, r, parent=None)
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
if not criterion.candidates:
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_vendor/resolvelib/structs.py", line 156, in __bool__
return bool(self._sequence)
^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
return any(self)
^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
return (c for c in iterator if id(c) not in self._incompatible_ids)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
candidate = func()
^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
self._link_candidate_cache[link] = LinkCandidate(
^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
super().__init__(
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
self.dist = self._prepare()
^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
dist = self._prepare_distribution()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
return self._prepare_linked_requirement(req, parallel_builds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 631, in _prepare_linked_requirement
dist = _get_prepared_distribution(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/prepare.py", line 68, in _get_prepared_distribution
with build_tracker.track(req):
File "/data/data/com.termux/files/usr/lib/python3.11/contextlib.py", line 137, in __enter__
return next(self.gen)
^^^^^^^^^^^^^^
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py", line 122, in track
self.add(req)
File "/data/data/com.termux/files/usr/lib/python3.11/site-packages/pip/_internal/operations/build/build_tracker.py", line 92, in add
raise LookupError(message)
LookupError: https://files.pythonhosted.org/packages/9e/cf/fbd014f36d3407f04fcbd4d216a1431ef05f2ec69a25a2ad3af1de75e835/cmake-3.29.3.tar.gz (from https://pypi.org/simple/cmake/) (requires-python:>=3.7) is already being built: cmake>=3.14 from https://files.pythonhosted.org/packages/9e/cf/fbd014f36d3407f04fcbd4d216a1431ef05f2ec69a25a2ad3af1de75e835/cmake-3.29.3.tar.gz
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 2
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install backend dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
I tried following the old advice from here, but it seems pyzmq was recently upgraded to cmake infra so it no longer is true.
Code to reproduce bug
No response
Traceback, if applicable
No response
More info
No response
building pyzmq from source requires cmake. Try installing cmake first. scikit-build-core tries to install cmake via a wheel if it's not present, but it doesn't work everywhere.
Thanks, actually I already did that, but it seems cmake required libxml2 to also be installed. Installing that fixed the current error, but now I get another one:
~ $ pip install zmq
Collecting zmq
Using cached zmq-0.0.0.zip (2.2 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting pyzmq (from zmq)
Using cached pyzmq-26.0.3.tar.gz (267 kB)
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: zmq, pyzmq
Building wheel for zmq (pyproject.toml) ... done
Created wheel for zmq: filename=zmq-0.0.0-py3-none-any.whl size=1262 sha256=6873696fe49cc5eca00f165831e304a886bfd8a0899c4f33cbd11f0ccb050318
Stored in directory: /data/data/com.termux/files/home/.cache/pip/wheels/16/9e/c7/d497825227491fa00cca08b88ae37f9bcc14809233db76342c
Building wheel for pyzmq (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for pyzmq (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [6 lines of output]
*** scikit-build-core 0.9.5 using CMake 3.29.4 (wheel)
*** Configuring CMake...
loading initial cache file /data/data/com.termux/files/usr/tmp/tmp_avl48zb/build/CMakeInit.txt
/home/builder/.termux-build/libuv/src/src/unix/process.c:957: assertion "!(options->flags & ~(UV_PROCESS_DETACHED | UV_PROCESS_SETGID | UV_PROCESS_SETUID | UV_PROCESS_WINDOWS_HIDE | UV_PROCESS_WINDOWS_HIDE_CONSOLE | UV_PROCESS_WINDOWS_HIDE_GUI | UV_PROCESS_WINDOWS_VERBATIM_ARGUMENTS))" failed
*** CMake configuration failed
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pyzmq
Successfully built zmq
Failed to build pyzmq
ERROR: Could not build wheels for pyzmq, which is required to install pyproject.toml-based projects
Hm, it looks like the cmake wheel built, but it doesn't work. Can you install cmake with your package manager and check the version? I don't know anything about termux.
I already installed cmake (and libxml2) otherwise we get the first error. cmake version is 3.29.4
It seems the cmake error is related to process flags.
Something's fishy, since it says
scikit-build-core 0.9.5 using CMake 3.29.4 (wheel)
which suggests scikit-build-core is not finding an acceptable cmake, and instead using the cmake wheel, which appears to not be built properly. What's are which -a cmake and cmake --version in the shell before you call pip install?
Also note, the package is not called zmq, it's pyzmq. zmq is a typo-squatting package that only depends on pyzmq.
which shows the path as /data/data/com.termux/files/usr/bin/cmake. The cmake version is 3.29.4.
I tried building the docker image, and got a pyzmq wheel file for aarch64. Installed that, but it couldn't find libzmq.so which is in /data/data/com.termux/files/usr/lib, so then I set LD_LIBRARY_PATH. Now when trying to import zmq, I get ImportError: dlopen failed: cannot locate symbol "PyExc_ImportError" referenced by "/data/data/com.termux/files/usr/lib/python3.11/site-packages/zmq/backend/cython/_zmq.cpython-311.so"...
I guess it's not to be... 😁
I guess it's not to be... 😁
FYI I just got it to install with
SODIUM_INSTALL=system CC=clang pip install --no-binary=pyzmq pyzmq
@tribixbite but does it also load? That's where I run into trouble.
The thing I needed it for worked after install and
python - c "import zmq; print('pyzmq loaded successfully')"
returns "pyzmq loaded successfully", so... seems to load ok
That can be tried once cibuildwheel supports android: https://github.com/pypa/cibuildwheel/issues/1960
cibuildwheel 3.1.0 has now been released with Android support.
good news, pyzmq can build android wheel for next release.
pyzmq 27.0.1 is out with android wheels. CI can't test them, so I have no idea if they work, but they exist! Any reports on whether they work or not would be much appreciated.
Thanks for your work, I notice pyzmq v27.0.1 have released x86_64 and arm64_v8a versions for android, would u offer armeabi_v7a version for some arm32 devices? @minrk
cibuildwheel doesn't appear to support them. Until/unless that changes, I don't think I can.
@mhsmith Hi, does cibuildwheel support to build armeabi_v7a wheel for android?
CI can't test them, so I have no idea if they work, but they exist! Any reports on whether they work or not would be much appreciated.
I've created #2117 to enable testing Android in CI.