crossenv
crossenv copied to clipboard
benfogle
Problem to compile a package with cffi dependencies
I've some problem to cross compile a package that uses the cffi dependency. Here is a part of my build script:
# install cffi for build env as recommended
build-pip install cffi
# just in case, I don't really know if it's usefull in this case
cross-expose cffi
# package that uses cryptography and so cffi
pip install azure-storage-blob==12.5.0
And the output in the console:
WARNING: CC is a compound command (['arm-gad-linux-gnueabi-gcc', '-march=armv7-a', '-mthumb-interwork', '-mfloat-abi=hard', '-mfpu=neon', '-mtune=cortex-a9', '--sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CC='arm-gad-linux-gnueabi-gcc' and CFLAGS='-march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'
WARNING: CXX is a compound command (['arm-gad-linux-gnueabi-g++', '-march=armv7-a', '-mthumb-interwork', '-mfloat-abi=hard', '-mfpu=neon', '-mtune=cortex-a9', '--sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'])
WARNING: This can cause issues for modules that don't expect it.
WARNING: Consider setting CXX='arm-gad-linux-gnueabi-g++' and CXXFLAGS='-march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi'
Collecting cffi
Downloading https://files.pythonhosted.org/packages/50/ca/bbca0fd95b24a1d4f0d2e016f09f35ae68d4fe72bf34cc538d0a0d2d3e10/cffi-1.14.3-cp36-cp36m-manylinux1_x86_64.whl (400kB)
Collecting pycparser (from cffi)
Downloading https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl (112kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.14.3 pycparser-2.20
You are using pip version 18.1, however version 20.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Collecting azure-storage-blob==12.5.0
Downloading https://files.pythonhosted.org/packages/b8/84/7e51b3e1156bcb89a20b9ec641d4fced4800aa79daac3a403898c32046be/azure_storage_blob-12.5.0-py2.py3-none-any.whl (326kB)
Collecting azure-core<2.0.0,>=1.6.0 (from azure-storage-blob==12.5.0)
Downloading https://files.pythonhosted.org/packages/4c/fa/46974f4a7ad78b27e3eda8a573cc0c2508849f0d7d360b61c07cc5b46014/azure_core-1.8.2-py2.py3-none-any.whl (122kB)
Collecting msrest>=0.6.10 (from azure-storage-blob==12.5.0)
Downloading https://files.pythonhosted.org/packages/fa/f5/9e315fe8cb985b0ce052b34bcb767883dc739f46fadb62f05a7e6d6eedbe/msrest-0.6.19-py2.py3-none-any.whl (84kB)
Collecting cryptography>=2.1.4 (from azure-storage-blob==12.5.0)
Downloading https://files.pythonhosted.org/packages/5d/4b/7bb135c5787c003cdbc44990c5f41908f0f37135e0bb554e880d90fd5f6f/cryptography-3.1.1.tar.gz (534kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /tmp/python-cross-env/cross/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-k4ijf2ws --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools>=40.6.0 wheel "cffi>=1.8,!=1.11.3; platform_python_implementation != 'PyPy'":
*******************************************************
* Crossenv has leaked into another Python interpreter!
* You should probably file a bug report.
* Version 3.6.8 (default, Dec 25 2018, 00:00:00)
[GCC 4.8.4]
* Executable /tmp/python-cross-env/cross/bin/python3.6
*******************************************************
Collecting setuptools>=40.6.0
Using cached https://files.pythonhosted.org/packages/6d/38/c21ef5034684ffc0412deefbb07d66678332290c14bb5269c85145fbd55e/setuptools-50.3.2-py3-none-any.whl
Collecting wheel
Using cached https://files.pythonhosted.org/packages/a7/00/3df031b3ecd5444d572141321537080b40c1c25e1caa3d86cdd12e5e919c/wheel-0.35.1-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8
Downloading https://files.pythonhosted.org/packages/cb/ae/380e33d621ae301770358eb11a896a34c34f30db188847a561e8e39ee866/cffi-1.14.3.tar.gz (470kB)
Collecting pycparser (from cffi!=1.11.3,>=1.8)
Using cached https://files.pythonhosted.org/packages/ae/e7/d9c3a176ca4b02024debf82342dab36efadfc5776f9c8db077e8f6e71821/pycparser-2.20-py2.py3-none-any.whl
Installing collected packages: setuptools, wheel, pycparser, cffi
Running setup.py install for cffi: started
Running setup.py install for cffi: finished with status 'error'
Complete output from command /tmp/python-cross-env/cross/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-b363qank/cffi/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-dcmdpimn/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-k4ijf2ws --compile --install-headers /tmp/python-cross-env/cross/include/site/python3.6/cffi:
*******************************************************
* Crossenv has leaked into another Python interpreter!
* You should probably file a bug report.
* Version 3.6.8 (default, Dec 25 2018, 00:00:00)
[GCC 4.8.4]
* Executable /tmp/python-cross-env/cross/bin/python3.6
*******************************************************
*******************************************************
* Crossenv has leaked into another Python interpreter!
* You should probably file a bug report.
* Version 3.6.8 (default, Dec 25 2018, 00:00:00)
[GCC 4.8.4]
* Executable /tmp/python-cross-env/cross/bin/python3.6
*******************************************************
running install
running build
running build_py
creating build
creating build/lib.linux-arm-3.6
creating build/lib.linux-arm-3.6/cffi
copying cffi/api.py -> build/lib.linux-arm-3.6/cffi
copying cffi/backend_ctypes.py -> build/lib.linux-arm-3.6/cffi
copying cffi/setuptools_ext.py -> build/lib.linux-arm-3.6/cffi
copying cffi/ffiplatform.py -> build/lib.linux-arm-3.6/cffi
copying cffi/verifier.py -> build/lib.linux-arm-3.6/cffi
copying cffi/error.py -> build/lib.linux-arm-3.6/cffi
copying cffi/cffi_opcode.py -> build/lib.linux-arm-3.6/cffi
copying cffi/model.py -> build/lib.linux-arm-3.6/cffi
copying cffi/recompiler.py -> build/lib.linux-arm-3.6/cffi
copying cffi/vengine_gen.py -> build/lib.linux-arm-3.6/cffi
copying cffi/__init__.py -> build/lib.linux-arm-3.6/cffi
copying cffi/cparser.py -> build/lib.linux-arm-3.6/cffi
copying cffi/lock.py -> build/lib.linux-arm-3.6/cffi
copying cffi/commontypes.py -> build/lib.linux-arm-3.6/cffi
copying cffi/vengine_cpy.py -> build/lib.linux-arm-3.6/cffi
copying cffi/pkgconfig.py -> build/lib.linux-arm-3.6/cffi
copying cffi/_cffi_include.h -> build/lib.linux-arm-3.6/cffi
copying cffi/parse_c_type.h -> build/lib.linux-arm-3.6/cffi
copying cffi/_embedding.h -> build/lib.linux-arm-3.6/cffi
copying cffi/_cffi_errors.h -> build/lib.linux-arm-3.6/cffi
running build_ext
building '_cffi_backend' extension
creating build/temp.linux-arm-3.6
creating build/temp.linux-arm-3.6/c
arm-gad-linux-gnueabi-gcc -march=armv7-a -mthumb-interwork -mfloat-abi=hard -mfpu=neon -mtune=cortex-a9 --sysroot=/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi -Wno-unused-result -Wsign-compare -DNDEBUG -g -O3 -Wall -fPIC -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/tmp/python-cross-env/cross/include -I/opt/gad/1.0.0/sysroots/cortexa9hf-vfp-neon-gad-linux-gnueabi/usr/include/python3.6m -c c/_cffi_backend.c -o build/temp.linux-arm-3.6/c/_cffi_backend.o
c/_cffi_backend.c:15:17: fatal error: ffi.h: No such file or directory
#include <ffi.h>
^
compilation terminated.
error: command 'arm-gad-linux-gnueabi-gcc' failed with exit status 1
You can see that the pip command uses --ignore-installed probably because I use a Debian environment to build, but I don't know if it's relevant. I've tried to deactivate it without success.
I'm not sure exactly what's happening, see if the following works for you:
$ build-pip install cffi pycparser
$ cross-expose cffi pycparser
$ pip install azure-storage-blob==12.5.0
I think the fact that cross-pip could see cffi but not its dependency, pycparser was causing issues. I'm not sure why, and it might be a regression somewhere.
I've tried what you suggest without success.
I should have specified that I cross-compile libffi and openssl upstream for other needs of my application. I don't know if it's relevant, but with these I managed to install the package by running crossenv and adding the libraries headers:
python3.6 -m crossenv --env CFLAGS="-I$(pwd)/libffi/arm-gad-linux-gnueabi/include -I${SYSROOT}/usr/local/ssl/include" "${SYSROOT}/usr/bin/python3.6" "${CROSSENV_PATH}"
. "${CROSSENV_PATH}/bin/activate"
# this is still mandatory as it doesn't compile without it
build-pip install cffi
pip install azure-storage-blob==12.5.0
I don't know if there's something I misunderstood about crossenv (very likely) or if it's a bug.
You need to prepare the toolchain properly. you need to cross compile libffi-dev and install it in toolchain to resolve the header/libraries related issue.
