cibuildwheel icon indicating copy to clipboard operation
cibuildwheel copied to clipboard

[BUG] Linux builds do not respect `.dockerignore`

Open XuehaiPan opened this issue 1 year ago • 1 comments

Description

I have an editable local installation in my project directory. It contains a .so file build in AMD64.

While building a 32-bit wheel in the docker container with:

$ python3 -m cibuildwheel --platform=linux --archs auto32

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.18.1

Build options:
  platform: linux
  architectures: i686
  build_selector: 
    build_config: *
    skip_config: *musllinux*
    requires_python: >=3.7
    prerelease_pythons: False
    free_threaded_support: False
  output_dir: /home/PanXuehai/PycharmProjects/optree/wheelhouse
  package_dir: /home/PanXuehai/PycharmProjects/optree
  test_selector: 
    skip_config:
  before_all: 
  before_build: 
  before_test: 
  build_frontend: build
  build_verbosity: 3
  config_settings: 
  container_engine: docker
  dependency_constraints: pinned
  environment: 
  manylinux_images: 
    x86_64: quay.io/pypa/manylinux2014_x86_64:2024-05-13-0983f6f
    i686: quay.io/pypa/manylinux2014_i686:2024-05-13-0983f6f
    pypy_x86_64: quay.io/pypa/manylinux2014_x86_64:2024-05-13-0983f6f
    aarch64: quay.io/pypa/manylinux2014_aarch64:2024-05-13-0983f6f
    ppc64le: quay.io/pypa/manylinux2014_ppc64le:2024-05-13-0983f6f
    s390x: quay.io/pypa/manylinux2014_s390x:2024-05-13-0983f6f
    pypy_aarch64: quay.io/pypa/manylinux2014_aarch64:2024-05-13-0983f6f
    pypy_i686: quay.io/pypa/manylinux2014_i686:2024-05-13-0983f6f
  musllinux_images: 
    x86_64: quay.io/pypa/musllinux_1_2_x86_64:2024-05-13-0983f6f
    i686: quay.io/pypa/musllinux_1_2_i686:2024-05-13-0983f6f
    aarch64: quay.io/pypa/musllinux_1_2_aarch64:2024-05-13-0983f6f
    ppc64le: quay.io/pypa/musllinux_1_2_ppc64le:2024-05-13-0983f6f
    s390x: quay.io/pypa/musllinux_1_2_s390x:2024-05-13-0983f6f
  repair_command: auditwheel repair -w {dest_dir} {wheel}
  test_command: make -C "{project}" test PYTHON=python
  test_extras: [test]
  test_requires:

  ...

Copying project into container...

    + mkdir -p /project

                                                             ✓ 12.89s

Building cp37-manylinux_i686 wheel
CPython 3.7 manylinux i686

Setting up build environment...

    + /opt/python/cp38-cp38/bin/python -c 'import sys, json, os; json.dump(os.environ.copy(), sys.stdout)'
    + which python
    + which pip

                                                              ✓ 0.39s
Building wheel...

...

adding 'optree/_C.cpython-312-x86_64-linux-gnu.so'  # <==== copied from the host directory
adding 'optree/_C.cpython-37m-i386-linux-gnu.so'    # <==== built inside the container
adding 'optree/_C.pyi'
adding 'optree/__init__.py'
adding 'optree/accessor.py'
adding 'optree/functools.py'
adding 'optree/ops.py'
adding 'optree/py.typed'
adding 'optree/registry.py'
adding 'optree/typing.py'
adding 'optree/utils.py'
adding 'optree/version.py'
adding 'optree/integration/__init__.py'
adding 'optree/integration/jax.py'
adding 'optree/integration/numpy.py'
adding 'optree/integration/torch.py'
adding 'optree-0.11.1.dev14+g6d7c718.dist-info/LICENSE'
adding 'optree-0.11.1.dev14+g6d7c718.dist-info/METADATA'
adding 'optree-0.11.1.dev14+g6d7c718.dist-info/WHEEL'
adding 'optree-0.11.1.dev14+g6d7c718.dist-info/top_level.txt'
adding 'optree-0.11.1.dev14+g6d7c718.dist-info/RECORD'
removing build/bdist.linux-i686/wheel
/tmp/build-env-d5mo26qt/lib/python3.7/site-packages/setuptools/config/pyprojecttoml.py:66: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
  config = read_configuration(filepath, True, ignore_option_errors, dist)
Successfully built optree-0.11.1.dev14+g6d7c718-cp37-cp37m-linux_i686.whl
    + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
    + rm -rf /tmp/cibuildwheel/repaired_wheel
    + mkdir -p /tmp/cibuildwheel/repaired_wheel

                                                             ✓ 29.50s
Repairing wheel...

    + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/optree-0.11.1.dev14+g6d7c718-cp37-cp37m-linux_i686.whl'
INFO:auditwheel.main_repair:Repairing optree-0.11.1.dev14+g6d7c718-cp37-cp37m-linux_i686.whl
usage: auditwheel [-h] [-V] [-v] command ...
auditwheel: error: cannot repair "/tmp/cibuildwheel/built_wheel/optree-0.11.1.dev14+g6d7c718-cp37-cp37m-linux_i686.whl" to "manylinux2014_i686" ABI because of the presence of too-recent versioned symbols. You'll need to compile the wheel on an older toolchain.

                                                              ✕ 2.62s
Error: Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/optree-0.11.1.dev14+g6d7c718-cp37-cp37m-linux_i686.whl'] failed with code 2. 

Checking for common errors...

NOTE: Shared object (.so) files found in this project.

These files might be built against the wrong OS, causing problems with
auditwheel. If possible, run cibuildwheel in a clean checkout.

If you're using Cython and have previously done an in-place build,
remove those build files (*.so and *.c) before starting cibuildwheel.

setuptools uses the build/ folder to store its build cache. It
may be necessary to remove those build files (*.so and *.o) before
starting cibuildwheel.

Files that belong to a virtual environment are probably not an issue
unless you used a custom command telling cibuildwheel to activate it.

To be more clarify:

adding 'optree/_C.cpython-312-x86_64-linux-gnu.so'  # <==== copied from the host directory
adding 'optree/_C.cpython-37m-i386-linux-gnu.so'    # <==== built inside the container

The copy command does not respect the .dockerignore file and copies the unexpected 64-bit shared library into the container. That raise an error while auditing the 32-bit wheels.

Copying project into container...

    + mkdir -p /project

Related source code ‎OCIContainer.copy_into‎():

https://github.com/pypa/cibuildwheel/blob/130fdd2548a297d8118d7d66efaf81087c38aabe/cibuildwheel/oci_container.py#L223-L262

Build log

No response

CI config

No response

XuehaiPan avatar Jun 08 '24 13:06 XuehaiPan

.dockerignore is a file used by docker for docker build context. There's no reason for cibuildwheel to use that file nor should it. If an exclusion list is needed, that would be a feature request but that would change the behavior even more between macOS/windows/pyodide that use the local file system and Linux so not sure it's something we'd want.

mayeut avatar Jul 07 '24 08:07 mayeut