torch-mlir icon indicating copy to clipboard operation
torch-mlir copied to clipboard

Empty `torch_mlir._mlir_libs._jit_ir_importer` lib causes package build problem on macOS

Open dbabokin opened this issue 1 year ago • 0 comments

There's something strange about how torch_mlir._mlir_libs._jit_ir_importer is defined in the build system.

When building Python packages, jit_ir_importer is declared as an extension: https://github.com/llvm/torch-mlir/blob/9a6fe58a027d701eff6799e86a65535a8c2f3708/setup.py#L238-L242

It's declared without a build recipe or source files, so it's built as an empty library. To replicate that, you can run python setup.py build_ext (which is a part of package build process typically invoked by python setup.py bdist_wheel). If you run it with -vv switch, you'll see the build command that builds the library (this is on Linux):

> python setup.py build_ext -vv
running build_ext
building 'torch_mlir._mlir_libs._jit_ir_importer' extension
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 -L/usr/lib/x86_64-linux-gnu -o setup_build/lib.linux-x86_64-cpython-311/torch_mlir/_mlir_libs/_jit_ir_importer.cpython-311-x86_64-linux-gnu.so

Look closely at gcc invocation. It has output library specified, but doesn't have any inputs specified!!! So the result is empty lib with just boilerplate code. And gcc does not complain, just silently produces empty lib.

But when building on macOS, clang is used instead of gcc and it does complain about no inputs provided:

> python setup.py build_ext -vv
running build_ext
clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -L/opt/homebrew/opt/libomp/lib -I/opt/homebrew/opt/libomp/include -o setup_build/lib.macosx-14.0-arm64-cpython-312/torch_mlir/_mlir_libs/_jit_ir_importer.cpython-312-darwin.so
clang: error: no input files
error: command '/usr/bin/clang' failed with exit code 1

And yes, clang is absolutely right, no inputs are provided.

But it's not the end of the story on macOS. With some Python distros it's reproducible and with some not. As it appears, the reason is this patch https://github.com/pypa/setuptools/commit/30b7331b07fbc404959cb37ac311afdfb90813be which already propagated to 3.12 setuptools, but looks like not to 3.11 yet (not sure how this really works). Before the patch setuptools were skipping the build of torch_mlir._mlir_libs._jit_ir_importer as the library has no source, but after the patch if the library is missing, it's being built even if it doesn't have sources. And that breaks macOS build.

dbabokin avatar Aug 23 '24 21:08 dbabokin