uvloop building wheel from sdist fails

building wheel from sdist fails

Open televi opened this issue 4 years ago • 4 comments

uvloop version: 0.14.0
Python version: 3.6 and 3.7
Platform: Linux (CentOS 7
Can you reproduce the bug with PYTHONASYNCIODEBUG in env?: N/A
Does uvloop behave differently from vanilla asyncio? How?: N/A

What I did: pip wheel --global-option build_ext --global-option --inplace uvloop==0.14.0

What I expected:

The command would build uvloop and bundle it as a wheel for me.

What I saw:

The command fails with error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.unixccompiler.UnixCCompiler object at 0x7f054bc1fa10>' compiler after build_ext runs (e.g. during the processing of bdist_wheel).

The (heavily) snipped output from the above command is:

</path/to/venv>/lib/python3.7/site-packages/pip/_internal/commands/wheel.py:117: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Looking in indexes: https://pypi.org/simple, https://<private repo>/simple
Collecting uvloop==0.14.0
  Using cached uvloop-0.14.0.tar.gz (2.0 MB)
Building wheels for collected packages: uvloop
  Building wheel for uvloop (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: </path/to/venv>/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-wheel-12kxwkdh/uvloop/setup.py'"'"'; __file__='"'"'/tmp/pip-wheel-12kxwkdh/uvloop/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' build_ext --inplace bdist_wheel -d /tmp/pip-wheel-exvxqjuv
       cwd: /tmp/pip-wheel-12kxwkdh/uvloop/
  Complete output (226 lines):
  running build_ext

 ... <lots of successful build output

 building 'uvloop.loop' extension

... <more successful built output>

  copying build/lib.linux-x86_64-3.7/uvloop/loop.cpython-37m-x86_64-linux-gnu.so -> uvloop
  running bdist_wheel
  running build
  running build_py

... <lots of copying/writing for building the wheel>

  copying uvloop/includes/uv.pxd -> build/lib.linux-x86_64-3.7/uvloop/includes
  running build_ext
  error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.unixccompiler.UnixCCompiler object at 0x7f054bc1fa10>' compiler
  ----------------------------------------
  ERROR: Failed building wheel for uvloop
  Running setup.py clean for uvloop
Failed to build uvloop
ERROR: Failed to build one or more wheels

Other observations:

The pip wheel command essentially downloads your sdist, unwinds it, cd's to that directory, and runs python setup.py build_ext --inplace bdist_wheel This mimics your make file other than the fact that build_ext and bdist_wheel are on one command line.

If I manually run python setup.py build_ext --inplace && python setup.py bdist_wheel, then the wheel is successfully built. I can't tell exactly how, but it seems like something is building the extension in bdist_wheel instead of simply inheriting what was build by build_ext. That, in turn, is triggering what may be a latent distutils issue that I documented here.

Mar 09 '20 22:03 televi

Do you have python3-devel installed?

Mar 17 '20 20:03 elprans

Yes I do. I'm able to reproduce this at will on both CentOS 7.7 and Mac OS X 10.15.3 (Catalina) with Python 3.6, 3.7, and 3.8.

Here's the full set of commands and output that I used on CentOS 7.7 with Python 3.6. Note: I'm able to compile dozens of other Python packages using the same command both with and without Cython.

In the output below, you'll see that it compiles fine, then tries to run bdist_wheel (search for "running bdist_wheel") and then tries to compile again and that fails. Note the error at the end. That should not point at a UnixCCompiler object - it should be a string. That string gets used as a key to a dict and, in the case of running on CentOS, should be "unix". If it were, this would work.

What seems to be happening is that (somehow) your setup.py file is passing a compiler object back to a method to create a new compiler object. I've not seen any other Cython package we build using this command do that so I don't know why that is happening.

$ yum list installed | grep python3
python3.x86_64                             3.6.8-10.el7                @base
python3-devel.x86_64                       3.6.8-10.el7                @base
python3-libs.x86_64                        3.6.8-10.el7                @base
python3-rpm-generators.noarch              6-2.el7                     @base
python3-rpm-macros.noarch                  3-32.el7                    @base
python3-setuptools.noarch                  39.2.0-10.el7               @base
python37.x86_64                            3.7.3-1                     @NFVFP-rpm-local
$ ls -a
.  ..
$ python3.6 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip install wheel
Collecting wheel
  Using cached https://files.pythonhosted.org/packages/8c/23/848298cccf8e40f5bbb59009b32848a4c38f4e7f3364297ab3c3e2e2cd14/wheel-0.34.2-py2.py3-none-any.whl
Installing collected packages: wheel
Successfully installed wheel-0.34.2
You are using pip version 9.0.3, however version 20.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
(.venv) $ pip wheel --global-option build_ext --global-option --inplace uvloop==0.14.0
/home/tlevi/tmp/.venv/lib64/python3.6/site-packages/pip/commands/wheel.py:127: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
  cmdoptions.check_install_build_global(options)
Collecting uvloop==0.14.0
  Using cached https://files.pythonhosted.org/packages/84/2e/462e7a25b787d2b40cf6c9864a9e702f358349fc9cfb77e83c38acb73048/uvloop-0.14.0.tar.gz
Building wheels for collected packages: uvloop
  Running setup.py bdist_wheel for uvloop ... error
  Complete output from command /home/tlevi/tmp/.venv/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-1sp03mdo/uvloop/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" build_ext --inplace bdist_wheel -d /tmp/tmpre81n5yfpip-wheel-:
  running build_ext
  checking for a BSD-compatible install... /bin/install -c
  checking whether build environment is sane... yes
  checking for a thread-safe mkdir -p... /bin/mkdir -p
  checking for gawk... gawk
  checking whether make sets $(MAKE)... yes
  checking whether make supports nested variables... yes
  checking build system type... x86_64-unknown-linux-gnu
  checking host system type... x86_64-unknown-linux-gnu
  checking for gcc... gcc
  checking whether the C compiler works... yes
  checking for C compiler default output file name... a.out
  checking for suffix of executables...
  checking whether we are cross compiling... no
  checking for suffix of object files... o
  checking whether we are using the GNU C compiler... yes
  checking whether gcc accepts -g... yes
  checking for gcc option to accept ISO C89... none needed
  checking whether gcc understands -c and -o together... yes
  checking for style of include used by make... GNU
  checking dependency style of gcc... gcc3
  checking if gcc supports -pedantic flag... yes
  checking for gcc way to treat warnings as errors... -Werror
  checking if gcc supports -fvisibility=hidden... yes
  checking if gcc supports -g flag... yes
  checking if gcc supports -std=gnu89 flag... yes
  checking if gcc supports -Wall flag... yes
  checking if gcc supports -Wextra flag... yes
  checking if gcc supports -Wno-long-long flag... yes
  checking if gcc supports -Wno-unused-parameter flag... yes
  checking if gcc supports -Wstrict-prototypes flag... yes
  checking for ar... ar
  checking the archiver (ar) interface... ar
  checking how to print strings... printf
  checking for a sed that does not truncate output... /bin/sed
  checking for grep that handles long lines and -e... /bin/grep
  checking for egrep... /bin/grep -E
  checking for fgrep... /bin/grep -F
  checking for ld used by gcc... /bin/ld
  checking if the linker (/bin/ld) is GNU ld... yes
  checking for BSD- or MS-compatible name lister (nm)... /bin/nm -B
  checking the name lister (/bin/nm -B) interface... BSD nm
  checking whether ln -s works... yes
  checking the maximum length of command line arguments... 1572864
  checking whether the shell understands some XSI constructs... yes
  checking whether the shell understands "+="... yes
  checking how to convert x86_64-unknown-linux-gnu file names to x86_64-unknown-linux-gnu format... func_convert_file_noop
  checking how to convert x86_64-unknown-linux-gnu file names to toolchain format... func_convert_file_noop
  checking for /bin/ld option to reload object files... -r
  checking for objdump... objdump
  checking how to recognize dependent libraries... pass_all
  checking for dlltool... no
  checking how to associate runtime and link libraries... printf %s\n
  checking for archiver @FILE support... @
  checking for strip... strip
  checking for ranlib... ranlib
  checking command to parse /bin/nm -B output from gcc object... ok
  checking for sysroot... no
  checking for mt... no
  checking if : is a manifest tool... no
  checking how to run the C preprocessor... gcc -E
  checking for ANSI C header files... yes
  checking for sys/types.h... yes
  checking for sys/stat.h... yes
  checking for stdlib.h... yes
  checking for string.h... yes
  checking for memory.h... yes
  checking for strings.h... yes
  checking for inttypes.h... yes
  checking for stdint.h... yes
  checking for unistd.h... yes
  checking for dlfcn.h... yes
  checking for objdir... .libs
  checking if gcc supports -fno-rtti -fno-exceptions... no
  checking for gcc option to produce PIC... -fPIC -DPIC
  checking if gcc PIC flag -fPIC -DPIC works... yes
  checking if gcc static flag -static works... no
  checking if gcc supports -c -o file.o... yes
  checking if gcc supports -c -o file.o... (cached) yes
  checking whether the gcc linker (/bin/ld -m elf_x86_64) supports shared libraries... yes
  checking whether -lc should be explicitly linked in... no
  checking dynamic linker characteristics... GNU/Linux ld.so
  checking how to hardcode library paths into programs... immediate
  checking whether stripping libraries is possible... yes
  checking if libtool supports shared libraries... yes
  checking whether to build shared libraries... yes
  checking whether to build static libraries... yes
  checking whether make supports nested variables... (cached) yes
  checking for dlopen in -ldl... yes
  checking for kstat_lookup in -lkstat... no
  checking for gethostbyname in -lnsl... yes
  checking for perfstat_cpu in -lperfstat... no
  checking for pthread_mutex_init in -lpthread... yes
  checking for clock_gettime in -lrt... yes
  checking for sendfile in -lsendfile... no
  checking for socket in -lsocket... no
  checking for special C compiler options needed for large files... no
  checking for _FILE_OFFSET_BITS value needed for large files... no
  checking sys/ahafs_evProds.h usability... no
  checking sys/ahafs_evProds.h presence... no
  checking for sys/ahafs_evProds.h... no
  checking that generated files are newer than configure... done
  configure: creating ./config.status
  config.status: creating Makefile
  config.status: creating libuv.pc
  config.status: executing depfiles commands
  config.status: executing libtool commands
    CC       src/libuv_la-fs-poll.lo
    CC       src/libuv_la-idna.lo
    CC       src/libuv_la-random.lo
    CC       src/libuv_la-inet.lo
    CC       src/libuv_la-timer.lo
    CC       src/libuv_la-strscpy.lo
    CC       src/libuv_la-threadpool.lo
    CC       src/libuv_la-version.lo
    CC       src/libuv_la-uv-data-getter-setters.lo
    CC       src/libuv_la-uv-common.lo
    CC       src/unix/libuv_la-fs.lo
    CC       src/unix/libuv_la-async.lo
    CC       src/unix/libuv_la-core.lo
    CC       src/unix/libuv_la-dl.lo
    CC       src/unix/libuv_la-loop.lo
    CC       src/unix/libuv_la-getnameinfo.lo
    CC       src/unix/libuv_la-getaddrinfo.lo
    CC       src/unix/libuv_la-pipe.lo
    CC       src/unix/libuv_la-poll.lo
    CC       src/unix/libuv_la-loop-watcher.lo
    CC       src/unix/libuv_la-process.lo
    CC       src/unix/libuv_la-random-devurandom.lo
    CC       src/unix/libuv_la-signal.lo
    CC       src/unix/libuv_la-stream.lo
    CC       src/unix/libuv_la-tcp.lo
    CC       src/unix/libuv_la-thread.lo
    CC       src/unix/libuv_la-tty.lo
    CC       src/unix/libuv_la-udp.lo
    CC       src/unix/libuv_la-linux-core.lo
    CC       src/unix/libuv_la-linux-inotify.lo
    CC       src/unix/libuv_la-linux-syscalls.lo
    CC       src/unix/libuv_la-procfs-exepath.lo
    CC       src/unix/libuv_la-proctitle.lo
    CC       src/unix/libuv_la-random-getrandom.lo
    CC       src/unix/libuv_la-random-sysctl.lo
    CC       src/unix/libuv_la-sysinfo-loadavg.lo
    CCLD     libuv.la
  building 'uvloop.loop' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/uvloop
  gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -I/home/tlevi/tmp/.venv/include -I/usr/include/python3.6m -I/tmp/pip-build-1sp03mdo/uvloop/vendor/libuv/include -c uvloop/loop.c -o build/temp.linux-x86_64-3.6/uvloop/loop.o -O2
  uvloop/loop.c: In function ‘__pyx_f_6uvloop_4loop_4Loop__is_main_thread’:
  uvloop/loop.c:10041:75: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
     __pyx_t_1 = __Pyx_PyBool_FromLong((__pyx_v_6uvloop_4loop_MAIN_THREAD_ID == PyThread_get_thread_ident())); if (unlikely(!__pyx_t_1)) __PYX_ERR(2, 188, __pyx_L1_error)
                                                                             ^
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/uvloop
  gcc -pthread -shared -Wl,-z,relro -g build/temp.linux-x86_64-3.6/uvloop/loop.o /tmp/pip-build-1sp03mdo/uvloop/build/libuv/.libs/libuv.a -L/usr/lib64 -lpython3.6m -lrt -lpthread -o build/lib.linux-x86_64-3.6/uvloop/loop.cpython-36m-x86_64-linux-gnu.so
  copying build/lib.linux-x86_64-3.6/uvloop/loop.cpython-36m-x86_64-linux-gnu.so -> uvloop
  running bdist_wheel
  running build
  running build_py
  copying uvloop/__init__.py -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/_patch.py -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/_testbase.py -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/_noop.py -> build/lib.linux-x86_64-3.6/uvloop
  running egg_info
  writing uvloop.egg-info/PKG-INFO
  writing dependency_links to uvloop.egg-info/dependency_links.txt
  writing top-level names to uvloop.egg-info/top_level.txt
  reading manifest file 'uvloop.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no previously-included files matching '*' found under directory 'vendor/libuv/.git'
  warning: no previously-included files matching '*' found under directory 'vendor/libuv/docs'
  warning: no previously-included files matching '*' found under directory 'vendor/libuv/img'
  writing manifest file 'uvloop.egg-info/SOURCES.txt'
  copying uvloop/cbhandles.pxd -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/cbhandles.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/dns.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/errors.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/loop.c -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/loop.pxd -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/loop.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/lru.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/pseudosock.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/request.pxd -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/request.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/server.pxd -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/server.pyx -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/sslproto.pxd -> build/lib.linux-x86_64-3.6/uvloop
  copying uvloop/sslproto.pyx -> build/lib.linux-x86_64-3.6/uvloop
  creating build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/async_.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/async_.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/basetransport.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/basetransport.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/check.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/check.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/handle.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/handle.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/idle.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/idle.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/pipe.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/pipe.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/poll.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/poll.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/process.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/process.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/stream.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/stream.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/streamserver.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/streamserver.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/tcp.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/tcp.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/timer.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/timer.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/udp.pxd -> build/lib.linux-x86_64-3.6/uvloop/handles
  copying uvloop/handles/udp.pyx -> build/lib.linux-x86_64-3.6/uvloop/handles
  creating build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/__init__.py -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/compat.h -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/consts.pxi -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/debug.h -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/debug.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/flowcontrol.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/fork_handler.h -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/python.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/stdlib.pxi -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/system.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes
  copying uvloop/includes/uv.pxd -> build/lib.linux-x86_64-3.6/uvloop/includes
  running build_ext
  error: don't know how to compile C/C++ code on platform 'posix' with '<distutils.unixccompiler.UnixCCompiler object at 0x7f9874fcce48>' compiler

  ----------------------------------------
  Failed building wheel for uvloop
  Running setup.py clean for uvloop
Failed to build uvloop
ERROR: Failed to build one or more wheels

Mar 18 '20 21:03 televi

I think I may have found it.

In your setup.py, you're short-circuiting the call to super().initialize_options() here. That means you're forcing subsequent commands to inherit variables that they expect to set for themselves (like self.compiler, which is what is causing the build failure). The same applies to finalize_options().

If you simply want to avoid your options from being reset, you could move the 4 lines that set your options into the body of that if and always call super()initialize_options().

I'm not sure why you're avoiding those calls to super().foo() in initialize_options() and finalize_options(), but those seem to be what is messing up builds with multiple commands on the same line.

Mar 18 '20 21:03 televi

Hm. Good catch. I'll take a look and see if this hack is still necessary.

Mar 19 '20 00:03 elprans

I know this is old, but it still fails. Any luck deciding on whether that hack was still required? As it stands, uvloop still can't be built from sdist 😦

Aug 19 '22 17:08 televi

I think we can drop this hack now, as test_suite is already dropped in setup.py, refs #84

Sep 09 '22 15:09 fantix

uvloop uvloop copied to clipboard

building wheel from sdist fails

uvloop
uvloop copied to clipboard