Installations from PyPI tarballs fail

[ulupo]

Description

I believe that all installs from tarballs on PyPI currently fail. The reason seems to be that we do not package the CMakeLists.txt file needed by CMake.

Steps/Code to Reproduce

1. Create a new environment in an OS not supported by current wheels. E.g., for giotto-tda-nightly, wheels only support macOS 10.14 or above following updates in the Azure VM images https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops, so one can test this on macOS 10.13.
2. Run pip install giotto-tda-nightly to download the package from PyPI.
3. pip will skip the wheels as no supported wheels will be found, and will attempt to install from tarball.

Expected Results

Package is installed as if it were being installed from source (pip install . from the root of a cloned repo).

Actual Results

On my macOS 10.13 machine and running pip install giotto-tda-nightly: https://gist.github.com/ulupo/8bdf6f97a55cf86688a62e9728348409

[rth]

There seem to be another issue with nightly wheels: I get a segfault when running tests in continuumio/miniconda3 container,

# needed to allow running GDB inside docker https://stackoverflow.com/a/46676907/1791279
$  docker run --rm -it --cap-add=SYS_PTRACE --security-opt seccomp=unconfined continuumio/miniconda3 /bin/bash

apt update
apt install gdb

conda create -n gtda-env python=3.7
conda activate gtda-env 
pip install giotto-tda-nightly

pip install hypothesis pytest

For instance,

$ python -c "import gtda.externals.python" 
Illegal instruction (core dumped)

and with a full backtrace,

$ gdb run --args python -c "import gtda.externals.python"
[...]
Reading symbols from python...done.                                                                                                    
(gdb) r                                                                                                                                
Starting program: /opt/conda/envs/gtda-env/bin/python -c import\ gtda.externals.python                                                 
[Thread debugging using libthread_db enabled]                                                                                          
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".                                                             
[New Thread 0x7ffff5178700 (LWP 596)]                                                                                                  
[New Thread 0x7ffff4977700 (LWP 597)]                                                                                                  
...                                                                                               
                                                                                                                                       
Thread 1 "python" received signal SIGILL, Illegal instruction. 
0x00007fffad9188b0 in ?? () from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so    
(gdb) bt
#0  0x00007fffad9188b0 in ?? ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#1  0x00007fffad92bac0 in ?? ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#2  0x00007fffad92ce7e in PyInit_giotto_ripser ()
   from /opt/conda/envs/gtda-env/lib/python3.7/site-packages/giotto_ripser.cpython-37m-x86_64-linux-gnu.so
#3  0x0000555555773695 in _PyImport_LoadDynamicModuleWithSpec (spec=0x7fffae9a7e10, fp=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/importdl.c:159
#4  0x00005555557738a9 in _imp_create_dynamic_impl.isra.15 (file=0x0, spec=0x7fffae9a7e10)
    at /tmp/build/80754af9/python_1578510683607/work/Python/import.c:2170
#5  _imp_create_dynamic (module=<optimized out>, args=<optimized out>, nargs=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/clinic/import.c.h:289
#6  0x000055555568cab2 in _PyMethodDef_RawFastCallDict (method=0x55555586f800 <imp_methods+320>, self=0x7ffff7a32cb0,
    args=0x7ffff78c31a8, nargs=1, kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:530
#7  0x000055555568cbd1 in _PyCFunction_FastCallDict (func=0x7ffff7a4b7d0, args=<optimized out>, nargs=<optimized out>,
    kwargs=<optimized out>) at /tmp/build/80754af9/python_1578510683607/work/Objects/call.c:586
#8  0x000055555571a839 in do_call_core (kwdict=0x7fffb8e58c80, callargs=0x7ffff78c3190, func=0x7ffff7a4b7d0)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:4641
#9  _PyEval_EvalFrameDefault (f=<optimized out>, throwflag=<optimized out>)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3191
#10 0x000055555566e6f9 in _PyEval_EvalCodeWithName (_co=0x7ffff7a3a150, globals=<optimized out>, locals=<optimized out>,
    args=<optimized out>, argcount=<optimized out>, kwnames=0x0, kwargs=0x7ffff7850ea0, kwcount=0, kwstep=1, defs=0x0, defcount=0,
    kwdefs=0x0, closure=0x0, name=0x7ffff7a38300, qualname=0x7ffff7a38300)
    at /tmp/build/80754af9/python_1578510683607/work/Python/ceval.c:3930

which is a Ubuntu based system,

$ uname -a
Linux d614d763e5fb 5.0.0-38-generic #41-Ubuntu SMP Tue Dec 3 00:27:35 UTC 2019 x86_64 GNU/Linux

Not much ideas what could be wrong so far.

[ulupo]

@rth, thanks for the report!

[rth]

For the error with .tar.gz I can confirm I get the same error on Linux with,

pip install https://files.pythonhosted.org/packages/be/b3/cbb2d3bf866f8f1cf820dbe32bfc69cf5bdb0265c26e875670f2bd20be3f/giotto-tda-nightly-20200214.9.tar.gz

which yields,

  fatal: not a git repository (or any of the parent directories): .git                                                       [185/1905]
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 178, in <module>
      cmdclass=dict(build_ext=CMakeBuild))
    File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
      return distutils.core.setup(**attrs)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/gtda-env/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 223, in run
      self.run_command('build')
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/opt/conda/envs/gtda-env/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 94, in run
      self.install_dependencies()
    File "/tmp/pip-req-build-3_74n4r_/setup.py", line 129, in install_dependencies
      '--init', '--recursive'])
    File "/opt/conda/envs/gtda-env/lib/python3.7/subprocess.py", line 363, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.

The issue is in this line https://github.com/giotto-ai/giotto-tda/blob/6b0411bb281303b96fc38711fd73d368efeea829/setup.py#L119 which assumes that gtda is a git repository, which is not the case in a .tar.gz.

[ulupo]

@rth thanks for diagnosing this and for opening #314! It seems that in the current approach we should simply not publish the tarballs at all as they are guaranteed to fail.

[madarax64]

Hello, So is there a workaround for this issue at the moment? For me, install via pip(3) fails with the same error regarding git, and when I try to install it directly as a wheel by saying pip3 install https://github.com/giotto-ai/giotto-tda/releases/download/v0.1.4/giotto_tda-0.1.4-cp36-cp36m-manylinux2010_x86_64.whl I get an error saying that xxxxx is not a supported wheel on this platform. My Python version is 3.6.8 and my pip3 version is 9.0.1

[ulupo]

Thanks for the report @madarax64! The message seems to suggest you are running on a potentially exotic (for giotto-tda at least) Linux version. Could you tell us what that is? Notice that the meaning of manylinux2010 is explained in https://www.python.org/dev/peps/pep-0571/.

Finally, what you mean by "fails with the same error regarding git"? Which error exactly?

[madarax64]

Hey @ulupo Weird - I'm on Ubuntu 18.04.

This one: subprocess.CalledProcessError: Command '['git', 'submodule', 'update', '--init', '--recursive']' returned non-zero exit status 128.

[ulupo]

Ubuntu 18.04 should definitely be supported by the manylinux wheels. So just to be clear, what does a run of

pip install giotto-tda

on a fresh environment, not in the repository root, yield?

[madarax64]

So I upgraded my pip3, and then simply did: pip3 install giotto-tda And that worked off the bat. But when I actually run code, I get an error reading: Illegal Instruction (core dumped)

Running with gdb like above gives the following error: Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so

[ulupo]

But when I actually run code, I get an error reading: Illegal Instruction (core dumped)

Could you give us the code you ran, to reproduce?

[ulupo]

Running with gdb like above gives the following error: Thread 1 "python3" received signal SIGILL, Illegal instruction. 0x00007fffd3b24870 in ?? () from /usr/local/lib/python3.6/dist-packages/giotto_ripser.cpython-36m-x86_64-linux-gnu.so

So this is installing from a clone of the repository? By wich I mean

git clone https://github.com/giotto-ai/giotto-tda
pip install -e .

The error looks like an esoteric C++-related error, which @MonkeyBreaker may be in a better position to diagnose.

[madarax64]

I'm not working in Anaconda or a Virtualenv, so this is directly on my machine. Running pip3 install giotto-tda in say, an empty folder works perfectly without any errors, after I upgraded my pip3 to version 20.

To reproduce the error, you can run the following: python3 -c "import gtda.homology as hl"

No, I never installed from the clone of the repository. This came from the version I got from pip after I upgraded pip.

[MonkeyBreaker]

Hi @madarax64,

Sorry that you encounter Illegal Instruction (core dumped), this was fixed yesterday in PR #360. This was caused by illegal instructions running on the CPU, in this case some AVX instructions that your CPU does not support.

I don't know when will the wheels integrate this change, but I think that giotto-tda-nightly should now integrate them, or am I wrong @ulupo ?

[ulupo]

@MonkeyBreaker @madarax64, I don't think a nightly with these changes has been pushed yet. But the new stable version 0.2.0 which will include these fixes will be out today or Monday.

[madarax64]

Hi @ulupo @MonkeyBreaker Thanks, I look forward to it. I really appreciate it.