Building S2 against anaconda Python in custom directory

[peterroelants]

Hi,

Has anyone succeeded in compiling S2 against an Anaconda Python environment?

When I tried I got:

>>> import pywraps2
Fatal Python error: _PyInterpreterState_Get(): no current thread state
Python runtime state: unknown

[1]    99346 abort      python

I created my environment and tried to build S2 using anaconda tools with:

export S2_ENV_DIR="$(realpath ./s2env/)"
mkdir -p $S2_ENV_DIR

conda create -y --prefix $S2_ENV_DIR python=3.8
conda activate $S2_ENV_DIR
conda install -y cmake swig gflags glog gtest make openssl conda-build pkgconfig

git clone https://github.com/google/s2geometry.git
cd s2geometry

export S2GEO_DIR=`pwd`

mkdir -p "${S2GEO_DIR}/build"
cd "${S2GEO_DIR}/build"

export PATH=$PATH:"$S2_ENV_DIR/bin"
cmake \
  -D WITH_GFLAGS=ON \
  -D WITH_GLOG=ON \
  -D CMAKE_INSTALL_PREFIX="${S2_ENV_DIR}" \
  -D PYTHON_EXECUTABLE:FILEPATH="$(which python)" \
  -D OPENSSL_INCLUDE_DIR="${S2_ENV_DIR}/include" \
  -D OPENSSL_SSL_LIBRARY="${S2_ENV_DIR}/lib/libssl.dylib" \
  -D OPENSSL_CRYPTO_LIBRARY="${S2_ENV_DIR}/lib/libcrypto.dylib" \
  ..
make
make install

I'm building in an OSX environment.

[peterroelants]

When trying to build against Python 3.7 I get a segfault

>>> import pywraps2
[1]    40318 segmentation fault  python

[jmr]

All I know about anaconda is it is python + R. Does this do what you want? https://r-spatial.github.io/s2/

[peterroelants]

All I know about anaconda is it is python + R. Does this do what you want? https://r-spatial.github.io/s2/

This seems to be an R-specific package, and not a Python package

[peterroelants]

I tried building with GTest and the full conda OSX build tools with Python 3.7 and also got a segfault:

        Start 100: pywraps2_test
100/100 Test #100: pywraps2_test ..................................***Exception: SegFault  6.41 sec

99% tests passed, 1 tests failed out of 100

Total Test time (real) =  41.72 sec

The following tests FAILED:
	100 - pywraps2_test (SEGFAULT)
Errors while running CTest
make: *** [Makefile:74: test] Error 8

My install script is at: install_env.txt

How would I go about debugging this segfault? Do you have any idea where it might originate?

[peterroelants]

When running the fault-handler: python -Xfaulthandler /Users/proelants/workspace/personal/s2/s2env/lib/python3.7/site-packages/pywraps2.py

Fatal Python error: Segmentation fault

Current thread 0x0000000117bcadc0 (most recent call first):
  File "<frozen importlib._bootstrap>", line 219 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1043 in create_module
  File "<frozen importlib._bootstrap>", line 583 in module_from_spec
  File "<frozen importlib._bootstrap>", line 670 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 967 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 983 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006 in _gcd_import
  File "/Users/proelants/workspace/personal/s2/s2env/lib/python3.7/importlib/__init__.py", line 127 in import_module
  File "/Users/proelants/workspace/personal/s2/s2env/lib/python3.7/site-packages/pywraps2.py", line 14 in swig_import_helper
  File "/Users/proelants/workspace/personal/s2/s2env/lib/python3.7/site-packages/pywraps2.py", line 17 in <module>
[1]    19157 segmentation fault  python -Xfaulthandler

[jmr]

Can you try with #128? I'm not sure -DPYTHON_EXECUTABLE was making it find the right libraries. The new way should have fewer ways to go wrong.

[peterroelants]

I'm trying #128 and I noticed I need to pass in -D Python3_EXECUTABLE:FILEPATH="${PYTHON_DIR}" explicitly otherwise it picks up the system Python again.

However, unfortunately, I'm getting the same segfault on the new branch.

This is from the CMakeCache.txt

//No help, variable specified on the command line.
PYTHON_EXECUTABLE:FILEPATH=/Users/proelants/workspace/personal/s2/s2env/bin/python

//Where can one of the python3.7 or python3 libraries be found
Python3_EXECUTABLE:FILEPATH=/Users/proelants/workspace/personal/s2/s2env/bin/python3.7

//Path to a file.
Python3_INCLUDE_DIR:PATH=/Users/proelants/workspace/personal/s2/s2env/include/python3.7m

//Path to a library.
Python3_LIBRARY_RELEASE:FILEPATH=/Users/proelants/workspace/personal/s2/s2env/lib/libpython3.7m.dylib

See full txt at: CMakeCache.txt

It seem like there is still something going wrong when trying to build against a non-system Python

[jmr]

#128 has been merged, so you can use HEAD now.

Quite possibly, something is just broken on OS X. Does it work with CPython and without messing around with the directories?

[jmr]

Works for me, both on my own OS X machine, and travis-ci with #133 .

https://travis-ci.org/github/google/s2geometry/jobs/722114562#L1464

[peterroelants]

It might be an anaconda issue.

[peterroelants]

I couldn't get the s2geometry library build against anaconda Python to work. However, I succeeded in building against python3 virtual environment.

[jmr]

Try searching for [anaconda swig]; it seems there are lots of problems. Maybe one is the same as yours.

https://astrodatascience.wordpress.com/2018/02/08/swig-tutorial-for-anaconda-python-on-macos/ indicates the standard methods don't work.

You could start with one a SWIG tutorial example and see if that works.

[thegeebe]

I'm not using Anaconda, just virtualenvs, but still had the issue of building against system python. I had success essentially changing the flag I'm passing to cmake to:

-DPython3_EXECUTABLE:FILEPATH=/path/to/venv/bin/python

When I use setuptools to build this as an extension I use sys.executable to resolve the correct location. Might help with #108 as well