build for python 3.6 in docker using amazonlinux

[dazza-codes]

From https://github.com/dazza-codes/docker-google-s2-lambda

The build works, but it has cmake/swig WARNINGs and the python import fails. The Dockerfile uses FROM amazonlinux and it installs the usual development libs to download and build/install python 3.6.10 (into /usr/local/bin/python3) and then the cmake3 uses it to configure the python paths. The gtest package is installed from yum. The build goes like:

Step 10/11 : ENV PY3 /usr/local/bin/python3
 ---> Using cache
 ---> ec80418cbfed
Step 11/11 : RUN cmake3         -DCMAKE_INSTALL_PREFIX:PATH=/usr         -DPYTHON_INCLUDE_DIR=$(${PY3} -c "from distutils.sysconfig import get_python_inc; print(get_python_inc())")          -DPYTHON_LIBRARY=$(${PY3} -c "import distutils.sysconfig as sysconfig; print(sysconfig.get_config_var('LIBDIR'))")         -DPYTHON_EXECUTABLE:FILEPATH=${PY3} ..     && make     && make install
 ---> Running in ccb0560057f6
-- The C compiler identification is GNU 7.3.1
-- The CXX compiler identification is GNU 7.3.1
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The following features have been enabled:

 * SHARED_LIBS, builds shared libraries instead of static.

-- The following features have been disabled:

 * GFLAGS, allows changing command line flags.
 * GLOG, provides logging configurability.

-- Found OpenSSL: /usr/lib64/libcrypto.so (found version "1.0.2k")  
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found SWIG: /usr/bin/swig (found version "3.0.12") 
-- Found PythonInterp: /usr/local/bin/python3 (found version "3.6.10") 
GTEST_ROOT: 
-- Found PythonLibs: /usr/local/lib (found version "3.6.10") 
CMake Warning (dev) at /usr/share/cmake3/Modules/UseSWIG.cmake:564 (message):
  Policy CMP0078 is not set.  Run "cmake --help-policy CMP0078" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.
Call Stack (most recent call first):
  src/python/CMakeLists.txt:12 (swig_add_library)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring done
WARNING: Target "_pywraps2" requests linking to directory "/usr/local/lib".  Targets may link only to libraries.  CMake is dropping the item.
-- Generating done
-- Build files have been written to: /app/s2geometry/build

SO the worry is

WARNING: Target "_pywraps2" requests linking to directory "/usr/local/lib".  Targets may link only to libraries.  CMake is dropping the item.

The rest of the build seems OK, e.g.

Scanning dependencies of target point_index
[ 97%] Building CXX object examples/CMakeFiles/point_index.dir/point_index.cc.o
[ 98%] Linking CXX executable point_index
[ 98%] Built target point_index
Scanning dependencies of target pywraps2_swig_compilation
[ 99%] Swig compile s2.i for python
/app/s2geometry/src/s2/s2region.h:45: Warning 362: operator= ignored
/app/s2geometry/src/s2/s2cell_union.h:73: Warning 509: Overloaded method S2CellUnion::S2CellUnion(std::vector< uint64,std::allocator< uint64 > > const &) effectively ignored,
/app/s2geometry/src/s2/s2cell_union.h:69: Warning 509: as it is shadowed by S2CellUnion::S2CellUnion(std::vector< S2CellId,std::allocator< S2CellId > >).
[ 99%] Built target pywraps2_swig_compilation
Scanning dependencies of target _pywraps2
[ 99%] Building CXX object python/CMakeFiles/_pywraps2.dir/CMakeFiles/_pywraps2.dir/s2PYTHON_wrap.cxx.o
[100%] Linking CXX shared module _pywraps2.so
[100%] Built target _pywraps2
[  3%] Built target s2testing
[ 94%] Built target s2
[ 96%] Built target term_index
[ 98%] Built target point_index
[ 99%] Built target pywraps2_swig_compilation
[100%] Built target _pywraps2

But it's not really. i.e. the python modules are messed up:

$ docker run --rm -it google-s2

bash-4.2# python3
Python 3.6.10 (default, Mar 13 2020, 01:24:15) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pywraps2 as s2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pywraps2'


bash-4.2# python3
Python 3.6.10 (default, Mar 13 2020, 01:24:15) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-6)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/usr/lib/python3.6/site-packages')
>>> import pywraps2 as s2
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pywraps2.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: libs2.so: cannot open shared object file: No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/site-packages/pywraps2.py", line 17, in <module>
    _pywraps2 = swig_import_helper()
  File "/usr/lib/python3.6/site-packages/pywraps2.py", line 16, in swig_import_helper
    return importlib.import_module('_pywraps2')
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libs2.so: cannot open shared object file: No such file or directory
>>> 
bash-4.2# exit

[jmr]

Can you try with #128?

[jmr]

#128 has been merged, you can try at HEAD.

[dazza-codes]

The build code is in https://github.com/dazza-codes/docker-google-s2-lambda and update that repo with any build changes to see what works or not. It might be working, using a base image from lambci/lambda:build-python${python_ver}

There could be a cmake config error

-- Configuring done
CMake Warning at /opt/cmake/share/cmake-3.19/Modules/UseSWIG.cmake:783 (add_library):
  Cannot generate a safe runtime search path for target _pywraps2 because
  files in some directories may conflict with libraries in implicit
  directories:

    runtime library [libssl.so.10] in /usr/lib64 may be hidden by files in:
      /var/lang/lib
    runtime library [libcrypto.so.10] in /usr/lib64 may be hidden by files in:
      /var/lang/lib

  Some of these libraries may not be found correctly.
Call Stack (most recent call first):
  src/python/CMakeLists.txt:8 (swig_add_library)

Trying to explicitly set some (old) cmake variables are ignored

-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON_EXECUTABLE
    PYTHON_INCLUDE_DIR
    PYTHON_LIBRARY

The lambci build image has ENV:

{'AWS_ACCESS_KEY_ID': '',
 'AWS_DEFAULT_REGION': 'us-east-1',
 'AWS_EXECUTION_ENV': 'AWS_Lambda_python3.7',
 'AWS_LAMBDA_FUNCTION_MEMORY_SIZE': '1536',
 'AWS_LAMBDA_FUNCTION_NAME': 'test',
 'AWS_LAMBDA_FUNCTION_VERSION': '$LATEST',
 'AWS_LAMBDA_LOG_GROUP_NAME': '/aws/lambda/test',
 'AWS_LAMBDA_LOG_STREAM_NAME': '2020/12/12/[$LATEST]b8e1190315f3fde1bc8eb1eaee6bc2de',
 'AWS_LAMBDA_RUNTIME_API': '127.0.0.1:9001',
 'AWS_REGION': 'us-east-1',
 'AWS_SECRET_ACCESS_KEY': '',
 'AWS_XRAY_CONTEXT_MISSING': 'LOG_ERROR',
 'AWS_XRAY_DAEMON_ADDRESS': '169.254.79.2:2000',
 'HOME': '/home/sbx_user1051',
 'HOSTNAME': 'd020b9dfd2ab',
 'LAMBDA_RUNTIME_DIR': '/var/runtime',
 'LAMBDA_TASK_ROOT': '/var/task',
 'LANG': 'en_US.UTF-8',
 'LD_LIBRARY_PATH': '/var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib',
 'PATH': '/var/lang/bin:/usr/local/bin:/usr/bin/:/bin:/opt/bin',
 'TZ': ':UTC',
 '_AWS_XRAY_DAEMON_ADDRESS': '169.254.79.2',
 '_AWS_XRAY_DAEMON_PORT': '2000',
 '_HANDLER': 'lambda_versions.lambda_handler',
 '_LAMBDA_CONSOLE_SOCKET': '16',
 '_LAMBDA_CONTROL_SOCKET': '14',
 '_LAMBDA_LOG_FD': '9',
 '_LAMBDA_RUNTIME_LOAD_TIME': '1530232235231',
 '_LAMBDA_SB_ID': '7',
 '_LAMBDA_SHARED_MEM_FD': '11',
 '_X_AMZN_TRACE_ID': 'Root=1-dc99d00f-c079a84d433534434534ef0d;Parent=91ed514f1e5c03b2;Sampled=1'}

[dazza-codes]

2c opinion - the swig wrappers do not produce a pythonic API and it would be awesome if some wrappers on the swig wrappers might provide a API something like https://github.com/sidewalklabs/s2sphere

• https://github.com/sidewalklabs/s2sphere/blob/master/s2sphere/sphere.py

If the license details are compatible, it might be possible to copy the sphere.py module and adapt it to encapsulate the swig wrappers. I understand that could introduce extra maintenance that this project would want to avoid. Clearly the s2sphere project is not getting updates, bug fixes and routine maintenance.