integrate/port/document find_libpython to find libpython for pyInitLibPath automatically

[timotheecour]

it's surprisingly hard to find libpython in a robust way; this does the trick but relies on an external python package find_libpython:

# ubuntu:
$ pip3 install find_libpython
$ python3 -c 'import find_libpython; print(find_libpython.find_libpython())'
/usr/lib/x86_64-linux-gnu/libpython3.8.so.1.0

# osx:
$ python3 -c 'import find_libpython; print(find_libpython.find_libpython())'
/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/Python

# this also works on osx directly via shell, but somehow not on ubuntu (probably because I installed it inside a miniconda environment):
find_libpython
/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/Python

$ file /usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/Python
/usr/local/Cellar/[email protected]/3.8.5/Frameworks/Python.framework/Versions/3.8/Python: Mach-O 64-bit dynamically linked shared library x86_64

links

• the main code is this file: https://github.com/ktbarrett/find_libpython/blob/master/src/find_libpython/init.py#L284
• findPythonDLL + friends in nimpy could be updated based on find_libpython

other

• this snippet extracted from it works on OSX but not ubuntu:

import ctypes.util
import os
import sys

class Dl_info(ctypes.Structure):
    _fields_ = [
        ("dli_fname", ctypes.c_char_p),
        ("dli_fbase", ctypes.c_void_p),
        ("dli_sname", ctypes.c_char_p),
        ("dli_saddr", ctypes.c_void_p),
    ]

def _linked_libpython_unix():
    libdl = ctypes.CDLL(ctypes.util.find_library("dl"))
    libdl.dladdr.argtypes = [ctypes.c_void_p, ctypes.POINTER(Dl_info)]
    libdl.dladdr.restype = ctypes.c_int

    dlinfo = Dl_info()
    retcode = libdl.dladdr(
        ctypes.cast(ctypes.pythonapi.Py_GetVersion, ctypes.c_void_p),
        ctypes.pointer(dlinfo))
    if retcode == 0:  # means error
        return None
    path = os.path.realpath(dlinfo.dli_fname.decode())
    if path == os.path.realpath(sys.executable):
        return None
    return path

print(_linked_libpython_unix())

find_library is described here: https://docs.python.org/2.5/lib/ctypes-finding-shared-libraries.html (https://github.com/python/cpython/blob/master/Lib/ctypes/util.py#L72) and would be useful to port/wrap to nim (more generally)

• before finding find_libpython I had also tried parsing otool -L (ldd on linux) directly on OSX but this doesn't seem robust, but at least avoids calling an external python program:

import std/[osproc,strformat,os,strutils]
proc getPythonDll*(path: string): string =
  when defined(osx):
    let result = execCmdEx(fmt"otool -L {path.quoteShell}")
    if result.exitCode != 0: return ""
    let s = result.output.splitLines
    # dbg s
    if s.len < 2: return ""
    let s2 = s[1].split(maxsplit=2)
    if s2.len < 2: return ""
    return s2[1]

when isMainModule:
  let exe = "python3".findExe
  let s = getPythonDll(exe)
  echo (exe, s)

proposal

• step 1: link to find_libpython in the README
• step 2: port find_libpython to nim; likely based on std/dyncalls etc