Symbol not found: _libintl_setlocale on macOS

[geekadalovelace]

Describe the bug On macOS 11.2(Big Sur), raise error below:

dyld: lazy symbol binding failed: Symbol not found: _libintl_setlocale
  Referenced from:.../.pyenv/versions/3.7.9/lib/python3.7/site-packages/jep/libjep.jnilib
  Expected in: flat namespace

[1]    38735 abort      jep

To Reproduce

$ pip install jep
$ pip install pororo
$ jep
>>> from pororo import Pororo

Expected behavior next prompt to input python scripts.

Environment (please complete the following information):

• OS Platform, Distribution, and Version: macOS 11.2(Big Sur)
• Python Distribution and Version: 3.7.9
• Java Distribution and Version: 1.8
• Jep Version: 3.9.1
• Python packages used (e.g. numpy, pandas, tensorflow): pororo

Additional context This error may not be your problem. I think libintl library has any problem. But please, let me know if you know any solution for the problem. Thank you.

[alexander-myltsev]

I've got the same error.

[ndjensen]

How did you install Python? Was it with conda or homebrew or from the python website or something else?

[alexander-myltsev]

How did you install Python? Was it with conda or homebrew or from the python website or something else?

I installed Python with Pyenv to the project directory. I used environment variables as follows:

export PYTHON_VERSION=`cat .python-version`
export POETRY_VIRTUALENVS_CREATE=false
export PYENV_ROOT=`pwd`/.pyenv
export PYTHONHOME=$PYENV_ROOT/versions/$PYTHON_VERSION/
export PATH=$PYENV_ROOT/shims:$PYTHONHOME/bin:$PATH

My Poetry dependencies are as follows:

[tool.poetry.dependencies]
python = "^3.9"
plac = "^1.3"
pandas = "^1.3"
selenium = "^3.141"
requests = "^2.25"
xmltodict = "^0.12"
pandas_datareader = "^0.9"
beautifulsoup4 = "^4.9"
quandl = "^3.6"
pyyaml = "^5.4"
xlrd = "^2.0"
html5lib = "^1.1"

[ndjensen]

I'm not familiar with Pyenv but it looks like @geekadalovelace was also using it. Please doublecheck that the Python that built Jep is the same Python you're running with. I don't have any other ideas.

[bsteffensmeier]

It looks like pyenv has experieince with problems in libintl, I didn't read the entire ticket to see if it is related but it may have some answers: https://github.com/pyenv/pyenv/issues/1877

[alexander-myltsev]

Also, if I use Python created by Poetry in the context of poetry shell, jep fails with the error:

Python path configuration:
  PYTHONHOME = '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Users/auser/.jenv/versions/oracle64-11.0.8/bin/java'
  sys.base_prefix = '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9'
  sys.base_exec_prefix = '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/auser/.jenv/versions/oracle64-11.0.8/bin/java'
  sys.prefix = '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9'
  sys.exec_prefix = '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9'
  sys.path = [
    '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9/lib/python39.zip',
    '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9/lib/python3.9',
    '/Users/auser/Library/Caches/pypoetry/virtualenvs/my-project-7avxlrD6-py3.9/lib/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000070000a36a000 (most recent call first):
<no Python frame>

What does the error mean? Any ideas how to fix this?

[alexander-myltsev]

I tried in a fresh environment in Docker:

FROM python:3.9-buster

ENV DEBIAN_FRONTEND noninteractive
ENV LANG=en_US.UTF-8

WORKDIR /app

RUN apt-get update && \
  apt-get upgrade -y
RUN apt-get install -q -y openjdk-11-jdk 

ENV JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64

RUN pip install --upgrade pip
RUN pip install poetry==1.1.7

COPY .python-version pyproject.toml poetry.lock /app/
RUN poetry install

Everything works as expected:

root@d29e4894ee74:/app# poetry shell
Spawning shell within /root/.cache/pypoetry/virtualenvs/project-8TtSrW0h-py3.9
root@d29e4894ee74:/app# . /root/.cache/pypoetry/virtualenvs/project-8TtSrW0h-py3.9/bin/activate
(project-8TtSrW0h-py3.9) root@d29e4894ee74:/app# jep
>>> import pandas as pd
>>>

While on the host system it fails:

$ jep                                                                                                                130 ↵
>>> import pandas as pd
dyld: lazy symbol binding failed: Symbol not found: _libintl_setlocale
  Referenced from: /Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/site-packages/jep/jep.cpython-39-darwin.so
  Expected in: flat namespace

dyld: Symbol not found: _libintl_setlocale
  Referenced from: /Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/site-packages/jep/jep.cpython-39-darwin.so
  Expected in: flat namespace

[1]    9441 abort      jep

---

I tried to build/install from jep sources. Same result. Another weirdness is with testing. When I run python setup.py test for the first time, it fails with:

$ python setup.py test
numpy include found at /Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/site-packages/numpy/core/include
running test
running build
running setup_java
running build_java
running build_jar
running build_py
running build_ext
running build_scripts
ln -sf jep.cpython-39-darwin.so build/lib.macosx-11.4-x86_64-3.9/jep/libjep.jnilib
.........................................................F................s........................................................ssssssssssss.......F.............................................................
======================================================================
FAIL: test_get_jpyobject (test_getvalue.TestGetValue)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/auser/project/jep-src/src/test/python/test_getvalue.py", line 23, in test_get_jpyobject
    jep_pipe(build_java_process_cmd('jep.test.TestGetJPyObject'))
  File "/Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/contextlib.py", line 244, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/contextlib.py", line 87, in __init__
    self.gen = func(*args, **kwds)
  File "/Users/auser/project/jep-src/src/test/python/jep_pipe.py", line 36, in jep_pipe
    assert False, stderr
AssertionError: b'sys:1: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode.\nImprovements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve.\n'

======================================================================
FAIL: test_numpy_prod_succeeds (test_shared_modules.TestSharedModules)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/auser/project/jep-src/src/test/python/test_shared_modules.py", line 19, in test_numpy_prod_succeeds
    jep_pipe(build_java_process_cmd('jep.test.numpy.TestNumpyProdShared'))
  File "/Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/contextlib.py", line 244, in helper
    return _GeneratorContextManager(func, args, kwds)
  File "/Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/contextlib.py", line 87, in __init__
    self.gen = func(*args, **kwds)
  File "/Users/auser/project/jep-src/src/test/python/jep_pipe.py", line 38, in jep_pipe
    assert False, stdout
AssertionError: b'2.2803010541544873e-31\n#\n# A fatal error has been detected by the Java Runtime Environment:\n#\n#  SIGSEGV (0xb) at pc=0x00007fff205cfa1c, pid=8469, tid=9987\n#\n# JRE version: OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (16.0.1+9) (build 16.0.1+9)\n# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (16.0.1+9, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-amd64)\n# Problematic frame:\n# C  [libsystem_platform.dylib+0x1a1c]  _platform_strchr$VARIANT$Haswell+0x1c\n#\n# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again\n#\n# An error report file with more information is saved as:\n# /Users/auser/project/jep-src/hs_err_pid8469.log\n#\n# If you would like to submit a bug report, please visit:\n#   https://github.com/AdoptOpenJDK/openjdk-support/issues\n# The crash happened outside the Java Virtual Machine in native code.\n# See problematic frame for where to report the bug.\n#\n'

----------------------------------------------------------------------
Ran 212 tests in 10.172s

FAILED (failures=2, skipped=13)
error: Unit tests failed with exit status 1

For the second and rest of times it fails with a different error:

$ python setup.py test                                                                                                 1 ↵
numpy include found at /Users/auser/project/.pyenv/versions/3.9.6/lib/python3.9/site-packages/numpy/core/include
running test
running build
running setup_java
running build_java
running build_jar
running build_py
running build_ext
running build_scripts
ln -sf jep.cpython-39-darwin.so build/lib.macosx-11.4-x86_64-3.9/jep/libjep.jnilib
.........................................................F.......#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007fff205cfa1c, pid=8630, tid=25095
#
# JRE version: OpenJDK Runtime Environment AdoptOpenJDK-16.0.1+9 (16.0.1+9) (build 16.0.1+9)
# Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (16.0.1+9, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-amd64)
# Problematic frame:
# C  [libsystem_platform.dylib+0x1a1c]  _platform_strchr$VARIANT$Haswell+0x1c
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/auser/project/jep-src/hs_err_pid8630.log
#
# If you would like to submit a bug report, please visit:
#   https://github.com/AdoptOpenJDK/openjdk-support/issues
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#
error: Unit tests failed with exit status -6

---

There is one more thing worth mentioning. When I upgraded to Big Sur, pyenv stopped building Python properly:

$ pyenv install
python-build: use [email protected] from homebrew
python-build: use readline from homebrew
Downloading Python-3.9.6.tar.xz...
-> https://www.python.org/ftp/python/3.9.6/Python-3.9.6.tar.xz
Installing Python-3.9.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 11.4 using python-build 2.0.3)

Inspect or clean up the working tree at /var/folders/vv/bk58y_8n48n2lrzy05f2tg580000gn/T/python-build.20210717203818.9926
Results logged to /var/folders/vv/bk58y_8n48n2lrzy05f2tg580000gn/T/python-build.20210717203818.9926.log

Last 10 log lines:
checking MACHDEP... "darwin"
checking for gcc... clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... configure: error: in `/var/folders/vv/bk58y_8n48n2lrzy05f2tg580000gn/T/python-build.20210717203818.9926/Python-3.9.6':
configure: error: cannot run C compiled programs.
If you meant to cross compile, use `--host'.
See `config.log' for more details
make: *** No targets specified and no makefile found.  Stop.

I found a workaround:

CC=gcc pyenv install

Still, export CC=gcc doesn't help to build and run jep properly.

---

My context:

• macOS BigSur 11.4
• jenv 0.5.4
• pyenv 2.0.3
• poetry 1.1.7
• python 3.9.6
• OpenJDK 64-Bit Server VM AdoptOpenJDK-16.0.1+9 (build 16.0.1+9, mixed mode, sharing)
• Homebrew clang version 12.0.1, Target: x86_64-apple-darwin20.5.0, InstalledDir: /usr/local/opt/llvm/bin

---

Hope you have some ideas on how to diagnose and fix the problem. It will be useful to run jep in osx.

[tburch]

I'm also facing this error and have tried various combinations of python (via pyenv) and jep, but nothing seems to work. I'm not a python expert, but happy to help troubleshoot...

My setup:

• Mac OS Monterey
• pyenv 2.2.2
• openjdk 11.0.11 2021-04-20

The error is

tburch> jep
java.lang.UnsatisfiedLinkError: /Users/tburch/.pyenv/versions/3.9.9/lib/python3.9/site-packages/jep/libjep.jnilib (dlopen(/Users/tburch/.pyenv/versions/3.9.9/lib/python3.9/site-packages/jep/libjep.jnilib, 0x0001): symbol not found in flat namespace '_libintl_bind_textdomain_codeset')
	at java.base/java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:1742)
	at java.base/java.lang.System.load(System.java:586)
	at jep.LibraryLocator.searchPackageDir(LibraryLocator.java:213)
	at jep.LibraryLocator.searchSitePackages(LibraryLocator.java:142)
	at jep.LibraryLocator.findJepLibrary(LibraryLocator.java:303)
	at jep.MainInterpreter.initialize(MainInterpreter.java:130)
	at jep.MainInterpreter.getMainInterpreter(MainInterpreter.java:101)
	at jep.Jep.<init>(Jep.java:256)
	at jep.SharedInterpreter.<init>(SharedInterpreter.java:56)
	at jep.Run.run(Run.java:49)
	at jep.Run.main(Run.java:146)