OpusFilter
OpusFilter copied to clipboard
Published
20 hours ago •
Helsinki-NLP
Helsinki-NLP
Issue with opus-fast-mosestokenizer dep for ARM-macs
Hi!
I am currently experiencing an issue with the installation on M-series Macs (via pip install opusfilter).
Unfortunately, the installation process fails due to a problem with the opus-fast-mosestokenizer dependency.
Tested with Python 3.8/3.9/3.10/3.11
Some output:
Building wheel for opus-fast-mosestokenizer (setup.py) ... error
error: subprocess-exited-with-error
× python setup.py bdist_wheel did not run successfully.
│ exit code: 1
╰─> [117 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-11.1-arm64-cpython-311
creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer
copying bindings/python/mosestokenizer/__init__.py -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer
creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share
copying bindings/python/mosestokenizer/share/README.txt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share
creating build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.hi -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.as -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.de -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.nl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.hu -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.bn -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.kn -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ml -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.yue -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.mni -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pa -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.zh -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ta -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.en -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.lt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.or -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.fr -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.es -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.et -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.fi -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.gu -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.mr -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ga -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sv -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.pl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sl -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.sk -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/README.txt -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ro -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.te -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ru -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.cs -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.it -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.is -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.ca -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.el -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
copying bindings/python/mosestokenizer/share/nonbreaking_prefixes/nonbreaking_prefix.lv -> build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/share/nonbreaking_prefixes
running build_ext
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
-- The CXX compiler identification is AppleClang 15.0.0.15000100
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
CMake Error at /opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
Could NOT find Boost (missing: Boost_INCLUDE_DIR program_options thread)
Call Stack (most recent call first):
/opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:600 (_FPHSA_FAILURE_MESSAGE)
/opt/homebrew/Cellar/cmake/3.28.3/share/cmake/Modules/FindBoost.cmake:2393 (find_package_handle_standard_args)
CMakeLists.txt:70 (find_package)
-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
File "<string>", line 2, in <module>
File "<pip-setuptools-caller>", line 34, in <module>
File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 107, in <module>
setup(
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
^^^^^^^^^^^^^^^^^^
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 364, in run
self.run_command("build")
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 131, in run
self.run_command(cmd_name)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 318, in run_command
self.distribution.run_command(command)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 53, in run
self.build_extension(ext)
File "/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/setup.py", line 91, in build_extension
subprocess.check_call(cmake_args, cwd=self.build_temp, env=env)
File "/Users/pstepach/miniconda3/envs/clear/lib/python3.11/subprocess.py", line 413, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/private/var/folders/tg/b_nprk2142s5bh07pw240f3r0000gp/T/pip-install-sd9_6q0a/opus-fast-mosestokenizer_86290359b51c412aaad6b283997c5351/build/lib.macosx-11.1-arm64-cpython-311/mosestokenizer/lib/', '-DPYTHON_EXECUTABLE=/Users/pstepach/miniconda3/envs/clear/bin/python', '-DBUILD_CLI:BOOL=OFF', '-DBUILD_PYTHON:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for opus-fast-mosestokenizer
Running setup.py clean for opus-fast-mosestokenizer
Building wheel for langid (setup.py) ... done
Created wheel for langid: filename=langid-1.1.6-py3-none-any.whl size=1941173 sha256=fb9479a4d2fdb43a32e92a248da54a802ec26086bea4c2a25ba98c612f77ac7a
Stored in directory: /Users/pstepach/Library/Caches/pip/wheels/32/6a/b6/b7eb43a6ad55b139c15c5daa29f3707659cfa6944d3c696f5b
Building wheel for pycld2 (setup.py) ... done
Created wheel for pycld2: filename=pycld2-0.41-cp311-cp311-macosx_11_0_arm64.whl size=4816791 sha256=16ddfa06735a6e70169b323d9f67f81eebbee850b8fa524a5a8aa9d87140c69c
Stored in directory: /Users/pstepach/Library/Caches/pip/wheels/9a/71/a1/89cdc6ae2ecd7bb3bf67baee27a134f761b3e2922ea7fdfbcb
Successfully built langid pycld2
Failed to build opus-fast-mosestokenizer
ERROR: Could not build wheels for opus-fast-mosestokenizer, which is required to install pyproject.toml-based projects```
There was a pending pull request on opus-fast-mosestokenizer related to this. It's now merged to the master branch, can you test if the change helps?
Still issues. I tried https://github.com/Helsinki-NLP/opus-fast-mosestokenizer/blob/master/INSTALL.md#from-source
(arm_fix) ➜ opus-fast-mosestokenizer git:(master) python -c "import pybind11; print(pybind11.get_cmake_dir())"
/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/pybind11/share/cmake/pybind11
opus-fast-mosestokenizer git:(master) python setup.py build_ext install
running build_ext
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
Compatibility with CMake < 3.5 will be removed from a future version of
CMake.
Update the VERSION argument <min> value or use a ...<max> suffix to tell
CMake that the project does not need compatibility with older versions.
CMake Error at CMakeLists.txt:154 (find_package):
By not providing "Findpybind11.cmake" in CMAKE_MODULE_PATH this project has
asked CMake to find a package configuration file provided by "pybind11",
but CMake did not find one.
Could not find a package configuration file provided by "pybind11" with any
of the following names:
pybind11Config.cmake
pybind11-config.cmake
Add the installation prefix of "pybind11" to CMAKE_PREFIX_PATH or set
"pybind11_DIR" to a directory containing one of the above files. If
"pybind11" provides a separate development package or SDK, be sure it has
been installed.
-- Configuring incomplete, errors occurred!
Traceback (most recent call last):
File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 107, in <module>
setup(
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/__init__.py", line 103, in setup
return distutils.core.setup(**attrs)
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
return run_commands(dist)
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
dist.run_commands()
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
self.run_command(cmd)
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/dist.py", line 989, in run_command
super().run_command(command)
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
cmd_obj.run()
File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 53, in run
self.build_extension(ext)
File "/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/setup.py", line 91, in build_extension
subprocess.check_call(cmake_args, cwd=self.build_temp, env=env)
File "/Users/pstepach/miniconda3/envs/arm_fix/lib/python3.10/subprocess.py", line 369, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer', '-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/Users/pstepach/Desktop/WORKING_FOLDER/arm_fix/opus-fast-mosestokenizer/build/lib.macosx-11.1-arm64-cpython-310/mosestokenizer/lib/', '-DPYTHON_EXECUTABLE=/Users/pstepach/miniconda3/envs/arm_fix/bin/python', '-DBUILD_CLI:BOOL=OFF', '-DBUILD_PYTHON:BOOL=ON', '-DBUILD_SHARED_LIBS:BOOL=ON', '-DCMAKE_BUILD_TYPE=Release']' returned non-zero exit status 1.```