pynini

pynini copied to clipboard

When I install via pip from current master, I get this:

Collecting pynini@ git+https://github.com/kylebgorman/pynini.git#egg=pynini
  Cloning https://github.com/kylebgorman/pynini.git to /tmp/pip-install-477ckk7a/pynini
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-477ckk7a/pynini/setup.py'"'"'; __file__='"'"'/tmp/pip-install-477ckk7a/pynini/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-vy_k3j1z
         cwd: /tmp/pip-install-477ckk7a/pynini/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-477ckk7a/pynini/setup.py", line 22, in <module>
        from Cython.Build import cythonize
    ModuleNotFoundError: No module named 'Cython'

(I do not recall seeing this problem in the past, but maybe I did things differently then).

The cause is simple: setup.py tries to import Cython directly, before it gets the chance to follow install_requires. Traditionally, people have coped with this by all kinds of tricks (like calling the pip cmdline from setup).

But there's good news: we can simply use pyproject.toml instead!

I have never tried to install this way (pip+git)---the only way that is likely to work in a wide variety is via Conda-Forge, since you need big non-Python dependencies---but I can replicate this in a totally clean client. Would you be willing to submit a PR?

(We probably should do this against the current Pynini 2.1.2---this is just an out of date mirror, though I'll update it when I get a chance.)

I have never tried to install this way (pip+git)---the only way that is likely to work in a wide variety is via Conda-Forge, since you need big non-Python dependencies

I know, it only works when manually compiling OpenFST, since Debian/Ubuntu are still at 1.6.3 :-/ Nevertheless, this setup at least encapsulates the Python part, and Conda is no option for me unfortunately.

Would you be willing to submit a PR?

I could work it, yes.

We probably should do this against the current Pynini 2.1.2---this is just an out of date mirror, though I'll update it when I get a chance.

Definitely. I am already running into #29 with 1.7.9 and cannot compile OpenFST 1.7.5 with recent g++ (internal compiler error) – so if you could upgrade this would help a lot.

I am already running into #29 with 1.7.9 and cannot compile OpenFST 1.7.5 with recent g++ (internal compiler error) – so if you could upgrade this would help a lot.

correction: I cannot compile pynini (both the version here and upstream) due to internal compiler error. So this is a completely independent issue (#31), and I should not wait for an update with my PR.

Hi, the build system bootstrap with Cython and setuptools can now be solved with using pyproject.toml files. See #62 for a fix.

JFR I'm also experiencing the same issue. Installing with pip and poetry (which uses pip underneath)

Installing in a fresh environment

╰─ pip install pynini
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [6 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-install-y62o9qws/pynini_3f8387eb1916426ba4ad0b9434a0e531/setup.py", line 22, in <module>
          from Cython.Build import cythonize
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

And this is testing whether is PEP517 compatible

╰─ pip install pynini --use-pep517
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  error: subprocess-exited-with-error

  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Traceback (most recent call last):
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
          self.run_setup()
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 485, in run_setup
          self).run_setup(setup_script=setup_script)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/private/var/folders/qp/8v0z3y5n3y5_g0ftmdrr0b800000gs/T/pip-build-env-_pjzuka3/overlay/lib/python3.11/site-packages/setuptools/build_meta.py", line 335, in run_setup
          exec(code, locals())
        File "<string>", line 22, in <module>
      ModuleNotFoundError: No module named 'Cython'
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Error even with Cython

If I add Cython before installing if fails to compile because it doesn't downloads openFST

╰─ pip install pynini
Collecting pynini
  Using cached pynini-2.1.5.tar.gz (627 kB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: Cython>=0.29 in /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages (from pynini) (0.29.33)
Installing collected packages: pynini
  DEPRECATION: pynini is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
  Running setup.py install for pynini ... error
  error: subprocess-exited-with-error

  × Running setup.py install for pynini did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]
      running install
      /Users/aalvarez/.pyenv/versions/test/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib.macosx-13.2-x86_64-cpython-311
      creating build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/chatspeak_model.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/weather.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/numbers.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/plurals.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/t9.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/case.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/dates.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/examples/g2p.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/edit_transducer.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/utf8.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/features.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/pynutil.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/paradigms.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/tagger.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rule_cascade.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/rewrite.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/lib/byte.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      creating build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/__init__.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/export.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/multi_grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pynini/export/grm_example.py -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      copying pywrapfst/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pywrapfst/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pywrapfst
      copying pynini/__init__.pyi -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini
      copying pynini/examples/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/examples
      copying pynini/lib/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/lib
      copying pynini/export/py.typed -> build/lib.macosx-13.2-x86_64-cpython-311/pynini/export
      running build_ext
      building '_pywrapfst' extension
      creating build/temp.macosx-13.2-x86_64-cpython-311
      creating build/temp.macosx-13.2-x86_64-cpython-311/extensions
      clang -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Applications/Xcode-14.2.0.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include -I/Users/aalvarez/.pyenv/versions/test/include -I/Users/aalvarez/.pyenv/versions/3.11.2/include/python3.11 -c extensions/_pywrapfst.cpp -o build/temp.macosx-13.2-x86_64-cpython-311/extensions/_pywrapfst.o -std=c++17 -Wno-register -Wno-deprecated-declarations -Wno-unused-function -Wno-unused-local-typedefs -funsigned-char -stdlib=libc++ -mmacosx-version-min=10.7
      extensions/_pywrapfst.cpp:815:10: fatal error: 'fst/util.h' file not found
      #include <fst/util.h>
               ^~~~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pynini

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

@ Alejandro this is a know issue and will be fixed in the next release, but the second error is unrelated.

Any update on this?

It should be fixed by the migration to pyproject.toml in the next release. I can't make any promises when that'll be though I am working on it this fall. In the meantime, consider using one of the two sources for precompiled binaries of the library.

Note to any readers: this is not platform-specific as far as I know.

I put out a post-release because a lot of people were having trouble with this. I hope it solves the issue. It'll be on PyPI shortly.