Basic doc example not running in poetry shell. "No module named _lzma"

[ahmed-moubtahij]

poetry add evaluate

Using version ^0.2.2 for evaluate

Updating dependencies
Resolving dependencies... (86.1s)

Writing lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing frozenlist (1.3.1)
  • Installing multidict (6.0.2)
  • Installing aiosignal (1.2.0)
  • Installing async-timeout (4.0.2)
  • Installing yarl (1.8.1)
  • Installing aiohttp (3.8.1)
  • Installing pytz (2022.1)
  • Installing fsspec (2022.7.1)
  • Installing multiprocess (0.70.13)
  • Installing pandas (1.4.3)
  • Installing pyarrow (9.0.0)
  • Installing responses (0.18.0)
  • Installing xxhash (3.0.0)
  • Installing datasets (2.4.0)
  • Installing evaluate (0.2.2)

import evaluate

rouge = evaluate.load('rouge')
predictions = ["hello there", "general kenobi"]
references = ["hello there", "general kenobi"]
results = rouge.compute(predictions=predictions,
                        references=references)
print(results)

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/user/LDS/experiments/ex_randomsum.py", line 3, in <module>
    import evaluate
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/evaluate/__init__.py", line 29, in <module>
    from .evaluator import (
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/evaluate/evaluator/__init__.py", line 27, in <module>
    from .base import Evaluator
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/evaluate/evaluator/base.py", line 20, in <module>
    from datasets import Dataset, load_dataset
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/__init__.py", line 37, in <module>
    from .arrow_dataset import Dataset
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 61, in <module>
    from .arrow_reader import ArrowReader
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/arrow_reader.py", line 29, in <module>
    from .download.download_config import DownloadConfig
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/download/__init__.py", line 9, in <module>
    from .download_manager import DownloadManager, DownloadMode
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/download/download_manager.py", line 29, in <module>
    from ..utils.file_utils import cached_path, get_from_cache, hash_url_to_filename, is_relative_path, url_or_path_join
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/utils/file_utils.py", line 30, in <module>
    from .extract import ExtractManager
  File "/home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10/lib/python3.10/site-packages/datasets/utils/extract.py", line 3, in <module>
    import lzma
  File "/usr/local/lib/python3.10/lzma.py", line 27, in <module>
    from _lzma import *
ModuleNotFoundError: No module named '_lzma'

[lvwerra]

Could this be similar to this issue: https://github.com/pandas-dev/pandas/issues/27532?

[ahmed-moubtahij]

Seems like it, however trying the same approach with Poetry instead of pyenv didn't work. poetry env remove /usr/local/bin/python3.10 Deleted virtualenv: /home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10 sudo apt-get install liblzma-dev

Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  liblzma-dev
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 147 kB of archives.
After this operation, 603 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 liblzma-dev amd64 5.2.4-1ubuntu1.1 [147 kB]
Fetched 147 kB in 0s (550 kB/s)
Selecting previously unselected package liblzma-dev:amd64.
(Reading database ... 71580 files and directories currently installed.)
Preparing to unpack .../liblzma-dev_5.2.4-1ubuntu1.1_amd64.deb ...
Unpacking liblzma-dev:amd64 (5.2.4-1ubuntu1.1) ...
Setting up liblzma-dev:amd64 (5.2.4-1ubuntu1.1) ...

poetry shell

Using python3.10 (3.10.5)
Creating virtualenv lds-dfhFRtYN-py3.10 in /home/ahmedm/.cache/pypoetry/virtualenvs
Spawning shell within /home/user/.cache/pypoetry/virtualenvs/lds-dfhFRtYN-py3.10

poetry update
python -m main # contains the huggingface doc example for ROUGE

ModuleNotFoundError: No module named '_lzma'

I ended up using google-research's implementation of ROUGE, it worked.

[lvwerra]

Maybe this helps? https://stackoverflow.com/questions/59690698/modulenotfounderror-no-module-named-lzma-when-building-python-using-pyenv-on