fairseq icon indicating copy to clipboard operation
fairseq copied to clipboard

Published 20 hours ago • verified publisher icon facebookresearch

fairseq-preprocess reported an error that fairseq could not be found

Open kkeleve opened this issue 3 years ago • 3 comments

The following is the command and log of my use of fairseq (the same operation can work a month ago, but now I tried the fairseq version that worked before and the latest version, but it doesn't work) code git clone https://github.com/pytorch/fairseq cd fairseq pip install --editable ./ log Looking in indexes: https://mirrors.aliyun.com/pypi/simple Obtaining file:///root/fairseq Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing wheel metadata ... done Collecting omegaconf<2.1 Downloading https://mirrors.aliyun.com/pypi/packages/d0/eb/9d63ce09dd8aa85767c65668d5414958ea29648a0eec80a4a7d311ec2684/omegaconf-2.0.6-py3-none-any.whl (36 kB) Requirement already satisfied: cffi in /usr/local/lib/python3.9/dist-packages (from fairseq==1.0.0a0+5b87224) (1.15.0) Collecting bitarray Downloading https://mirrors.aliyun.com/pypi/packages/39/0a/905eab325d3f8d0104351870da7013606d5502e8fddcbc7fec326790c041/bitarray-2.3.6.tar.gz (88 kB) |████████████████████████████████| 88 kB 468 kB/s Requirement already satisfied: numpy in /usr/local/lib/python3.9/dist-packages (from fairseq==1.0.0a0+5b87224) (1.22.0) Requirement already satisfied: torch in /usr/local/lib/python3.9/dist-packages (from fairseq==1.0.0a0+5b87224) (1.10.0+cu113) Collecting regex Downloading https://mirrors.aliyun.com/pypi/packages/89/8c/d587899aee993e201b369fb4007419b8a627190c52b40c2de0615f46dec1/regex-2022.1.18-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (763 kB) |████████████████████████████████| 763 kB 16.7 MB/s Collecting cython Using cached https://mirrors.aliyun.com/pypi/packages/9a/26/d2b6bc4cb7d716c82ebc89690cbd5ba0f547db364809cd42dad34d593182/Cython-0.29.28-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (1.9 MB) Collecting tqdm Downloading https://mirrors.aliyun.com/pypi/packages/63/f3/b7a1b8e40fd1bd049a34566eb353527bb9b8e9b98f8b6cf803bb64d8ce95/tqdm-4.62.3-py2.py3-none-any.whl (76 kB) |████████████████████████████████| 76 kB 395 kB/s Requirement already satisfied: torchaudio>=0.8.0 in /usr/local/lib/python3.9/dist-packages (from fairseq==1.0.0a0+5b87224) (0.10.0+cu113) Collecting hydra-core<1.1,>=1.0.7 Downloading https://mirrors.aliyun.com/pypi/packages/5f/2a/9c698daa12ed6e09e7629e6908528f043fa9de8a441c56cc13608d765fb2/hydra_core-1.0.7-py3-none-any.whl (123 kB) |████████████████████████████████| 123 kB 14.6 MB/s Collecting sacrebleu>=1.4.12 Downloading https://mirrors.aliyun.com/pypi/packages/fa/63/b3c11f951eafa2dc296862431f29fb12dbe191cb72217cf88ed04c32086b/sacrebleu-2.0.0-py3-none-any.whl (90 kB) |████████████████████████████████| 90 kB 744 kB/s Collecting antlr4-python3-runtime==4.8 Downloading https://mirrors.aliyun.com/pypi/packages/56/02/789a0bddf9c9b31b14c3e79ec22b9656185a803dc31c15f006f9855ece0d/antlr4-python3-runtime-4.8.tar.gz (112 kB) |████████████████████████████████| 112 kB 14.0 MB/s Collecting PyYAML>=5.1.* Downloading https://mirrors.aliyun.com/pypi/packages/12/fc/a4d5a7554e0067677823f7265cb3ae22aed8a238560b5133b58cda252dad/PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB) |████████████████████████████████| 661 kB 14.7 MB/s Requirement already satisfied: typing-extensions in /usr/local/lib/python3.9/dist-packages (from omegaconf<2.1->fairseq==1.0.0a0+5b87224) (4.0.1) Collecting tabulate>=0.8.9 Downloading https://mirrors.aliyun.com/pypi/packages/ca/80/7c0cad11bd99985cfe7c09427ee0b4f9bd6b048bd13d4ffb32c6db237dfb/tabulate-0.8.9-py3-none-any.whl (25 kB) Collecting colorama Downloading https://mirrors.aliyun.com/pypi/packages/44/98/5b86278fbbf250d239ae0ecb724f8572af1c91f4a11edf4d36a206189440/colorama-0.4.4-py2.py3-none-any.whl (16 kB) Collecting portalocker Downloading https://mirrors.aliyun.com/pypi/packages/f1/4e/1030afbf2e64e676e968bbbc82014ce4ddf1cc1ed0b492585958768cf79a/portalocker-2.4.0-py2.py3-none-any.whl (16 kB) Requirement already satisfied: pycparser in /usr/local/lib/python3.9/dist-packages (from cffi->fairseq==1.0.0a0+5b87224) (2.21) Building wheels for collected packages: antlr4-python3-runtime, bitarray Building wheel for antlr4-python3-runtime (setup.py) ... done Created wheel for antlr4-python3-runtime: filename=antlr4_python3_runtime-4.8-py3-none-any.whl size=141230 sha256=88c3cce7daacdfb2e2cc813e239921c6e4450dfd8fe7d5abd962b1bbeee8631e Stored in directory: /root/.cache/pip/wheels/7a/a5/04/563b329136444ecddffd9079b8e38cf1052dbc171f03e5314e Building wheel for bitarray (setup.py) ... done Created wheel for bitarray: filename=bitarray-2.3.6-cp39-cp39-linux_x86_64.whl size=172802 sha256=30473f574b9c03af0efa54fa81aec232058b0a38dc1087cff9a03a13064026b9 Stored in directory: /root/.cache/pip/wheels/13/12/09/c423942f6362665bafdfe713284cc21ba05b5a4a57037637c1 Successfully built antlr4-python3-runtime bitarray Installing collected packages: PyYAML, tabulate, regex, portalocker, omegaconf, colorama, antlr4-python3-runtime, tqdm, sacrebleu, hydra-core, cython, bitarray, fairseq Running setup.py develop for fairseq Successfully installed PyYAML-6.0 antlr4-python3-runtime-4.8 bitarray-2.3.6 colorama-0.4.4 cython-0.29.28 fairseq hydra-core-1.0.7 omegaconf-2.0.6 portalocker-2.4.0 regex-2022.1.18 sacrebleu-2.0.0 tabulate-0.8.9 tqdm-4.62.3 WARNING: You are using pip version 21.0.1; however, version 22.0.3 is available. You should consider upgrading via the '/usr/bin/python3.9 -m pip install --upgrade pip' command.

fairseq-preprocess --source-lang zh --target-lang en
--trainpref ~/data_prepare_wmt17/train
--validpref ~/data_prepare_wmt17/valid
--testpref ~/data_prepare_wmt17/test
--workers 1 --destdir ~/data-bin/wmt17_zh_en

error： Traceback (most recent call last): File "/usr/bin/fairseq-preprocess", line 33, in sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')()) File "/usr/bin/fairseq-preprocess", line 22, in importlib_load_entry_point for entry_point in distribution(dist_name).entry_points File "/usr/lib/python3.9/importlib/metadata.py", line 524, in distribution return Distribution.from_name(distribution_name) File "/usr/lib/python3.9/importlib/metadata.py", line 187, in from_name raise PackageNotFoundError(name) importlib.metadata.PackageNotFoundError: fairseq

❓ Questions and Help

I wonder why the package for fairseq is not found?

What's your environment?

  • fairseq Version (e.g., 1.0 or main): 0.10x
  • PyTorch Version (e.g., 1.0)：1.10
  • OS (e.g., Linux): Ubuntu 18.04.5
  • How you installed fairseq (pip, source): source
  • Build command you used (if compiling from source): git clone https://github.com/pytorch/fairseq cd fairseq pip install --editable ./
  • Python version: 3.9
  • CUDA/cuDNN version: 11.3
  • GPU models and configuration: 3090*1

kkeleve avatar Feb 21 '22 10:02 kkeleve

Hi I am getting the same error when trying with translation codebase here https://github.com/pytorch/fairseq/tree/main/examples/translation

running this command:

TEXT=examples/translation/iwslt14.tokenized.de-en
fairseq-preprocess --source-lang de --target-lang en \
    --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \
    --destdir data-bin/iwslt14.tokenized.de-en \
    --workers 20

dorost1234 avatar Mar 07 '22 22:03 dorost1234

Hi I am getting the same error when trying with translation codebase here https://github.com/pytorch/fairseq/tree/main/examples/translation

running this command:

TEXT=examples/translation/iwslt14.tokenized.de-en
fairseq-preprocess --source-lang de --target-lang en \
    --trainpref $TEXT/train --validpref $TEXT/valid --testpref $TEXT/test \
    --destdir data-bin/iwslt14.tokenized.de-en \
    --workers 20

hi, I've solved it, I'm just redeclaring fairseq location export PYTHONPATH=$PYTHONPATH:/root/fairseq/ hope it helps you

kkeleve avatar Mar 08 '22 03:03 kkeleve

hi, I've solved it, I'm just redeclaring fairseq location export PYTHONPATH=$PYTHONPATH:/root/fairseq/ hope it helps you

I've tried this but still get the same error

jessicalfr avatar Oct 23 '23 16:10 jessicalfr