ModuleNotFoundError: No module named 'examples.simultaneous_translation'

[RSid8]

🐛 Bug

Performing transfer learning using Roberta by following the custom classification readme in the Examples directory of Roberta. This code was working up to 1 week ago and now gives an error: ModuleNotFoundError: No module named 'examples.simultaneous_translation'

Steps to reproduce the behavior:

fairseq-preprocess \
    --only-source \
    --trainpref "/content/Task1a/train.input0.bpe" \
    --validpref "/content/Task1a/val.input0.bpe" \
    --destdir "/content/Task1a-bin/input0" \
    --workers 60 \
    --srcdict dict.txt

fairseq-preprocess \
    --only-source \
    --trainpref "/content/Task1a/train.label" \
    --validpref "/content/Task1a/val.label" \
    --destdir "/content/Task1a-bin/label" \
    --workers 60

2. Traceback:

--2021-02-28 05:54:18--  https://dl.fbaipublicfiles.com/fairseq/gpt2_bpe/dict.txt
Resolving dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)... 104.22.75.142, 172.67.9.4, 104.22.74.142, ...
Connecting to dl.fbaipublicfiles.com (dl.fbaipublicfiles.com)|104.22.75.142|:443... connected.
HTTP request sent, awaiting response... 304 Not Modified
File ‘dict.txt’ not modified on server. Omitting download.

Traceback (most recent call last):
  File "/usr/local/bin/fairseq-preprocess", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')())
  File "/usr/local/bin/fairseq-preprocess", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/__init__.py", line 100, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/content/fairseq/fairseq_cli/preprocess.py", line 18, in <module>
    from fairseq import options, tasks, utils
  File "/content/fairseq/fairseq/__init__.py", line 32, in <module>
    import fairseq.criterions  # noqa
  File "/content/fairseq/fairseq/criterions/__init__.py", line 36, in <module>
    importlib.import_module("fairseq.criterions." + file_name)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/content/fairseq/fairseq/criterions/label_smoothed_cross_entropy_latency_augmented.py", line 6, in <module>
    from examples.simultaneous_translation.utils.latency import LatencyTraining
ModuleNotFoundError: No module named 'examples.simultaneous_translation'
Traceback (most recent call last):
  File "/usr/local/bin/fairseq-preprocess", line 33, in <module>
    sys.exit(load_entry_point('fairseq', 'console_scripts', 'fairseq-preprocess')())
  File "/usr/local/bin/fairseq-preprocess", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.7/dist-packages/importlib_metadata/__init__.py", line 100, in load
    module = import_module(match.group('module'))
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/content/fairseq/fairseq_cli/preprocess.py", line 18, in <module>
    from fairseq import options, tasks, utils
  File "/content/fairseq/fairseq/__init__.py", line 32, in <module>
    import fairseq.criterions  # noqa
  File "/content/fairseq/fairseq/criterions/__init__.py", line 36, in <module>
    importlib.import_module("fairseq.criterions." + file_name)
  File "/usr/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/content/fairseq/fairseq/criterions/label_smoothed_cross_entropy_latency_augmented.py", line 6, in <module>
    from examples.simultaneous_translation.utils.latency import LatencyTraining
ModuleNotFoundError: No module named 'examples.simultaneous_translation'

Expected behavior

Expected to save the preprocessed and tokenized input and labels for training

Environment

• fairseq Version (master):
• Google Colab

[lematt1991]

Can you try re-running pip install . --editable. I'm unable to reproduce this on my side.

[KirillKonovalov]

I had the same issue. What can be done?

[RSid8]

@lematt1991 I did try that but it failed to work. However, I did a regular

!pip install fairseq

and cloned the repository after. This caused the error to stop appearing. @KirillKonovalov that might be helpful to you.

[benjamin3344]

@lematt1991 I did try that but it failed to work. However, I did a regular

!pip install fairseq

and cloned the repository after. This caused the error to stop appearing. @KirillKonovalov that might be helpful to you.

If I am installing and running fairseq locally (installing from setup.py), how can I solve this problem? @RSid8

[wesleywt]

Adding fairseq to the PYTHONPATH solved the problem for me. I got the solution from this issue

cd /content 
!echo $PYTHONPATH

import os

os.environ['PYTHONPATH'] += ":/content/fairseq/" #path to your fairseq 

!echo $PYTHONPATH

[aanchan]

This probably is not a bug. The examples folder falls outside of the purview of what is "included" in the package. If you check at the repo level its fairseq/fairseq and fairseq/examples. Anything inside fairseq/fairseq appears importable. So fairseq/examples will not be importable.

[wangtianrui]

A simple solution inspired by @wesleywt :

PYTHONPATH=/home/wang/codes/py/fairseq python -u examples/speech_to_speech/preprocessing/prep_s2ut_data.py 

[donglixp]

FAIRSEQ_DIR=$(pip list -v | grep 'fairseq' | awk '{print $3}')
export PYTHONPATH=$PYTHONPATH:$FAIRSEQ_DIR