Cannot import fairseq in an environment that doesn't support multiprocessing

[tmehlinger]

🐛 Bug

I've opened #4886 to fix this.

Fire up a Lambda with fairseq in it and watch the fireworks!

[ERROR] OSError: [Errno 38] Function not implemented Traceback (most recent call last):
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/APP_DIRECTORY/__init__.py", line 1, in <module>
    from .transformer import transformer
  File "/app/APP_DIRECTORY/transformer/transformer.py", line 1, in <module>
    from fairseq.models import register_model_architecture
  File "/app/.venv/lib/python3.8/site-packages/fairseq/__init__.py", line 39, in <module>
    import fairseq.pdb  # noqa
  File "/app/.venv/lib/python3.8/site-packages/fairseq/pdb.py", line 16, in <module>
    _stdin_lock = multiprocessing.Lock()
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

To Reproduce

Import fairseq in a Python Lambda function.

Expected behavior

Normal behavior, rather than an OSError. Ideally, the library shouldn't import pdb at runtime at all but I think that's a separate discussion. 😄

Environment

• AWS Lambda, Python 3.8
• fairseq 0.10.2
• pytorch 1.8.1