fairseq icon indicating copy to clipboard operation
fairseq copied to clipboard

Implement dummy lock in pdb module.

Open tmehlinger opened this issue 2 years ago • 0 comments

This implements a dummy lock in the pdb module to replace the multiprocessing.Lock on platforms where not all features are available to support the multiprocessing module. At the very least, this is the case on AWS Lambda, where importing the fairseq package triggers importing fairseq.pdb, which fails thus:

[ERROR] OSError: [Errno 38] Function not implemented Traceback (most recent call last):
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/APP_DIRECTORY/__init__.py", line 1, in <module>
    from .transformer import transformer
  File "/app/APP_DIRECTORY/transformer/transformer.py", line 1, in <module>
    from fairseq.models import register_model_architecture
  File "/app/.venv/lib/python3.8/site-packages/fairseq/__init__.py", line 39, in <module>
    import fairseq.pdb  # noqa
  File "/app/.venv/lib/python3.8/site-packages/fairseq/pdb.py", line 16, in <module>
    _stdin_lock = multiprocessing.Lock()
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(

The modified code will catch the OSError and supply a fake lock instead. This is safe because in the event _cmdloop executes under a dummy lock, the code won't be running in a multiprocessing environment.

Before submitting

  • [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
  • [x] Did you read the contributor guideline?
  • [ ] Did you make sure to update the docs?
  • [ ] Did you write any new necessary tests?

What does this PR do?

Fixes # (issue).

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

tmehlinger avatar Dec 02 '22 23:12 tmehlinger