fairseq
fairseq copied to clipboard
Implement dummy lock in pdb module.
This implements a dummy lock in the pdb module to replace the multiprocessing.Lock
on platforms where not all features are available to support the multiprocessing module. At the very least, this is the case on AWS Lambda, where importing the fairseq
package triggers importing fairseq.pdb
, which fails thus:
[ERROR] OSError: [Errno 38] Function not implemented Traceback (most recent call last):
File "/opt/pyenv/versions/3.8.15/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/app/APP_DIRECTORY/__init__.py", line 1, in <module>
from .transformer import transformer
File "/app/APP_DIRECTORY/transformer/transformer.py", line 1, in <module>
from fairseq.models import register_model_architecture
File "/app/.venv/lib/python3.8/site-packages/fairseq/__init__.py", line 39, in <module>
import fairseq.pdb # noqa
File "/app/.venv/lib/python3.8/site-packages/fairseq/pdb.py", line 16, in <module>
_stdin_lock = multiprocessing.Lock()
File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/context.py", line 68, in Lock
return Lock(ctx=self.get_context())
File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
File "/opt/pyenv/versions/3.8.15/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
sl = self._semlock = _multiprocessing.SemLock(
The modified code will catch the OSError and supply a fake lock instead. This is safe because in the event _cmdloop
executes under a dummy lock, the code won't be running in a multiprocessing environment.
Before submitting
- [ ] Was this discussed/approved via a Github issue? (no need for typos, doc improvements)
- [x] Did you read the contributor guideline?
- [ ] Did you make sure to update the docs?
- [ ] Did you write any new necessary tests?
What does this PR do?
Fixes # (issue).
PR review
Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in Github issues there's a high chance it will not be merged.
Did you have fun?
Make sure you had fun coding 🙃