Error Using Back-translate Functionality

[IllyShaieb]

TL;DR

When trying to run backtranslation on some text data, I encountered two errors, one was a urllib.error.HTTPError and the other was with installing dependencies for backtranslation.

urllib.error.HTTPError

I was trying to run back-translation on some text data, and encountered the following error:

...
Traceback (most recent call last):
  File "C:\Code\ENVIRONMENTS\ace-env\ACE\.tmp\test2.py", line 35, in <module>
    augmented_text = tx(row[0])
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 180, in add_backtranslation
    t = _Translator()
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 46, in __init__
    self.load_models()
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\niacin\text\en\sentence.py", line 64, in load_models
    cls.translators["en2de"] = torch.hub.load(
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 362, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 162, in _get_cache_or_reload
    _validate_not_a_forked_repo(repo_owner, repo_name, branch)
  File "C:\Code\ENVIRONMENTS\ace-env\lib\site-packages\torch\hub.py", line 124, in _validate_not_a_forked_repo
    with urlopen(url) as r:
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 523, in open
    response = meth(req, response)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 632, in http_response
    response = self.parent.error(
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 561, in error
    return self._call_chain(*args)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 494, in _call_chain
    result = func(*args)
  File "c:\users\illys\appdata\local\programs\python\python39\lib\urllib\request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: rate limit exceeded

Which led me to see if the error was due to dependencies not being installed.

Installing Dependencies for Backtranslation.

I ran the command pip install niacin[backtranslate] in Windows 10 Powershell, to see if it would fix the urllib.error.HTTPError. However, I received the following error:

...
Building wheel for fastbpe (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\IllyS\AppData\Local\Temp\pip-wheel-4khl2d2a'
       cwd: C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\
  Complete output (19 lines):
  running bdist_wheel
  running build
  running build_py
  package init file 'fastBPE\__init__.py' not found (or not a regular file)
  running build_ext
  building 'fastBPE' extension
  creating build
  creating build\temp.win-amd64-3.9
  creating build\temp.win-amd64-3.9\Release
  creating build\temp.win-amd64-3.9\Release\fastBPE
  C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IfastBPE -Ic:\code\environments\ace-env\include -Ic:\users\illys\appdata\local\programs\python\python39\include -Ic:\users\illys\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /TpfastBPE/fastBPE.cpp /Fobuild\temp.win-amd64-3.9\Release\fastBPE/fastBPE.obj -std=c++11 -Ofast -pthread
  cl : Command line warning D9025 : overriding '/Os' with '/Ot'
  cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
  cl : Command line warning D9002 : ignoring unknown option '-Of'
  cl : Command line warning D9002 : ignoring unknown option '-Oa'
  cl : Command line warning D9002 : ignoring unknown option '-pthread'
  fastBPE.cpp
  C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\fastBPE\fastBPE.hpp(15): fatal error C1083: Cannot open include file: 'sys/mman.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
  ----------------------------------------
  ERROR: Failed building wheel for fastbpe
  Running setup.py clean for fastbpe
Failed to build fastbpe
Installing collected packages: fastbpe
    Running setup.py install for fastbpe ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\IllyS\AppData\Local\Temp\pip-record-7y660u4y\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\code\environments\ace-env\include\site\python3.9\fastbpe'
         cwd: C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\
    Complete output (19 lines):
    running install
    running build
    running build_py
    package init file 'fastBPE\__init__.py' not found (or not a regular file)
    running build_ext
    building 'fastBPE' extension
    creating build
    creating build\temp.win-amd64-3.9
    creating build\temp.win-amd64-3.9\Release
    creating build\temp.win-amd64-3.9\Release\fastBPE
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IfastBPE -Ic:\code\environments\ace-env\include -Ic:\users\illys\appdata\local\programs\python\python39\include -Ic:\users\illys\appdata\local\programs\python\python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\include -IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /EHsc /TpfastBPE/fastBPE.cpp /Fobuild\temp.win-amd64-3.9\Release\fastBPE/fastBPE.obj -std=c++11 -Ofast -pthread
    cl : Command line warning D9025 : overriding '/Os' with '/Ot'
    cl : Command line warning D9002 : ignoring unknown option '-std=c++11'
    cl : Command line warning D9002 : ignoring unknown option '-Of'
    cl : Command line warning D9002 : ignoring unknown option '-Oa'
    cl : Command line warning D9002 : ignoring unknown option '-pthread'
    fastBPE.cpp
    C:\Users\IllyS\AppData\Local\Temp\pip-install-ksmbdnd6\fastbpe_27df61036fe14cf4a894595f913b7246\fastBPE\fastBPE.hpp(15): fatal error C1083: Cannot open include file: 'sys/mman.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\code\environments\ace-env\scripts\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"'; __file__='"'"'C:\\Users\\IllyS\\AppData\\Local\\Temp\\pip-install-ksmbdnd6\\fastbpe_27df61036fe14cf4a894595f913b7246\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\IllyS\AppData\Local\Temp\pip-record-7y660u4y\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\code\environments\ace-env\include\site\python3.9\fastbpe' Check the logs for full command output.

Please could you advise if there is anything I can do to remedy these errors?

[deniederhut]

Sorry for taking so long to get to this! Okay so I googled around a bit and it looks like this is a known issue with FastBPE. Unfortunately the solution looks to be pretty complicated -> https://github.com/pytorch/fairseq/issues/1224#issuecomment-881893627.

This issue is from last year, so it may have been fixed upstream? I might try installing the dependency again, and if it continues to fail, try the steps from that comment in the fairseq issue tracker. Or, if that's too much trouble, you might want to stick with the other augmentation methods.