Error running unit tests when building with setup.py install

[ronakmal]

🐛 Bug

Describe the bug A clear and concise description of what the bug is. When building with python setup.py install, running pytest from either the project root directory or the test/ directory causes the error ImportError: torchtext C++ Extension is not found. This can be worked-around by renaming the torchtext subdirectory, or by instead using python setup.py develop like the CI does (see .circleci/unittest/linux/scripts/install.sh#L36).

To Reproduce Steps to reproduce the behavior:

1. Follow the build steps like normal, running python setup.py install
2. Run pytest
3. Every test fails with the error ImportError: torchtext C++ Extension is not found.

Expected behavior A clear and concise description of what you expected to happen.

The tests should succeed even when installing with setup.py install, either running pytest from the project root or the test/ directory (this is the case in pytorch) without having to rename the torchtext subdirectory.

Screenshots If applicable, add screenshots to help explain your problem.

Environment

Please copy and paste the output from our environment collection script (or fill out the checklist below manually).

You can get the script and run it with:

wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
# For security purposes, please check the contents of collect_env.py before running it.
python collect_env.py
python -c "import torchtext; print(\"torchtext version is \", torchtext.__version__)"

• PyTorch Version (e.g., 1.0): 1.12
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, source): Compiled from source
• Build command you used (if compiling from source): python3 ./setup.py install
• Python version: 3.7.13
• CUDA/cuDNN version: ROCm version 5.2
• GPU models and configuration: N/A
• Any other relevant information:

Additional context Add any other context about the problem here.

[parmeet]

https://github.com/pytorch/text/pull/1874 resolves this.

[ronakmal]

Hi @parmeet. While installing torchdata does satisfy one of the testing requirements, I'm still running into the problem where the tests will error out with ImportError: torchtext C++ Extension is not found if I build with setup.py install instead of setup.py develop. Is there any way to fix this issue?

[mthrok]

Hi @parmeet. While installing torchdata does satisfy one of the testing requirements, I'm still running into the problem where the tests will error out with ImportError: torchtext C++ Extension is not found if I build with setup.py install instead of setup.py develop. Is there any way to fix this issue?

This happens because the source code directory shadows the installed package. One proper fix is to introduce intermediate directory and put all the unit test code in there. In TorchAudio, unit test codes are placed in torchaudio_unittest directory.

There was a PR to do this clean up but it got staled. https://github.com/pytorch/text/pull/1223

cc @Nayef211