apex
apex copied to clipboard
NVIDIA
Problem installation
I tried to install Apex with CUDA and C++ extenstions, but I get the following error:
/usr/lib/python3/dist-packages/pip/commands/install.py:212: UserWarning: Disabling all use of wheels due to the use of --build-options / --global-options / --install-options.
cmdoptions.check_install_build_global(options)
Processing /home/emanuele/apex
Running setup.py (path:/tmp/pip-b5r41wi9-build/setup.py) egg_info for package from file:///home/emanuele/apex
Running command python setup.py egg_info
torch.__version__ = 1.1.0
running egg_info
creating pip-egg-info/apex.egg-info
writing pip-egg-info/apex.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/apex.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/apex.egg-info/top_level.txt
writing manifest file 'pip-egg-info/apex.egg-info/SOURCES.txt'
reading manifest file 'pip-egg-info/apex.egg-info/SOURCES.txt'
writing manifest file 'pip-egg-info/apex.egg-info/SOURCES.txt'
/tmp/pip-b5r41wi9-build/setup.py:44: UserWarning: Option --pyprof not specified. Not installing PyProf dependencies!
warnings.warn("Option --pyprof not specified. Not installing PyProf dependencies!")
Source in /tmp/pip-b5r41wi9-build has version 0.1, which satisfies requirement apex==0.1 from file:///home/emanuele/apex
Installing collected packages: apex
Running setup.py install for apex ... Running command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-b5r41wi9-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --pyprof install --record /tmp/pip-fg1d3igz-record/install-record.txt --single-version-externally-managed --compile --user --prefix=
torch.__version__ = 1.1.0
Collecting cxxfilt>=0.2.0
Collecting tqdm>=4.28.1
Using cached https://files.pythonhosted.org/packages/e1/c1/bc1dba38b48f4ae3c4428aea669c5e27bd5a7642a74c8348451e0bd8ff86/tqdm-4.36.1-py2.py3-none-any.whl
Collecting numpy>=1.15.3
Using cached https://files.pythonhosted.org/packages/0e/46/ae6773894f7eacf53308086287897ec568eac9768918d913d5b9d366c5db/numpy-1.17.3-cp36-cp36m-manylinux1_x86_64.whl
Collecting PyYAML>=5.1
Collecting pytest>=3.5.1
Using cached https://files.pythonhosted.org/packages/0c/91/d68f68ce54cd3e8afa1ef73ea1ad44df2438521b64c0820e5fd9b9f13b7d/pytest-5.2.1-py3-none-any.whl
Collecting atomicwrites>=1.0 (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/52/90/6155aa926f43f2b2a22b01be7241be3bfd1ceaf7d0b3267213e8127d41f4/atomicwrites-1.3.0-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting packaging (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/cf/94/9672c2d4b126e74c4496c6b3c58a8b51d6419267be9e70660ba23374c875/packaging-19.2-py2.py3-none-any.whl
Collecting pluggy<1.0,>=0.12 (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/92/c7/48439f7d5fd6bddb4c04b850bb862b42e3e2b98570040dfaf68aedd8114b/pluggy-0.13.0-py2.py3-none-any.whl
Collecting wcwidth (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/7e/9f/526a6947247599b084ee5232e4f9190a38f398d7300d866af3ab571a5bfe/wcwidth-0.1.7-py2.py3-none-any.whl
Collecting py>=1.5.0 (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/76/bc/394ad449851729244a97857ee14d7cba61ddb268dce3db538ba2f2ba1f0f/py-1.8.0-py2.py3-none-any.whl
Collecting importlib-metadata>=0.12; python_version < "3.8" (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/f6/d2/40b3fa882147719744e6aa50ac39cf7a22a913cbcba86a0371176c425a3b/importlib_metadata-0.23-py2.py3-none-any.whl
Collecting more-itertools>=4.0.0 (from pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/45/dc/3241eef99eb45f1def35cf93af35d1cf9ef4c0991792583b8f33ea41b092/more_itertools-7.2.0-py3-none-any.whl
Collecting six (from packaging->pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting pyparsing>=2.0.2 (from packaging->pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/11/fa/0160cd525c62d7abd076a070ff02b2b94de589f1a9789774f17d7c54058e/pyparsing-2.4.2-py2.py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata>=0.12; python_version < "3.8"->pytest>=3.5.1)
Using cached https://files.pythonhosted.org/packages/74/3d/1ee25a26411ba0401b43c6376d2316a71addcc72ef8690b101b4ea56d76a/zipp-0.6.0-py2.py3-none-any.whl
Installing collected packages: cxxfilt, tqdm, numpy, PyYAML, atomicwrites, attrs, six, pyparsing, packaging, more-itertools, zipp, importlib-metadata, pluggy, wcwidth, py, pytest
Successfully installed PyYAML-5.1.2 atomicwrites-1.3.0 attrs-19.3.0 cxxfilt-0.2.0 importlib-metadata-0.23 more-itertools-7.2.0 numpy-1.17.3 packaging-19.2 pluggy-0.13.0 py-1.8.0 pyparsing-2.4.2 pytest-5.2.1 six-1.12.0 tqdm-4.36.1 wcwidth-0.1.7 zipp-0.6.0
error
Cleaning up...
Removing source in /tmp/pip-b5r41wi9-build
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-b5r41wi9-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --pyprof install --record /tmp/pip-fg1d3igz-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code -11 in /tmp/pip-b5r41wi9-build/
Exception information:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/usr/lib/python3/dist-packages/pip/commands/install.py", line 360, in run
prefix=options.prefix_path,
File "/usr/lib/python3/dist-packages/pip/req/req_set.py", line 784, in install
**kwargs
File "/usr/lib/python3/dist-packages/pip/req/req_install.py", line 878, in install
spinner=spinner,
File "/usr/lib/python3/dist-packages/pip/utils/__init__.py", line 725, in call_subprocess
% (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-b5r41wi9-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" --pyprof install --record /tmp/pip-fg1d3igz-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code -11 in /tmp/pip-b5r41wi9-build/
My system settings are: CUDA version 10.0.130 Pytorch version 1.1.0 Python 3.6 Pip 9.0.1
Has anyone faced the same problem?
@mcarilli I used the following command (for enabling PyProf)
pip install -v --no-cache-dir --global-option="--pyprof" --global-option="--cpp_ext" --global-option="--cuda_ext" ./
Nevertheless, when I use the same command without pyprof, I can install the software correctly. Is there any particular problem with Pyprof?
so, I have figured the problem in my case.
The problem exist in setup.py :
if "--pyprof" in sys.argv:
with open('requirements.txt') as f:
required_packages = f.read().splitlines()
pipmain(["install"] + required_packages)
try:
sys.argv.remove("--pyprof")
except:
pass
I just commented everything inside the if statement, and I installed all the requirements separately. I assume "pipmain" doesn't work correctly with the above mentioned configuration.
This is caused by having a pip version >= 19.3, this commit moved the function into its own module. So the new import would be from pip._internal.main import main as pipmain.
Generally its probably not a great idea to use things called _internal but I have no idea what the proper way to do this in setup.py would be.
Ok, i've opened a PR to fix it https://github.com/NVIDIA/apex/pull/594. It's hacky but Apex is intended as a way for people to experiment with mixed precision, my main effort right now is getting automatic mixed precision upstream (https://github.com/pytorch/pytorch/issues/25081), so I'm ok with it.
