pytorch3d
pytorch3d copied to clipboard
Published
20 hours ago •
facebookresearch
facebookresearch
Inability to install pytorch3d due to torch not found
🐛 Bugs / Unexpected behaviors
Versions
Python: 3.8.10 OS: Ubuntu 20.04.5 LTS Poetry: 1.6.1 pip: 23.2.1
Description
I am trying to set up a poetry environment with this library installed. I started off with a completely new, empty environment. Then I attempted to add the poetry3d package:
poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git
Instead of adding it to the poetry environment, I got the following error:
Cloning https://github.com/facebookresearch/pytorch3d.git at 'HEAD' to /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d
Stack trace:
1 ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:344 in _run
342│ try:
343│ if input_:
→ 344│ output: str = subprocess.run(
345│ cmd,
346│ stdout=subprocess.PIPE,
CalledProcessError
Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', '-']' returned non-zero exit status 1.
at /usr/lib/python3.8/subprocess.py:516 in run
512│ # We don't call process.wait() as .__exit__ does that for us.
513│ raise
514│ retcode = process.poll()
515│ if check and retcode:
→ 516│ raise CalledProcessError(retcode, process.args,
517│ output=stdout, stderr=stderr)
518│ return CompletedProcess(process.args, retcode, stdout, stderr)
519│
520│
The following error occurred when trying to handle this error:
Stack trace:
3 ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:601 in get_pep517_metadata
599│ *PEP517_META_BUILD_DEPS,
600│ )
→ 601│ venv.run(
602│ "python",
603│ "-",
2 ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
312│ def run(self, bin: str, *args: str, **kwargs: Any) -> str:
313│ cmd = self.get_command_from_bin(bin) + list(args)
→ 314│ return self._run(cmd, **kwargs)
315│
316│ def run_pip(self, *args: str, **kwargs: Any) -> str:
1 ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
94│ def _run(self, cmd: list[str], **kwargs: Any) -> str:
95│ kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
→ 96│ return super()._run(cmd, **kwargs)
97│
98│ def get_temp_environ(
EnvCommandError
Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1
Output:
Traceback (most recent call last):
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
yield
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
return set(get_requires(config_settings))
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
self._subprocess_runner(
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
self._hook_runner(cmd, cwd, extra_environ)
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 16, in <module>
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
return set(get_requires(config_settings))
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
raise BuildBackendException( # noqa: B904 # use raise from
build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Input:
import build
import build.env
import pyproject_hooks
source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
dest = '/tmp/tmpxdbpwgq0/dist'
with build.env.IsolatedEnvBuilder() as env:
builder = build.ProjectBuilder(
srcdir=source,
scripts_dir=env.scripts_dir,
python_executable=env.executable,
runner=pyproject_hooks.quiet_subprocess_runner,
)
env.install(builder.build_system_requires)
env.install(builder.get_requires_for_build('wheel'))
builder.metadata_path(dest)
at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
359│ output = subprocess.check_output(
360│ cmd, stderr=stderr, env=env, text=True, **kwargs
361│ )
362│ except CalledProcessError as e:
→ 363│ raise EnvCommandError(e, input=input_)
364│
365│ return output
366│
367│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
Stack trace:
2 ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:359 in _run
357│ output = ""
358│ else:
→ 359│ output = subprocess.check_output(
360│ cmd, stderr=stderr, env=env, text=True, **kwargs
361│ )
1 /usr/lib/python3.8/subprocess.py:415 in check_output
413│ kwargs['input'] = empty
414│
→ 415│ return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
416│ **kwargs).stdout
417│
CalledProcessError
Command '['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info']' returned non-zero exit status 1.
at /usr/lib/python3.8/subprocess.py:516 in run
512│ # We don't call process.wait() as .__exit__ does that for us.
513│ raise
514│ retcode = process.poll()
515│ if check and retcode:
→ 516│ raise CalledProcessError(retcode, process.args,
517│ output=stdout, stderr=stderr)
518│ return CompletedProcess(process.args, retcode, stdout, stderr)
519│
520│
The following error occurred when trying to handle this error:
Stack trace:
3 ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:622 in get_pep517_metadata
620│ os.chdir(path)
621│ try:
→ 622│ venv.run("python", "setup.py", "egg_info")
623│ info = PackageInfo.from_metadata(path)
624│ except EnvCommandError as fbe:
2 ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:314 in run
312│ def run(self, bin: str, *args: str, **kwargs: Any) -> str:
313│ cmd = self.get_command_from_bin(bin) + list(args)
→ 314│ return self._run(cmd, **kwargs)
315│
316│ def run_pip(self, *args: str, **kwargs: Any) -> str:
1 ~/.local/lib/python3.8/site-packages/poetry/utils/env/virtual_env.py:96 in _run
94│ def _run(self, cmd: list[str], **kwargs: Any) -> str:
95│ kwargs["env"] = self.get_temp_environ(environ=kwargs.get("env"))
→ 96│ return super()._run(cmd, **kwargs)
97│
98│ def get_temp_environ(
EnvCommandError
Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
Output:
Traceback (most recent call last):
File "setup.py", line 15, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
at ~/.local/lib/python3.8/site-packages/poetry/utils/env/base_env.py:363 in _run
359│ output = subprocess.check_output(
360│ cmd, stderr=stderr, env=env, text=True, **kwargs
361│ )
362│ except CalledProcessError as e:
→ 363│ raise EnvCommandError(e, input=input_)
364│
365│ return output
366│
367│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
The following error occurred when trying to handle this error:
Stack trace:
18 ~/.local/lib/python3.8/site-packages/cleo/application.py:327 in run
325│
326│ try:
→ 327│ exit_code = self._run(io)
328│ except BrokenPipeError:
329│ # If we are piped to another process, it may close early and send a
17 ~/.local/lib/python3.8/site-packages/poetry/console/application.py:190 in _run
188│ self._load_plugins(io)
189│
→ 190│ exit_code: int = super()._run(io)
191│ return exit_code
192│
16 ~/.local/lib/python3.8/site-packages/cleo/application.py:431 in _run
429│ io.input.interactive(interactive)
430│
→ 431│ exit_code = self._run_command(command, io)
432│ self._running_command = None
433│
15 ~/.local/lib/python3.8/site-packages/cleo/application.py:473 in _run_command
471│
472│ if error is not None:
→ 473│ raise error
474│
475│ return terminate_event.exit_code
14 ~/.local/lib/python3.8/site-packages/cleo/application.py:457 in _run_command
455│
456│ if command_event.command_should_run():
→ 457│ exit_code = command.run(io)
458│ else:
459│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
13 ~/.local/lib/python3.8/site-packages/cleo/commands/base_command.py:119 in run
117│ io.input.validate()
118│
→ 119│ status_code = self.execute(io)
120│
121│ if status_code is None:
12 ~/.local/lib/python3.8/site-packages/cleo/commands/command.py:62 in execute
60│
61│ try:
→ 62│ return self.handle()
63│ except KeyboardInterrupt:
64│ return 1
11 ~/.local/lib/python3.8/site-packages/poetry/console/commands/add.py:160 in handle
158│ return 0
159│
→ 160│ requirements = self._determine_requirements(
161│ packages,
162│ allow_prereleases=self.option("allow-prereleases"),
10 ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:375 in _determine_requirements
373│
374│ result = []
→ 375│ for requirement in self._parse_requirements(requires):
376│ if "git" in requirement or "url" in requirement or "path" in requirement:
377│ result.append(requirement)
9 ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in _parse_requirements
439│ cwd=cwd,
440│ )
→ 441│ return [parser.parse(requirement) for requirement in requirements]
442│
443│ def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:
8 ~/.local/lib/python3.8/site-packages/poetry/console/commands/init.py:441 in <listcomp>
439│ cwd=cwd,
440│ )
→ 441│ return [parser.parse(requirement) for requirement in requirements]
442│
443│ def _format_requirements(self, requirements: list[dict[str, str]]) -> Requirements:
7 ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:89 in parse
87│
88│ specification = (
→ 89│ self._parse_url(requirement)
90│ or self._parse_path(requirement)
91│ or self._parse_simple(requirement)
6 ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:149 in _parse_url
147│
148│ if url_parsed.scheme in GIT_URL_SCHEMES:
→ 149│ return self._parse_git_url(requirement)
150│
151│ if url_parsed.scheme in ["http", "https"]:
5 ~/.local/lib/python3.8/site-packages/poetry/utils/dependency_specification.py:133 in _parse_git_url
131│
132│ source_root = self._env.path.joinpath("src") if self._env else None
→ 133│ package = self._direct_origin.get_package_from_vcs(
134│ "git",
135│ url=url.url,
4 ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:111 in get_package_from_vcs
109│ raise ValueError(f"Unsupported VCS dependency {vcs}")
110│
→ 111│ return _get_package_from_git(
112│ url=url,
113│ branch=branch,
3 ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:46 in _get_package_from_git
44│ path = path.joinpath(subdirectory)
45│
→ 46│ package = DirectOrigin.get_package_from_directory(path)
47│ package._source_type = "git"
48│ package._source_url = url
2 ~/.local/lib/python3.8/site-packages/poetry/packages/direct_origin.py:75 in get_package_from_directory
73│ @classmethod
74│ def get_package_from_directory(cls, directory: Path) -> Package:
→ 75│ return PackageInfo.from_directory(path=directory).to_package(root_dir=directory)
76│
77│ def get_package_from_url(self, url: str) -> Package:
1 ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:499 in from_directory
497│ info = cls.from_setup_files(path)
498│ else:
→ 499│ info = get_pep517_metadata(path)
500│ except PackageInfoError:
501│ if not info:
PackageInfoError
Unable to determine package info for path: /home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d
Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', '-'] errored with the following return code 1
Output:
Traceback (most recent call last):
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 466, in _handle_backend
yield
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
return set(get_requires(config_settings))
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
return self._call_hook('get_requires_for_build_wheel', {
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 311, in _call_hook
self._subprocess_runner(
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 302, in _runner
self._hook_runner(cmd, cwd, extra_environ)
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_impl.py", line 71, in quiet_subprocess_runner
check_output(cmd, cwd=cwd, env=env, stderr=STDOUT)
File "/usr/lib/python3.8/subprocess.py", line 415, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/usr/lib/python3.8/subprocess.py", line 516, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/build-env-8vw3hcw7/bin/python', '/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/pyproject_hooks/_in_process/_in_process.py', 'get_requires_for_build_wheel', '/tmp/tmp9cn0rpj_']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<stdin>", line 16, in <module>
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 356, in get_requires_for_build
return set(get_requires(config_settings))
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/tmp/tmpxdbpwgq0/.venv/lib/python3.8/site-packages/build/__init__.py", line 474, in _handle_backend
raise BuildBackendException( # noqa: B904 # use raise from
build.BuildBackendException: Backend subprocess exited when trying to invoke get_requires_for_build_wheel
Input:
import build
import build.env
import pyproject_hooks
source = '/home/myfolder/poetry_env/poetry_env_demo/.venv/src/pytorch3d'
dest = '/tmp/tmpxdbpwgq0/dist'
with build.env.IsolatedEnvBuilder() as env:
builder = build.ProjectBuilder(
srcdir=source,
scripts_dir=env.scripts_dir,
python_executable=env.executable,
runner=pyproject_hooks.quiet_subprocess_runner,
)
env.install(builder.build_system_requires)
env.install(builder.get_requires_for_build('wheel'))
builder.metadata_path(dest)
Fallback egg_info generation failed.
Command ['/tmp/tmpxdbpwgq0/.venv/bin/python', 'setup.py', 'egg_info'] errored with the following return code 1
Output:
Traceback (most recent call last):
File "setup.py", line 15, in <module>
import torch
ModuleNotFoundError: No module named 'torch'
at ~/.local/lib/python3.8/site-packages/poetry/inspection/info.py:625 in get_pep517_metadata
621│ try:
622│ venv.run("python", "setup.py", "egg_info")
623│ info = PackageInfo.from_metadata(path)
624│ except EnvCommandError as fbe:
→ 625│ raise PackageInfoError(
626│ path, e, "Fallback egg_info generation failed.", fbe
627│ )
628│ finally:
629│ os.chdir(cwd)
Alright, so pytorch3d has a dependency on torch but despite this pre-requisite, the installation for some reason doesn't install the dependency. So I went ahead and installed torch:
poetry add torch
Which went successfully. Now I retried the previous installation command, knowing that torch is certainly included in my environment by now:
poetry add -vvv git+https://github.com/facebookresearch/pytorch3d.git
But the error is exactly the same.
Edit 1
I saw this issue from earlier: https://github.com/facebookresearch/pytorch3d/pull/1619. But it doesn't offer a clear workaround on how to install pytorch3d with poetry. As I mentioned above, I also installed torch separately before attempting to install pytorch3d, but the installation still fails, as pytorch3d somehow doesn't see the already present torch dependency in the local venv.
Edit 2
I also came across this: https://github.com/facebookresearch/pytorch3d/pull/1479 . I assume this change would solve the issue, but it seems to have been never merged for some reason.
Meanwhile, the pip install works fine:
pip install git+https://github.com/facebookresearch/pytorch3d.git
pip install git+https://github.com/facebookresearch/pytorch3d.git
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/facebookresearch/pytorch3d.git
Cloning https://github.com/facebookresearch/pytorch3d.git to /tmp/pip-req-build-4m0519uo
Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/pytorch3d.git /tmp/pip-req-build-4m0519uo
Resolved https://github.com/facebookresearch/pytorch3d.git to commit a8c70161a1c99c1878ae7cf312cf7907a84f01b0
Preparing metadata (setup.py) ... done
Requirement already satisfied: fvcore in /home/myfolder/.local/lib/python3.8/site-packages (from pytorch3d==0.7.4) (0.1.5.post20221221)
Requirement already satisfied: iopath in /home/myfolder/.local/lib/python3.8/site-packages (from pytorch3d==0.7.4) (0.1.10)
Requirement already satisfied: Pillow in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (10.1.0)
Requirement already satisfied: numpy in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (1.24.4)
Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from fvcore->pytorch3d==0.7.4) (5.3.1)
Requirement already satisfied: tabulate in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (0.9.0)
Requirement already satisfied: termcolor>=1.1 in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (2.3.0)
Requirement already satisfied: tqdm in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (4.66.1)
Requirement already satisfied: yacs>=0.1.6 in /home/myfolder/.local/lib/python3.8/site-packages (from fvcore->pytorch3d==0.7.4) (0.1.8)
Requirement already satisfied: portalocker in /home/myfolder/.local/lib/python3.8/site-packages (from iopath->pytorch3d==0.7.4) (2.8.2)
Requirement already satisfied: typing-extensions in /home/myfolder/.local/lib/python3.8/site-packages (from iopath->pytorch3d==0.7.4) (4.8.0)
Building wheels for collected packages: pytorch3d
Building wheel for pytorch3d (setup.py) ... done
Created wheel for pytorch3d: filename=pytorch3d-0.7.4-cp38-cp38-linux_x86_64.whl size=63584603 sha256=fd2a5cc4e9308aadb7d0b10626d1fbad5d766f29c06afb2fb6170a9301ab03fa
Stored in directory: /tmp/pip-ephem-wheel-cache-cad_9osc/wheels/5e/a7/7c/814c3888eaaecaaf0f533faeb3c6812d82274484be51f7441c
Successfully built pytorch3d
DEPRECATION: distro-info 0.23ubuntu1 has a non-standard version number. pip 23.3 will enforce this behaviour change. A possible replacement is to upgrade to a newer version of distro-info or contact the author to suggest that they release a version with a conforming version number. Discussion can be found at https://github.com/pypa/pip/issues/12063
Installing collected packages: pytorch3d
Successfully installed pytorch3d-0.7.4
The problem is, I want to use poetry, not pip, and in that case, I am running into the aforementioned ModuleNotFound error
I think poetry might be deliberately not (yet) designed to deal with packages like PyTorch3D which have tight build time dependencies on versions of other packages. So it might be "by design" that this doesn't work easily. If that's not the case, we'd like help, e.g. pointers to documentation. If it is possible to adapt PyTorch3D nicely into poetry, it hasn't been done.
Thank you for your reply @bottler .
It seems that this issue is essentially a duplicate of https://github.com/facebookresearch/pytorch3d/issues/1419, as torch is a build-time dependency of the library, but it is not declared in a pyproject.toml file, hence poetry has no way to include it in build time. As such, it seems to be not possible to install this package with poetry :/