Poetry install fails when using a github dependency with subdirectory AND in-project virtual environments.

[mattcoulter7]

Description

Poetry install fails when using a github dependency with subdirectory AND in-project virtual environments.

I have confirmed that this issue happens on Poetry 1.6, 1.7 and 1.8.5. This happens regardless of Github Repository Visibility (can be private or public).

Steps to reproduce:

Creating Package A in a Github Repo Subdirectory.

1. Create a Github repository monorepo, and clone it.
2. Navigate to your local monorepo-a directory.
3. Create a folder in the root directory called package-a, the cd into it.
4. Create pyproject.toml with the following contents

[tool.poetry]
name = "package-a"
version = "0.1.0"
description = ""
authors = ["Matthew Coulter <[email protected]>"]
package-mode = false

[tool.poetry.dependencies]
python = "^3.12"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

5. Commit your changes, and publish a released tagged with 0.1.0.

Creating Package B which depends on Package A via git subdirectory.

6. Create a new folder called package-b, the cd into it.
7. Create pyproject.toml with the following contents:

[tool.poetry]
name = "package-b"
version = "0.1.0"
description = ""
authors = ["Matthew Coulter <[email protected]>"]
package-mode = false

[tool.poetry.dependencies]
python = "^3.12"
package-a = { git = "https://github.com/mattcoulter7/monorepo-a.git", tag = "0.1.0", subdirectory = "package-a" }

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

8. Create poetry.toml with the following contents:

[virtualenvs]
path = "."
in-project = false

9. Run poetry install inside of package-b
10. You will encounter the same FileNotFoundError due to an incorrect directory. Error Log:

PS F:\Github\monorepo-a\package-b> poetry install
Creating virtualenv package-b-muMprWRZ-py3.12 in .
Updating dependencies
Resolving dependencies... (3.0s)

Package operations: 1 install, 0 updates, 0 removals

  - Installing package-a (0.1.0 fbdbad7): Failed                                                                                                                              

  FileNotFoundError

  [Errno 2] No such file or directory: 'package-b-muMprWRZ-py3.12\\src\\monorepo-a\\package-a\\package-b-muMprWRZ-py3.12\\src\\monorepo-a\\package-a'

  at ~\AppData\Local\Programs\Python\Python312\Lib\gzip.py:192 in __init__
      188│             raise ValueError("Invalid mode: {!r}".format(mode))
      189│         if mode and 'b' not in mode:
      190│             mode += 'b'
      191│         if fileobj is None:
    → 192│             fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
      193│         if filename is None:
      194│             filename = getattr(fileobj, 'name', '')
      195│             if not isinstance(filename, (str, bytes)):
      196│                 filename = ''

Cannot install package-a.

Workarounds

The issue can be bypassed through the following:

1. Navigate to package-b.
2. Ensure poetry.lock is deleted.
3. Ensure any in-project virtual environments are deleted (they should look like package-b-...-py3.12).
4. Delete the poetry.toml file.
5. Run poetry install, this will succeed.
6. Restore the poetry.toml file.
7. Run poetry install again, and notice that it now succeeds with an in-project virtual environment now exists.

It seems that something in the poetry cache enables it to succeed after it has previously succeeded with the out of project directory virtual environment.

Poetry Installation Method

pip

Operating System

Microsoft Windows 11 Home Version [10.0.22621 Build 22621]

Poetry Version

Poetry (version 1.8.5)

Poetry Configuration

cache-dir = "F:\\poetry_cache"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "."
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

Platform: "win-amd64"
Python version: "3.12"
Current installation scheme: "nt"

Paths: 
	data = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	include = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Include"
	platinclude = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Include"
	platlib = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib\site-packages"
	platstdlib = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib"
	purelib = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib\site-packages"
	scripts = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Scripts"
	stdlib = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib"

Variables: 
	BINDIR = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	BINLIBDEST = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib"
	EXE = ".exe"
	EXT_SUFFIX = ".cp312-win_amd64.pyd"
	INCLUDEPY = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Include"
	LIBDEST = "C:\Users\matt\AppData\Local\Programs\Python\Python312\Lib"
	TZPATH = ""
	VERSION = "312"
	VPATH = "..\.."
	abiflags = ""
	base = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	exec_prefix = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	installed_base = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	installed_platbase = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	platbase = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	platlibdir = "DLLs"
	prefix = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	projectbase = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	py_version = "3.12.5"
	py_version_nodot = "312"
	py_version_nodot_plat = "312"
	py_version_short = "3.12"
	srcdir = "C:\Users\matt\AppData\Local\Programs\Python\Python312"
	userbase = "C:\Users\matt\AppData\Roaming\Python"

Example pyproject.toml

See above steps to reproduce

Poetry Runtime Logs

Loading configuration file C:\Users\matt\AppData\Roaming\pypoetry\config.toml
Loading configuration file F:\Github\monorepo-a\package-b\poetry.toml
Using virtualenv: package-b-muMprWRZ-py3.12
Updating dependencies
Resolving dependencies...
   1: fact: package-b is 0.1.0
   1: derived: package-b
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[win32ctypes:win32ctypes.core.ctypes] Loaded ctypes backend
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'Windows WinVaultKeyring'
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /mattcoulter7/monorepo-a.git/info/refs?service=git-upload-pack HTTP/11" 200 None
Cloning https://github.com/mattcoulter7/monorepo-a.git at '0.1.0' to package-b-muMprWRZ-py3.12\src\monorepo-a
   1: fact: package-b depends on package-a (0.1.0)
   1: selecting package-b (0.1.0)
   1: derived: package-a (0.1.0) @ git+https://github.com/mattcoulter7/[email protected]#subdirectory=package-a
   1: selecting package-a (0.1.0 fbdbad7)
   1: Version solving took 1.010 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals

  - Installing package-a (0.1.0 fbdbad7)

  Stack trace:

  11  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:285 in _execute_operation
       283| 
       284|             try:
     > 285|                 result = self._do_execute_operation(operation)
       286|             except EnvCommandError as e:
       287|                 if e.e.returncode == -2:

  10  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:395 in _do_execute_operation
       393|             return 0
       394| 
     > 395|         result: int = getattr(self, f"_execute_{method}")(operation)
       396| 
       397|         if result != 0:

   9  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:520 in _execute_install
       518| 
       519|     def _execute_install(self, operation: Install | Update) -> int:
     > 520|         status_code = self._install(operation)
       521| 
       522|         self._save_url_reference(operation)

   8  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:547 in _install
       545|         cleanup_archive: bool = False
       546|         if package.source_type == "git":
     > 547|             archive = self._prepare_git_archive(operation)
       548|             cleanup_archive = operation.package.develop
       549|         elif package.source_type == "file":

   7  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:671 in _prepare_git_archive
       669|             )
       670| 
     > 671|         archive = self._prepare_archive(operation, output_dir=output_dir)
       672|         if not package.develop:
       673|             package._source_url = original_url

   6  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\executor.py:624 in _prepare_archive
       622|         self._populate_hashes_dict(archive, package)
       623| 
     > 624|         return self._chef.prepare(
       625|             archive, editable=package.develop, output_dir=output_dir
       626|         )

   5  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\chef.py:123 in prepare
       121|             return self._prepare(archive, destination=destination, editable=editable)
       122| 
     > 123|         return self._prepare_sdist(archive, destination=output_dir)
       124| 
       125|     def _prepare(

   4  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\installation\chef.py:176 in _prepare_sdist
       174|         with temporary_directory() as tmp_dir:
       175|             archive_dir = Path(tmp_dir)
     > 176|             extractall(source=archive, dest=archive_dir, zip=zip)
       177| 
       178|             elements = list(archive_dir.glob("*"))

   3  ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\poetry\utils\helpers.py:366 in extractall
       364|         # https://github.com/python/cpython/issues/107845.
       365|         broken_tarfile_filter = {(3, 8, 17), (3, 9, 17), (3, 10, 12), (3, 11, 4)}
     > 366|         with tarfile.open(source) as archive:
       367|             if (
       368|                 hasattr(tarfile, "data_filter")

   2  ~\AppData\Local\Programs\Python\Python312\Lib\tarfile.py:1802 in open
       1800|                     saved_pos = fileobj.tell()
       1801|                 try:
     > 1802|                     return func(name, "r", fileobj, **kwargs)
       1803|                 except (ReadError, CompressionError) as e:
       1804|                     error_msgs.append(f'- method {comptype}: {e!r}')

   1  ~\AppData\Local\Programs\Python\Python312\Lib\tarfile.py:1870 in gzopen
       1868| 
       1869|         try:
     > 1870|             fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
       1871|         except OSError as e:
       1872|             if fileobj is not None and mode == 'r':

  FileNotFoundError

  [Errno 2] No such file or directory: 'package-b-muMprWRZ-py3.12\\src\\monorepo-a\\package-a\\package-b-muMprWRZ-py3.12\\src\\monorepo-a\\package-a'

  at ~\AppData\Local\Programs\Python\Python312\Lib\gzip.py:192 in __init__
      188|             raise ValueError("Invalid mode: {!r}".format(mode))
      189|         if mode and 'b' not in mode:
      190|             mode += 'b'
      191|         if fileobj is None:
    > 192|             fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
      193|         if filename is None:
      194|             filename = getattr(fileobj, 'name', '')
      195|             if not isinstance(filename, (str, bytes)):
      196|                 filename = ''

Cannot install package-a.