"subdirectory" argument ignored

[oyvinev]

• Poetry version: 1.4.0
• Python version: 3.8-10
• OS version and name: Ubuntu 20.04
• pyproject.toml: https://gist.github.com/oyvinev/4df478d8dee2fbdce3128043d4666468

• [x] I am on the latest stable Poetry version, installed using a recommended method.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] I have consulted the FAQ and blog for any relevant entries or release notes.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Using virtualenv: /home/oyvinev/.cache/pypoetry/virtualenvs/gpbuilder-rFATLHJD-py3.8
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 1 install, 0 updates, 0 removals, 1 skipped

  • Installing numpy (1.16.6): Pending...
  • Installing numpy (1.16.6): Skipped for the following reason: Already installed
  • Installing pyd4 (0.3.6.2 39001f4): Pending...
  • Installing pyd4 (0.3.6.2 39001f4): Cloning...
  • Installing pyd4 (0.3.6.2 39001f4): Building...
  • Installing pyd4 (0.3.6.2 39001f4): Preparing...
  • Installing pyd4 (0.3.6.2 39001f4): Failed

  Stack trace:

  10  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:271 in _execute_operation
       269│ 
       270│             try:
     → 271│                 result = self._do_execute_operation(operation)
       272│             except EnvCommandError as e:
       273│                 if e.e.returncode == -2:

   9  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:357 in _do_execute_operation
       355│             return 0
       356│ 
     → 357│         result: int = getattr(self, f"_execute_{method}")(operation)
       358│ 
       359│         if result != 0:

   8  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:477 in _execute_install
       475│ 
       476│     def _execute_install(self, operation: Install | Update) -> int:
     → 477│         status_code = self._install(operation)
       478│ 
       479│         self._save_url_reference(operation)

   7  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:504 in _install
       502│         cleanup_archive: bool = False
       503│         if package.source_type == "git":
     → 504│             archive = self._prepare_git_archive(operation)
       505│             cleanup_archive = True
       506│         elif package.source_type == "file":

   6  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:622 in _prepare_git_archive
       620│         package._source_url = str(source.path)
       621│ 
     → 622│         archive = self._prepare_directory_archive(operation)
       623│ 
       624│         package._source_url = original_url

   5  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:598 in _prepare_directory_archive
       596│             req /= package.source_subdirectory
       597│ 
     → 598│         return self._prepare_archive(operation)
       599│ 
       600│     def _prepare_git_archive(self, operation: Install | Update) -> Path:

   4  ~/.local/lib/python3.8/site-packages/poetry/installation/executor.py:577 in _prepare_archive
       575│         self._populate_hashes_dict(archive, package)
       576│ 
     → 577│         return self._chef.prepare(archive, editable=package.develop)
       578│ 
       579│     def _prepare_directory_archive(self, operation: Install | Update) -> Path:

   3  ~/.local/lib/python3.8/site-packages/poetry/installation/chef.py:105 in prepare
       103│             tmp_dir = tempfile.mkdtemp(prefix="poetry-chef-")
       104│ 
     → 105│             return self._prepare(archive, Path(tmp_dir), editable=editable)
       106│ 
       107│         return self._prepare_sdist(archive, destination=output_dir)

   2  ~/.local/lib/python3.8/site-packages/poetry/installation/chef.py:116 in _prepare
       114│         with ephemeral_environment(self._env.python) as venv:
       115│             env = IsolatedEnv(venv, self._pool)
     → 116│             builder = ProjectBuilder(
       117│                 directory,
       118│                 python_executable=env.executable,

   1  ~/.local/lib/python3.8/site-packages/build/__init__.py:267 in __init__
       265│         """
       266│         self._srcdir: str = os.path.abspath(srcdir)
     → 267│         _validate_source_directory(srcdir)
       268│ 
       269│         spec_file = os.path.join(srcdir, 'pyproject.toml')

  BuildException

  Source /home/oyvinev/.cache/pypoetry/virtualenvs/gpbuilder-rFATLHJD-py3.8/src/d4-format does not appear to be a Python project: no pyproject.toml or setup.py

  at ~/.local/lib/python3.8/site-packages/build/__init__.py:135 in _validate_source_directory
      131│         raise BuildException(f'Source {srcdir} is not a directory')
      132│     pyproject_toml = os.path.join(srcdir, 'pyproject.toml')
      133│     setup_py = os.path.join(srcdir, 'setup.py')
      134│     if not os.path.exists(pyproject_toml) and not os.path.exists(setup_py):
    → 135│         raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
      136│ 
      137│ 
      138│ def check_dependency(
      139│     req_string: str, ancestral_req_strings: tuple[str, ...] = (), parent_extras: Set[str] = frozenset()

Issue

"subdirectory" seems to be ignored (at least in this specific case). I've verified that it builds on 1.3.2, but fails with 1.4.0.

Expected behaviour: It should enter the subdirectory pyd4 under src/d4-format

[simon-sk]

I am experiencing the same issue with poetry 1.4.0 and get the following error message:

  ChefBuildError

  Backend subprocess exited when trying to invoke get_requires_for_build_wheel
  
  error: Multiple top-level packages discovered in a flat-layout: ['module1', 'module2', 'module3'].

My repository has the following structure:

- module1
  - pyproject.toml
  - module1
     - __init__.py
     - ...
- module2
  - pyproject.toml
  - module2
     - __init__.py
     - ...
- ...

[nrser]

@oyvinev I'm seeing the same issue, able to work around it by setting experimental.new-installer to false. You can do this for just the project in question by running

poetry config --local experimental.new-installer false

in the project directory (or edit the poetry.toml file directly). If you want to do it globally just omit the --local flag.

[nrser]

@simon-sk not sure if your issue is the same thing but you could try that setting and see if it works?

[harrymander]

I am also seeing this with v1.4.0, Python 3.10.6, Ubuntu 22.04.2:

pyproject.toml:

[tool.poetry]
name = "package-name"
version = "0.2.0"
description = ""
authors = ["Harry Mander <[email protected]>"]
readme = "README.md"
packages = [{include = "package_name"}]

[tool.poetry.dependencies]
python = ">=3.10, <3.11"
click = "^8.1.3"
tqdm = "^4.64.1"
tomli = "^2.0.1"
Jinja2 = "^3.1.2"

[tool.poetry.dependencies.git-dependency-1]
git = "git@<git-repo-1>"
rev = "<git-ref>"

[tool.poetry.dependencies.git-dependency-2]
git = "git@<git-repo-2>"
rev = "<git-ref>"
subdirectory = "path/to/project"

[tool.poetry.scripts]
package-name = "package_name.cli:cli"

[tool.poetry.group.dev.dependencies]
flake8 = "^6.0.0"
mypy = "^0.991"
types-tqdm = "^4.64.7.11"
ipython = "^8.9.0"
rich = "^13.3.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The issue occurs with git-dependency-2 which is contained in a subdirectory of its repo. On poetry install:

Installing dependencies from lock file

Package operations: 0 installs, 1 update, 0 removals

  • Updating git-dependency-2 (0.4.1 ff58332 -> 0.5.0 7c527d4): Failed

  BuildException

  Source /home/harry/.cache/pypoetry/virtualenvs/validation-study-z_l8ILkD-py3.10/src/git-repo-2 does not appear to be a Python project: no pyproject.toml or setup.py

  at ~/.local/share/pypoetry/venv/lib/python3.10/site-packages/build/__init__.py:135 in _validate_source_directory
      131│         raise BuildException(f'Source {srcdir} is not a directory')
      132│     pyproject_toml = os.path.join(srcdir, 'pyproject.toml')
      133│     setup_py = os.path.join(srcdir, 'setup.py')
      134│     if not os.path.exists(pyproject_toml) and not os.path.exists(setup_py):
    → 135│         raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
      136│ 
      137│ 
      138│ def check_dependency(
      139│     req_string: str, ancestral_req_strings: tuple[str, ...] = (), parent_extras: Set[str] = frozenset()

In the above the path /home/harry/.cache/pypoetry/virtualenvs/validation-study-z_l8ILkD-py3.10/src/git-repo-2 is the path of the top-level repo, not the subdirectory which contains the pyproject.toml.

Setting

poetry config --local experimental.new-installer false

Works as a temporary solution.

[radoering]

I don't recommend setting experimental.new-installer to false. (The name is misleading.) It should be sufficient to set installer.modern-installation to false.

A possible fix is mentioned in https://github.com/python-poetry/poetry/issues/755#issuecomment-1449275865. A PR with a fix and a test is welcome.

[nrser]

It should be sufficient to set installer.modern-installation to false

I tried that first but it didn't work. In my case it's a git dependency, which seems like it makes a difference from looking at the source.

Check it out:

$ poetry config --list
cache-dir = "/Users/nrser/Library/Caches/pypoetry"
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = false
installer.no-binary = null
installer.parallel = true
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /Users/nrser/Library/Caches/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

$ poetry install -vv
Using virtualenv: /Users/nrser/src/github.com/nrser/clavier/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 0 installs, 4 updates, 0 removals, 132 skipped

  • Updating docspec (2.0.2 fa3c53f -> 2.0.2)
  • Updating docspec-python (2.0.2+namespace.packages /Users/nrser/src/github.com/nrser/clavier/.venv/src/docspec/docspec-python -> 2.0.2+namespace-packages fa3c53f): Failed

  Stack trace:

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:271 in _execute_operation
        result = self._do_execute_operation(operation)

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:357 in _do_execute_operation
        result: int = getattr(self, f"_execute_{method}")(operation)

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:484 in _execute_update
        status_code = self._update(operation)

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:543 in _update
        return self._install(operation)

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:504 in _install
        archive = self._prepare_git_archive(operation)

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:622 in _prepare_git_archive
        archive = self._prepare_directory_archive(operation)

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:598 in _prepare_directory_archive
        return self._prepare_archive(operation)

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/executor.py:577 in _prepare_archive
        return self._chef.prepare(archive, editable=package.develop)

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:105 in prepare
        return self._prepare(archive, Path(tmp_dir), editable=editable)

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/chef.py:116 in _prepare
        builder = ProjectBuilder(

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/build/__init__.py:267 in __init__
        _validate_source_directory(srcdir)

  BuildException

  Source /Users/nrser/src/github.com/nrser/clavier/.venv/src/docspec does not appear to be a Python project: no pyproject.toml or setup.py

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/build/__init__.py:135 in _validate_source_directory
      131│         raise BuildException(f'Source {srcdir} is not a directory')
      132│     pyproject_toml = os.path.join(srcdir, 'pyproject.toml')
      133│     setup_py = os.path.join(srcdir, 'setup.py')
      134│     if not os.path.exists(pyproject_toml) and not os.path.exists(setup_py):
    → 135│         raise BuildException(f'Source {srcdir} does not appear to be a Python project: no pyproject.toml or setup.py')
      136│
      137│
      138│ def check_dependency(
      139│     req_string: str, ancestral_req_strings: tuple[str, ...] = (), parent_extras: Set[str] = frozenset()

[harrymander]

Yes I also found installer.modern-installation didn't work. My PR above fixes the issue for me with default settings on v1.4.0 - perhaps you could test that too?

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.