poetry never resolves package combination: google-cloud-aiplatform kfp

[kler]

• Poetry version: 1.6.1

• Python version: 3.11.5

• OS version and name: docker image python:3.11.5

• pyproject.toml: Providing reproducible docker command instead

• [X] I am on the latest stable Poetry version, installed using a recommended method.

• [X] I have searched the issues of this repo and believe that this is not a duplicate.

• [X] I have consulted the FAQ and blog for any relevant entries or release notes.

• [X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The following combination of packages never resolves: google-cloud-aiplatform kfp

Poetry is stuck in Resolving dependencies... after >10h.

Reproduced in both Native Macos 13.5.1 (22G90) and in Docker.

In Macos envs, cache and poetry.lock was removed like this, but despite this the error persists:

poetry env remove --all \
&& poetry cache clear --all -q . \
&& rm -rf $(poetry config cache-dir)/artifacts \
&& rm -f poetry.lock

Tested package combinations

❌ Doesn't resolve (within reasonable time)

• google-cloud-aiplatform zenml mlflow gcsfs kfp <-- This combo was left running >10h
• google-cloud-aiplatform kfp

✅ Does resolve

• google-cloud-aiplatform zenml
• google-cloud-aiplatform zenml mlflow
• google-cloud-aiplatform zenml mlflow gcsfs
• google-cloud-aiplatform gcsfs

How to reproduce

docker run --rm python:3.11.5 sh -c " \
curl -sSL https://install.python-poetry.org | python3 - --version=1.6.1 \
&& poetry --version \
&& mkdir app && cd app \
&& /root/.local/bin/poetry init --name=app --description=app --author=test@localhost --python='>=3.10,<3.12' -n \
&& /root/.local/bin/poetry add google-cloud-aiplatform kfp"

Output

Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.6.1)
Installing Poetry (1.6.1): Creating environment
Installing Poetry (1.6.1): Installing Poetry
Installing Poetry (1.6.1): Creating script
Installing Poetry (1.6.1): Done

Poetry (1.6.1) is installed now. Great!

To get started you need Poetry's bin directory (/root/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/root/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/root/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Creating virtualenv app-9TtSrW0h-py3.11 in /root/.cache/pypoetry/virtualenvs
Using version ^1.33.1 for google-cloud-aiplatform
Using version ^2.3.0 for kfp

Updating dependencies
Resolving dependencies...

[Davidnet]

Still present in 1.7.0

[Davidnet]

Now at Nov29, and poetry 1.7.1 the packages:

$poetry add google-cloud-pipeline-components kfp

It fails with:

The following error occurred when trying to handle this error:


  OverrideNeeded

  {Package('google-api-core', '2.14.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-cloud-bigquery', '3.13.0'): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.4'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-api-core', '2.13.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.13.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.12.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.3'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.2'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}}

  at ~/.local/share/pypoetry/venv/lib/python3.11/site-packages/poetry/puzzle/provider.py:660 in complete_package
      656│                     current_overrides.update({dependency_package: package_overrides})
      657│                     overrides.append(current_overrides)
      658│ 
      659│             if overrides:
    → 660│                 raise OverrideNeeded(*overrides)
      661│ 
      662│         # Modifying dependencies as needed
      663│         clean_dependencies = []
      664│         for dep in dependencies:

The following error occurred when trying to handle this error:


  AssertionError
  

[Davidnet]

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <[email protected]>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[HenriqueAJNB]

I have the same problem:

• Poetry version: 1.7.1
• Python version: 3.10.12
• OS version and name: Ubuntu 22.04.3 LTS
• pyproject.toml

[tool.poetry]
name = "my-project"
version = "0.0.1"
description = ""
authors = []
readme = "README.md"
packages = [{ include = "my_pkg" }]

[tool.poetry.dependencies]
python = ">=3.10,<3.12"
kfp = "1.8.22"
google-cloud-aiplatform = "1.25.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Issue

To reproduce the issue using the same environment, just run poetry install

It will fall into infinite dependency resolving.

$ poetry install
Creating virtualenv my-project in /home/.../.venv
Updating dependencies
Resolving dependencies... (2003.3s)

The logs using -vvv option shows what is the package that is causing the problem:

Found 0 compatible packages for grpcio-status
   1: fact: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: conflict: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.0rc1,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.51.0rc1)
   1: ! which is caused by "grpcio-status (1.51.0rc1) depends on protobuf (>=4.21.6)"
   1: ! thus: grpcio-status (>1.49.1,<1.50.0 || >1.50.0,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0) requires protobuf (>=4.21.6)
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.52.0rc1)
   1: ! which is caused by "grpcio-status (1.52.0rc1) depends on protobuf (>=4.21.6)"
...
   1: conflict: grpcio-status (1.56.2) depends on protobuf (>=4.21.6)
...
   1: conflict: grpcio-status (1.60.0) depends on protobuf (>=4.21.6)
...

Tons of those conflits appears, but no error or exception is raised.

The pip way

But, when I use built-in package venv instead of poetry it runs smootly.

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip3 install kfp==1.8.22 google-cloud-aiplatform==1.25.0

And the most important, pip can found a version for grpcio-status...

(.venv) $ pip3 show grpcio-status
Name: grpcio-status
Version: 1.48.2
Summary: Status proto mapping for gRPC
Home-page: https://grpc.io
Author: The gRPC Authors
Author-email: [email protected]
License: Apache License 2.0
Location: /home/.../.venv/lib/python3.10/site-packages
Requires: googleapis-common-protos, grpcio, protobuf
Required-by:

Attempts

I've tried to use the following attempts with no sucess:

• Force grpcio-status = "1.48.2" in pyproject.toml
• Be flexible with dependencies using kfp = "^1.8" and google-cloud-aiplatform = "^1"
• Clear all caches by running poetry cache clear pypi --all and rm -rf ~/.cache/pypoetry/cache

Sum up

In short terms: this dependencies in pyproject.toml can't be installed using poetry, but they can using pip

Extra information

• This dependencies could be installed normally in macOS Sonoma 14.1.1
• The updated versions of kfp v2.x and google-cloud-platform it seems to work fine. The issue occurs in kfp v1.x only.

[HenriqueAJNB]

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <[email protected]>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

@Davidnet, could you try to install both kfp and google-cloud-aiplatform now? Here it works, kfp released v2.7.0 recently, just want to be sure that issue is kfp v1.x related.