zenml icon indicating copy to clipboard operation
zenml copied to clipboard
verified publisher icon zenml-io

[BUG]: No matching distribution found for gcsfs

Open ridhoofadilah opened this issue 9 months ago • 1 comments

Contact Details [Optional]

No response

System Information

ZENML_LOCAL_VERSION: 0.80.0 ZENML_SERVER_VERSION: 0.80.0 ZENML_SERVER_DATABASE: sqlite ZENML_SERVER_DEPLOYMENT_TYPE: other ZENML_CONFIG_DIR: /home/user/.config/zenml ZENML_LOCAL_STORE_DIR: /home/user/.config/zenml/local_stores ZENML_SERVER_URL: sqlite:////home/user/.config/zenml/local_stores/default_zen_store/zenml.db ZENML_ACTIVE_REPOSITORY_ROOT: /home/user/Documents/work/project/zenml PYTHON_VERSION: 3.9.18 ENVIRONMENT: native SYSTEM_INFO: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '23.04'} ACTIVE_PROJECT: default ACTIVE_STACK: gcp_stack ACTIVE_USER: default TELEMETRY_STATUS: enabled ANALYTICS_CLIENT_ID: 86995762-8c28-4f97-b67c-062152de57f6 ANALYTICS_USER_ID: 20a597eb-b670-4eb3-8ecd-9a2125adfa20 ANALYTICS_SERVER_ID: 86995762-8c28-4f97-b67c-062152de57f6 INTEGRATIONS: ['airflow', 'bitbucket', 'gcp', 'kaniko', 'mlflow', 'numpy', 'pandas', 'pigeon', 'pillow', 'scipy', 'sklearn'] PACKAGES: {'brotli': '1.1.0', 'deprecated': '1.2.18', 'gitpython': '3.1.44', 'mako': '1.3.9', 'markdown': '3.7', 'markupsafe': '3.0.2', 'pymysql': '1.1.1', 'pyyaml': '6.0.2', 'sqlalchemy-utils': '0.41.2', 'aiofiles': '24.1.0', 'aiohappyeyeballs': '2.6.1', 'aiohttp': '3.11.14', 'aiokafka': '0.12.0', 'aiosignal': '1.3.2', 'alembic': '1.8.1', 'annotated-types': '0.7.0', 'anyio': '4.9.0', 'asgiref': '3.8.1', 'asttokens': '3.0.0', 'async-timeout': '5.0.1', 'attrs': '25.3.0', 'bcrypt': '4.0.1', 'blinker': '1.9.0', 'cachetools': '5.5.2', 'certifi': '2025.1.31', 'charset-normalizer': '3.4.1', 'click': '8.1.3', 'cloudpickle': '2.2.1', 'comm': '0.2.2', 'contourpy': '1.3.0', 'cycler': '0.12.1', 'databricks-sdk': '0.47.0', 'decorator': '5.2.1', 'distro': '1.9.0', 'docker': '7.1.0', 'docstring-parser': '0.16', 'exceptiongroup': '1.2.2', 'executing': '2.2.0', 'fastapi': '0.109.2', 'flask': '3.1.0', 'fonttools': '4.56.0', 'frozenlist': '1.5.0', 'fsspec': '2025.3.0', 'gcsfs': '2025.3.0', 'gevent': '24.11.1', 'geventhttpclient': '2.0.2', 'gitdb': '4.0.12', 'google-api-core': '2.24.2', 'google-auth': '2.38.0', 'google-auth-oauthlib': '1.2.1', 'google-cloud-aiplatform': '1.85.0', 'google-cloud-artifact-registry': '1.15.2', 'google-cloud-bigquery': '3.31.0', 'google-cloud-build': '3.31.1', 'google-cloud-container': '2.56.1', 'google-cloud-core': '2.4.3', 'google-cloud-pipeline-components': '2.19.0', 'google-cloud-resource-manager': '1.14.2', 'google-cloud-secret-manager': '2.23.2', 'google-cloud-storage': '2.19.0', 'google-crc32c': '1.7.1rc1', 'google-resumable-media': '2.7.2', 'googleapis-common-protos': '1.69.2', 'graphene': '3.4.3', 'graphql-core': '3.2.6', 'graphql-relay': '3.2.0', 'greenlet': '3.1.1', 'grpc-google-iam-v1': '0.14.2', 'grpcio': '1.71.0', 'grpcio-status': '1.63.0rc1', 'gunicorn': '23.0.0', 'h11': '0.14.0', 'idna': '3.10', 'imageio': '2.37.0', 'importlib-metadata': '7.0.0', 'importlib-resources': '5.13.0', 'ipython': '8.18.1', 'ipywidgets': '8.1.5', 'itsdangerous': '2.2.0', 'jedi': '0.19.2', 'jinja2': '3.1.6', 'joblib': '1.4.2', 'jupyterlab-widgets': '3.0.13', 'kfp': '2.10.1', 'kfp-pipeline-spec': '0.5.0', 'kfp-server-api': '2.3.0', 'kiwisolver': '1.4.7', 'kubernetes': '30.1.0', 'lazy-loader': '0.4', 'markdown-it-py': '3.0.0', 'matplotlib': '3.9.4', 'matplotlib-inline': '0.1.7', 'mdurl': '0.1.2', 'mlflow': '2.20.4', 'mlflow-skinny': '2.20.4', 'mlserver': '1.4.0', 'mlserver-mlflow': '1.6.1', 'multidict': '6.2.0', 'networkx': '3.2.1', 'numpy': '1.26.4', 'oauthlib': '3.2.2', 'opentelemetry-api': '1.27.0', 'opentelemetry-exporter-otlp-proto-common': '1.27.0', 'opentelemetry-exporter-otlp-proto-grpc': '1.27.0', 'opentelemetry-instrumentation': '0.48b0', 'opentelemetry-instrumentation-asgi': '0.48b0', 'opentelemetry-instrumentation-fastapi': '0.48b0', 'opentelemetry-instrumentation-grpc': '0.48b0', 'opentelemetry-proto': '1.27.0', 'opentelemetry-sdk': '1.27.0', 'opentelemetry-semantic-conventions': '0.48b0', 'opentelemetry-util-http': '0.48b0', 'orjson': '3.10.16', 'packaging': '24.2', 'pandas': '2.2.3', 'parso': '0.8.4', 'passlib': '1.7.4', 'pexpect': '4.9.0', 'pillow': '11.1.0', 'pip': '25.0.1', 'prometheus-client': '0.21.1', 'prompt-toolkit': '3.0.50', 'propcache': '0.3.0', 'proto-plus': '1.26.1', 'protobuf': '4.25.6', 'psutil': '7.0.0', 'ptyprocess': '0.7.0', 'pure-eval': '0.2.3', 'py-grpc-prometheus': '0.8.0', 'pyarrow': '19.0.1', 'pyasn1': '0.6.1', 'pyasn1-modules': '0.4.1', 'pydantic': '2.8.2', 'pydantic-core': '2.20.1', 'pydantic-settings': '2.8.1', 'pygments': '2.19.1', 'pyparsing': '3.2.3', 'python-dateutil': '2.9.0.post0', 'python-dotenv': '1.0.1', 'python-rapidjson': '1.14', 'pytz': '2025.2', 'requests': '2.32.3', 'requests-oauthlib': '2.0.0', 'requests-toolbelt': '0.10.1', 'rich': '13.9.4', 'rsa': '4.9', 'scikit-image': '0.24.0', 'scikit-learn': '1.6.1', 'scipy': '1.13.1', 'setuptools': '75.8.2', 'shapely': '2.0.7', 'six': '1.17.0', 'smmap': '5.0.2', 'sniffio': '1.3.1', 'sqlalchemy': '2.0.39', 'sqlmodel': '0.0.18', 'sqlparse': '0.5.3', 'stack-data': '0.6.3', 'starlette': '0.36.3', 'starlette-exporter': '0.23.0', 'tabulate': '0.9.0', 'threadpoolctl': '3.6.0', 'tifffile': '2024.8.30', 'traitlets': '5.14.3', 'tritonclient': '2.41.1', 'typing-extensions': '4.12.2', 'tzdata': '2025.2', 'urllib3': '1.26.20', 'uvicorn': '0.34.0', 'uvloop': '0.21.0', 'wcwidth': '0.2.13', 'websocket-client': '1.8.0', 'werkzeug': '3.1.3', 'wheel': '0.45.1', 'widgetsnbextension': '4.0.13', 'wrapt': '1.17.2', 'yarl': '1.18.3', 'zenml': '0.80.0', 'zipp': '3.21.0', 'zope.event': '5.0', 'zope.interface': '7.2', 'autocommand': '2.2.2', 'backports.tarfile': '1.2.0', 'inflect': '7.3.1', 'jaraco.collections': '5.1.0', 'jaraco.context': '5.3.0', 'jaraco.functools': '4.0.1', 'jaraco.text': '3.12.1', 'more-itertools': '10.3.0', 'platformdirs': '4.2.2', 'tomli': '2.0.1', 'typeguard': '4.3.0'}

CURRENT STACK

Name: gcp_stack ID: f4cb9e2d-f34a-480b-8fbf-c7790aacde0c User: default / 20a597eb-b670-4eb3-8ecd-9a2125adfa20

ARTIFACT_STORE: gcp_artifact_store

Name: gcp_artifact_store ID: bf5a4af1-4f54-42e1-aaca-1158126b35cd Type: artifact_store Flavor: gcp Configuration: {'path': 'gs://BUCKET-NAME/zenml-testing/'} User: default / 20a597eb-b670-4eb3-8ecd-9a2125adfa20

CONTAINER_REGISTRY: gcp_container_registry

Name: gcp_container_registry ID: d46133fc-d388-4288-8b44-0742deeb8abd Type: container_registry Flavor: gcp Configuration: {'uri': 'asia-southeast1-docker.pkg.dev/PROJECT-ID/data-science/zenml/testing'} User: default / 20a597eb-b670-4eb3-8ecd-9a2125adfa20

ORCHESTRATOR: gcp_vertex_orchestrator

Name: gcp_vertex_orchestrator ID: f3eb7611-3dc7-4075-8a54-fc815198865c Type: orchestrator Flavor: vertex Configuration: {'synchronous': True, 'project': 'PROJECT-ID', 'location': 'asia-southeast1'} User: default / 20a597eb-b670-4eb3-8ecd-9a2125adfa20

What happened?

when running the pipeline, no matching distribution found for gcsfs

Image

Reproduction steps

im following the steps specified in gcp guide

  1. Choose a GCP project
  2. Enable GCloud APIs
  3. Create a dedicated service account
  4. Create a JSON Key for your service account
  5. Create a Service Connector within ZenML
  6. Create Stack Components (Artifact Store, Orchestrator, Container Registry)
  7. Create Stack

after the setup is done

  1. run the pipeline

Relevant log output

<zenml.steps.entrypoint_function_utils.StepArtifact object at 0x7fc95d551af0>
Initiating a new run for the pipeline: training_pipeline.
Unable to find a build to reuse. A previous build can be reused when the following conditions are met:
  * The existing build was created for the same stack, ZenML version and Python version
  * The stack contains a container registry
  * The Docker settings of the pipeline and all its steps are the same as for the existing build.
Building Docker image(s) for pipeline training_pipeline.
Building Docker image asia-southeast1-docker.pkg.dev/{project_id}/data-science/zenml/testing/zenml:training_pipeline-orchestrator.
- Including stack requirements: gcsfs, google-cloud-aiplatform>=1.34.0, google-cloud-artifact-registry>=1.11.3, google-cloud-build>=3.11.0, google-cloud-container>=2.21.0, google-cloud-pipeline-components>=2.19.0, google-cloud-secret-manager, google-cloud-storage>=2.9.0, kfp>=2.6.0, kubernetes
Step 1/9 : FROM zenmldocker/zenml:0.80.0-py3.9
Step 2/9 : WORKDIR /app
Step 3/9 : ENV ZENML_LOGGING_COLORS_DISABLED=False
Step 4/9 : COPY .zenml_stack_integration_requirements .
Step 5/9 : RUN pip install --no-cache-dir --default-timeout=60 -r .zenml_stack_integration_requirements
ERROR: Could not find a version that satisfies the requirement gcsfs (from versions: none)
ERROR: No matching distribution found for gcsfs

Code of Conduct

  • [x] I agree to follow this project's Code of Conduct

ridhoofadilah avatar Mar 26 '25 07:03 ridhoofadilah

Hi, I'd like to work on this issue. Could you please assign it to me?

readmyname95 avatar May 22 '25 17:05 readmyname95

Hi team, I’ve been working on this issue, but I’ve run into a couple of blockers. It seems I’ve run out of GCP credits, which prevents me from running the pipeline and testing things further. Also, from what I can tell, the root cause might be related to a version compatibility issue. I think this could potentially be bypassed by pinning a specific version, but without the ability to run and verify it, I’m currently stuck. Unfortunately, I won’t be able to continue troubleshooting this properly, so I’ll leave this issue here in case someone else can take it forward. Thanks again for the opportunity — I’d still love to contribute to other issues that don’t require GCP resources.

readmyname95 avatar Jun 29 '25 00:06 readmyname95

Hey @readmyname95, sorry about the late response. As you reported, we have had some issues with the gcsfs installation. Upon closer look, we have found:

  • the pip installation was getting stuck if you had a gcp stack as you did. Using uv as the installer solves this issue though.
  • additionally, we have limited the version of the gcsfs for now.

Let us know if this helps in any way.

bcdurak avatar Sep 23 '25 15:09 bcdurak

Hey @readmyname95, this issue must be fixed now. So I am closing this, however, feel free to reopen it if you encounter the same problem again.

bcdurak avatar Nov 10 '25 14:11 bcdurak