gcsfs
gcsfs copied to clipboard
fsspec
gcsfs does not work with google-auth-oauthlib v0.5.3+
When using gcsfs with the latest version of google-auth-oauthlib, i run into this error.
In [1]: import gcsfs
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Cell In [1], line 1
----> 1 import gcsfs
File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/__init__.py:5
3 __version__ = get_versions()["version"]
4 del get_versions
----> 5 from .core import GCSFileSystem
6 from .mapping import GCSMap
8 __all__ = ["GCSFileSystem", "GCSMap"]
File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/core.py:23
21 from .retry import retry_request, validate_response
22 from .checkers import get_consistency_checker
---> 23 from .credentials import GoogleCredentials
24 from . import __version__ as version
25 from urllib.parse import quote as quote_urllib
File /srv/conda/envs/notebook/lib/python3.9/site-packages/gcsfs/credentials.py:8
6 import google.auth.exceptions
7 from google.oauth2.credentials import Credentials
----> 8 from google_auth_oauthlib.flow import InstalledAppFlow
9 from google.oauth2 import service_account
10 from google.auth.transport.requests import Request
File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/__init__.py:21
1 # Copyright 2019 Google LLC
2 #
3 # Licensed under the Apache License, Version 2.0 (the "License");
(...)
12 # See the License for the specific language governing permissions and
13 # limitations under the License.
15 """oauthlib integration for Google Auth
16
17 This library provides `oauthlib <https://oauthlib.readthedocs.io/>`__
18 integration with `google-auth <https://google-auth.readthedocs.io/>`__.
19 """
---> 21 from .interactive import get_user_credentials
23 __all__ = ["get_user_credentials"]
File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/interactive.py:27
24 import contextlib
25 import socket
---> 27 import google_auth_oauthlib.flow
30 LOCALHOST = "localhost"
31 DEFAULT_PORTS_TO_TRY = 100
File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/flow.py:69
66 import google.auth.transport.requests
67 import google.oauth2.credentials
---> 69 import google_auth_oauthlib.helpers
72 _LOGGER = logging.getLogger(__name__)
73 _OOB_REDIRECT_URIS = [
74 "urn:ietf:wg:oauth:2.0:oob",
75 "urn:ietf:wg:oauth:2.0:oob:auto",
76 "oob",
77 ]
File /srv/conda/envs/notebook/lib/python3.9/site-packages/google_auth_oauthlib/helpers.py:27
24 import datetime
25 import json
---> 27 from google.auth import external_account_authorized_user
28 import google.oauth2.credentials
29 import requests_oauthlib
ImportError: cannot import name 'external_account_authorized_user' from 'google.auth' (/srv/conda/envs/notebook/lib/python3.9/site-packages/google/auth/__init__.py)
I have tried downgrading to version 0.6.0 of "google-auth-oauthlib" without success. i was able to import gcsfs after downgrading google-auth-oauthlib to v0.5.3. The newer versions of Google-auth-oauthlib were released after this repository's latest CI runs. This is probably why the issue didn't arise sooner.
Broken Environment
(notebook) root@80c07463551d:~# conda list gcsfs
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
gcsfs 2022.10.0 pyhd8ed1ab_0 conda-forge
(notebook) root@80c07463551d:~# conda list fsspec
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
fsspec 2022.10.0 pyhd8ed1ab_0 conda-forge
(notebook) root@80c07463551d:~# conda list google
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
google-api-core 2.10.1 pyhd8ed1ab_0 conda-forge
google-api-core-grpc 2.10.1 hd8ed1ab_0 conda-forge
google-apitools 0.5.31 pyh9f0ad1d_0 conda-forge
google-auth 1.35.0 pyh6c4a22f_0 conda-forge
google-auth-httplib2 0.1.0 pyhd8ed1ab_1 conda-forge
google-auth-oauthlib 0.7.0 pyhd8ed1ab_0 conda-forge
google-cloud-bigquery-core 2.34.4 pyh44b312d_0 conda-forge
google-cloud-bigquery-storage 2.11.0 pyh6c4a22f_0 conda-forge
google-cloud-bigquery-storage-core 2.11.0 pyh6c4a22f_0 conda-forge
google-cloud-bigtable 1.7.1 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge
google-cloud-datastore 1.15.5 pyhd8ed1ab_0 conda-forge
google-cloud-dlp 3.9.0 pyhd8ed1ab_0 conda-forge
google-cloud-language 1.3.1 pyh6c4a22f_0 conda-forge
google-cloud-pubsub 2.13.7 pyhd8ed1ab_0 conda-forge
google-cloud-pubsublite 1.6.0 pyhd8ed1ab_0 conda-forge
google-cloud-recommendations-ai 0.7.1 pyhd8ed1ab_0 conda-forge
google-cloud-spanner 1.19.3 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.5.0 pyh6c4a22f_0 conda-forge
google-cloud-videointelligence 1.16.2 pyhd8ed1ab_0 conda-forge
google-cloud-vision 1.0.1 pyhd8ed1ab_0 conda-forge
google-crc32c 1.1.2 py39h8db88ab_4 conda-forge
google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge
googleapis-common-protos 1.56.4 py39hf3d152e_1 conda-forge
googleapis-common-protos-grpc 1.56.4 hf3d152e_1 conda-forge
grpc-google-iam-v1 0.12.4 py39hf3d152e_0 conda-forge
libgoogle-cloud 2.1.0 h9ebe8e8_2 conda-forge
Working environment
(notebook) root@9c6cc1082cce:~# conda list google
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
google-api-core 2.10.1 pyhd8ed1ab_0 conda-forge
google-api-core-grpc 2.10.1 hd8ed1ab_0 conda-forge
google-apitools 0.5.31 pyh9f0ad1d_0 conda-forge
google-auth 1.35.0 pyh6c4a22f_0 conda-forge
google-auth-httplib2 0.1.0 pyhd8ed1ab_1 conda-forge
google-auth-oauthlib 0.5.3 pyhd8ed1ab_0 conda-forge
google-cloud-bigquery-core 2.34.4 pyh44b312d_0 conda-forge
google-cloud-bigquery-storage 2.11.0 pyh6c4a22f_0 conda-forge
google-cloud-bigquery-storage-core 2.11.0 pyh6c4a22f_0 conda-forge
google-cloud-bigtable 1.7.1 pyhd8ed1ab_0 conda-forge
google-cloud-core 2.3.2 pyhd8ed1ab_0 conda-forge
google-cloud-datastore 1.15.5 pyhd8ed1ab_0 conda-forge
google-cloud-dlp 3.9.0 pyhd8ed1ab_0 conda-forge
google-cloud-language 1.3.1 pyh6c4a22f_0 conda-forge
google-cloud-pubsub 2.13.7 pyhd8ed1ab_0 conda-forge
google-cloud-pubsublite 1.6.0 pyhd8ed1ab_0 conda-forge
google-cloud-recommendations-ai 0.7.1 pyhd8ed1ab_0 conda-forge
google-cloud-spanner 1.19.3 pyhd8ed1ab_0 conda-forge
google-cloud-storage 2.5.0 pyh6c4a22f_0 conda-forge
google-cloud-videointelligence 1.16.2 pyhd8ed1ab_0 conda-forge
google-cloud-vision 1.0.1 pyhd8ed1ab_0 conda-forge
google-crc32c 1.1.2 py39h8db88ab_4 conda-forge
google-resumable-media 2.4.0 pyhd8ed1ab_0 conda-forge
googleapis-common-protos 1.56.4 py39hf3d152e_1 conda-forge
googleapis-common-protos-grpc 1.56.4 hf3d152e_1 conda-forge
grpc-google-iam-v1 0.12.4 py39hf3d152e_0 conda-forge
libgoogle-cloud 2.1.0 h9ebe8e8_2 conda-forge
(notebook) root@9c6cc1082cce:~#
Appears to work if you also install google-auth==2.13. Given that 0.7.0 so closely followed 0.6.0, seems like google messed something up here.
Appears to work if you also install google-auth==2.13
My attempt at install google-auth=2.13 fails due to another dependency in my environment that seems not to support v2+
Encountered problems while solving:
- package apache-beam-with-gcp-2.42.0-py39hf3d152e_0 requires google-auth >=1.18.0,<2, but none of the providers can be installed
This is good to know though.
Ah, ok. So that ^ is the principle problem; but also google got wrong the version constraints in its release.
Looking at the most recent CI run at https://github.com/fsspec/gcsfs/actions/runs/5244478235/jobs/9470552946#step:4:41, I see google-auth-oauthlib=1.0.0 installed and the tests seem to be passing. Also tried this locally:
mamba create --name temp python=3.10 google-auth-oauthlib=1.0.0 gcsfs=2023.6.0
mamba activate temp
python -c "import gcsfs; print(gcsfs.__version__)"
which printed 2023.6.0, so I guess import gcsfs should be working again?
