dvc
dvc copied to clipboard
Published
20 hours ago •
iterative
iterative
gdrive: raises unexpected error - name: drive version: v2 (again)
Bug Report
Description
As reported here, the latest version (2.11.0) of dvc on Ubuntu 22.04 LTS still has the bug reported in issue #5618. Downgrading dvc to 2.10.2 allows the push to work properly.
Reproduce
On an existing repository using gdrive, add some files then try to push them.
brett@br-workhorse:~/repos/arctic-ice$ dvc status
models.dvc:
changed outs:
modified: models
brett@br-workhorse:~/repos/arctic-ice$ dvc add models
100% Adding...|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████|1/1 [00:01, 1.52s/file]
brett@br-workhorse:~/repos/arctic-ice$ dvc push
ERROR: unexpected error - name: drive version: v2
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
With extra verbosity:
brett@br-workhorse:~/repos/arctic-ice$ dvc push -vv
2022-06-29 19:59:18,216 TRACE: Namespace(all_branches=False, all_commits=False, all_tags=False, cd='.', cmd='push', cprofile=False, cprofile_dump=None, func=<class 'dvc.commands.data_sync.CmdDataPush'>, glob=False, instrument=False, instrument_open=False, jobs=None, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False), pdb=False, quiet=0, recursive=False, remote=None, run_cache=False, targets=[], verbose=2, version=None, viztracer=False, viztracer_depth=None, with_deps=False, yappi=False)
2022-06-29 19:59:18,399 TRACE: 16.49 ms in collecting stages from /home/brett/repos/arctic-ice
2022-06-29 19:59:18,405 TRACE: 6.02 ms in collecting stages from /home/brett/repos/arctic-ice/RadarMask
2022-06-29 19:59:18,406 TRACE: 18.37 mks in collecting stages from /home/brett/repos/arctic-ice/RadarMask/RadarMask
2022-06-29 19:59:18,420 DEBUG: Preparing to transfer data from '/home/brett/repos/arctic-ice/.dvc/cache' to '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,420 DEBUG: Preparing to collect status from '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,420 DEBUG: Collecting status from '1uTdQZ6FntcI-JtX0ofC4LYPfFM37oZp0'
2022-06-29 19:59:18,423 DEBUG: Querying 11 hashes via object_exists
2022-06-29 19:59:18,635 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,637 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,638 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,639 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,641 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,642 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,642 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,643 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,644 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,645 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,645 DEBUG: GDrive remote auth with config '{'client_config_backend': 'settings', 'client_config_file': 'client_secrets.json', 'save_credentials': True, 'oauth_scope': ['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'], 'save_credentials_backend': 'file', 'save_credentials_file': '/home/brett/repos/arctic-ice/.dvc/tmp/gdrive-user-credentials.json', 'get_refresh_token': True, 'client_config': {'client_id': '710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com', 'client_secret': 'a1Fz59uTpVNeG_VGuSKDLJXv', 'auth_uri': 'https://accounts.google.com/o/oauth2/auth', 'token_uri': 'https://oauth2.googleapis.com/token', 'revoke_uri': 'https://oauth2.googleapis.com/revoke', 'redirect_uri': ''}}'.
2022-06-29 19:59:18,647 ERROR: unexpected error - name: drive version: v2
------------------------------------------------------------
Traceback (most recent call last):
File "dvc/cli/__init__.py", line 185, in main
File "dvc/cli/command.py", line 22, in do_run
File "dvc/commands/data_sync.py", line 58, in run
File "dvc/repo/__init__.py", line 49, in wrapper
File "dvc/repo/push.py", line 68, in push
File "dvc/data_cloud.py", line 109, in push
File "dvc/data_cloud.py", line 88, in transfer
File "dvc_data/transfer.py", line 159, in transfer
File "dvc_data/status.py", line 179, in compare_status
File "dvc_data/status.py", line 136, in status
File "dvc_data/status.py", line 43, in _indexed_dir_hashes
File "tqdm/std.py", line 1195, in __iter__
File "dvc_objects/db.py", line 355, in list_hashes_exists
File "concurrent/futures/_base.py", line 611, in result_iterator
File "concurrent/futures/_base.py", line 439, in result
File "concurrent/futures/_base.py", line 388, in __get_result
File "concurrent/futures/thread.py", line 57, in run
File "dvc_objects/fs/base.py", line 263, in exists
File "pydrive2/fs/spec.py", line 225, in exists
File "pydrive2/fs/spec.py", line 192, in _get_item_id
File "pydrive2/fs/spec.py", line 174, in _path_to_item_ids
File "pydrive2/fs/spec.py", line 170, in _get_cached_item_ids
File "funcy/objects.py", line 50, in __get__
File "funcy/objects.py", line 28, in __get__
File "pydrive2/fs/spec.py", line 82, in _ids_cache
File "pydrive2/fs/spec.py", line 132, in _gdrive_list
File "funcy/objects.py", line 28, in __get__
File "pydrive2/fs/spec.py", line 100, in _list_params
File "funcy/decorators.py", line 45, in wrapper
File "funcy/flow.py", line 127, in retry
File "funcy/decorators.py", line 66, in __call__
File "pydrive2/fs/spec.py", line 121, in _gdrive_shared_drive_id
File "pydrive2/auth.py", line 64, in _decorated
File "pydrive2/auth.py", line 671, in Authorize
File "googleapiclient/_helpers.py", line 130, in positional_wrapper
File "googleapiclient/discovery.py", line 287, in build
File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive version: v2
------------------------------------------------------------
2022-06-29 19:59:19,332 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
File "dvc/cli/__init__.py", line 185, in main
File "dvc/cli/command.py", line 22, in do_run
File "dvc/commands/data_sync.py", line 58, in run
File "dvc/repo/__init__.py", line 49, in wrapper
File "dvc/repo/push.py", line 68, in push
File "dvc/data_cloud.py", line 109, in push
File "dvc/data_cloud.py", line 88, in transfer
File "dvc_data/transfer.py", line 159, in transfer
File "dvc_data/status.py", line 179, in compare_status
File "dvc_data/status.py", line 136, in status
File "dvc_data/status.py", line 43, in _indexed_dir_hashes
File "tqdm/std.py", line 1195, in __iter__
File "dvc_objects/db.py", line 355, in list_hashes_exists
File "concurrent/futures/_base.py", line 611, in result_iterator
File "concurrent/futures/_base.py", line 439, in result
File "concurrent/futures/_base.py", line 388, in __get_result
File "concurrent/futures/thread.py", line 57, in run
File "dvc_objects/fs/base.py", line 263, in exists
File "pydrive2/fs/spec.py", line 225, in exists
File "pydrive2/fs/spec.py", line 192, in _get_item_id
File "pydrive2/fs/spec.py", line 174, in _path_to_item_ids
File "pydrive2/fs/spec.py", line 170, in _get_cached_item_ids
File "funcy/objects.py", line 50, in __get__
File "funcy/objects.py", line 28, in __get__
File "pydrive2/fs/spec.py", line 82, in _ids_cache
File "pydrive2/fs/spec.py", line 132, in _gdrive_list
File "funcy/objects.py", line 28, in __get__
File "pydrive2/fs/spec.py", line 100, in _list_params
File "funcy/decorators.py", line 45, in wrapper
File "funcy/flow.py", line 127, in retry
File "funcy/decorators.py", line 66, in __call__
File "pydrive2/fs/spec.py", line 121, in _gdrive_shared_drive_id
File "pydrive2/auth.py", line 64, in _decorated
File "pydrive2/auth.py", line 671, in Authorize
File "googleapiclient/_helpers.py", line 130, in positional_wrapper
File "googleapiclient/discovery.py", line 287, in build
File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive version: v2
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "dvc_objects/fs/generic.py", line 68, in _try_links
File "dvc_objects/fs/generic.py", line 28, in _link
File "dvc_objects/fs/base.py", line 282, in reflink
File "dvc_objects/fs/implementations/local.py", line 157, in reflink
File "dvc_objects/fs/system.py", line 115, in reflink
File "dvc_objects/fs/system.py", line 101, in _reflink_linux
OSError: [Errno 95] Operation not supported
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "dvc_objects/fs/generic.py", line 127, in _test_link
File "dvc_objects/fs/generic.py", line 76, in _try_links
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/.cFRhX4rL87pLC64LbFJkgt.tmp'
2022-06-29 19:59:19,333 DEBUG: Removing '/home/brett/repos/arctic-ice/.dvc/cache/.LAtLtAtBaEGib7LXwXJKVF.tmp'
2022-06-29 19:59:19,337 DEBUG: Version info for developers:
DVC version: 2.11.0 (deb)
---------------------------------
Platform: Python 3.8.3 on Linux-5.15.0-37-generic-x86_64-with-glibc2.14
Supports:
azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
gdrive (pydrive2 = 1.10.1),
gs (gcsfs = 2022.5.0),
hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
https (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
ssh (sshfs = 2022.6.0),
oss (ossfs = 2021.8.0),
webdav (webdav4 = 0.9.7),
webdavs (webdav4 = 0.9.7)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-06-29 19:59:19,338 DEBUG: Analytics is enabled.
2022-06-29 19:59:19,339 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp0h_rt9la']'
2022-06-29 19:59:19,341 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp0h_rt9la']'
Expected
The push to succeed without errors.
Environment information
Output of dvc doctor:
brett@br-workhorse:~/repos/arctic-ice$ dvc doctor
DVC version: 2.11.0 (deb)
---------------------------------
Platform: Python 3.8.3 on Linux-5.15.0-37-generic-x86_64-with-glibc2.14
Supports:
azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
gdrive (pydrive2 = 1.10.1),
gs (gcsfs = 2022.5.0),
hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
https (aiohttp = 3.8.1, aiohttp-retry = 2.4.8),
s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
ssh (sshfs = 2022.6.0),
oss (ossfs = 2021.8.0),
webdav (webdav4 = 0.9.7),
webdavs (webdav4 = 0.9.7)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
This is running with python 3.10.4 in a virtual environment set up with direnv.
brett@br-workhorse:~/repos/arctic-ice$ python --version
Python 3.10.4
brett@br-workhorse:~/repos/arctic-ice$ cat .envrc
layout python-venv
brett@br-workhorse:~/repos/arctic-ice$ cat .direnv/python-venv-3.10.4/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.10.4
brett@br-workhorse:~/repos/arctic-ice$ which dvc
/usr/bin/dvc
brett@br-workhorse:~/repos/arctic-ice$ pip freeze -l
asttokens==2.0.5
backcall==0.2.0
certifi==2021.10.8
charset-normalizer==2.0.12
cycler==0.11.0
decorator==5.1.1
executing==0.8.3
fonttools==4.33.3
idna==3.3
imageio==2.18.0
imgaug==0.4.0
ipython==8.2.0
jedi==0.18.1
kiwisolver==1.4.2
matplotlib==3.5.1
matplotlib-inline==0.1.3
memory-profiler==0.60.0
networkx==2.8
numpy==1.22.3
opencv-contrib-python==4.5.5.64
opencv-python==4.5.5.64
packaging==21.3
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.1.0
Pillow-SIMD==8.3.2.post0
prompt-toolkit==3.0.29
psutil==5.9.0
ptyprocess==0.7.0
pure-eval==0.2.2
Pygments==2.12.0
pyparsing==3.0.8
python-dateutil==2.8.2
PyWavelets==1.3.0
requests==2.27.1
scikit-image==0.19.2
scipy==1.8.0
Shapely==1.8.1.post1
six==1.16.0
stack-data==0.2.0
tifffile==2022.4.26
torch==1.11.0
torchvision==0.12.0
tqdm==4.64.0
traitlets==5.1.1
typing_extensions==4.2.0
urllib3==1.26.9
wcwidth==0.2.5
Seems like an issue with pyinstaller hook added in https://github.com/iterative/dvc/pull/5619
I the latest deb build the hook-googleapiclient.model.py didn't work as expected, giving this warning:
38027 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...
38070 WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.
However, in the previous build (2.10.2) the hook worked as intended:
36358 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...
36436 INFO: Loading module hook 'hook-dvc.utils.flatten.py' from '/dvc-s3-repo/dvc/scripts/pyinstaller/hooks'...
Issue appears to come from latest Pyinstaller release -> https://github.com/pyinstaller/pyinstaller/pull/6789
@daavoo should we merge the bump (5.2) and release the new version?
See also this https://github.com/iterative/PyDrive2/issues/197#issuecomment-1186367674 and https://github.com/iterative/PyDrive2/issues/197#issuecomment-1186297551
Those might be relevant, maybe we can do something on the static_discovery=False to not depend on that file? A bit of research is required.
This issue I think is causing this https://stackoverflow.com/questions/73025023/dvc-pull-returns-error-configuration-error-failed-to-authenticate-gdrive-remo problem.
I wonder, why does it work on a local machine, but not on GitHub actions? Maybe it is possible to create a workaround until this is fixed?
My GitHub actions job runs Python 3.8.10 (python --version) btw. (dvc doctor returns python 3.9)
argcomplete==2.0.0
attrs==[19](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:20).3.0
Automat==0.8.0
blinker==1.4
certifi==[20](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:21)19.11.28
chardet==3.0.4
Click==7.0
cloud-init==22.2
colorama==0.4.3
command-not-found==0.3
configobj==5.0.6
constantly==15.1.0
crcmod==1.7
crit==0.0.1
cryptography==2.8
dbus-python==1.2.16
distro==1.4.0
distro-info===0.23ubuntu1
entrypoints==0.3
httplib2==0.14.0
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.5.0
incremental==16.10.1
Jinja2==2.10.1
jsonpatch==1.22
jsonpointer==2.0
jsonschema==3.2.0
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
MarkupSafe==1.1.0
more-itertools==4.2.0
netifaces==0.10.4
oauthlib==3.1.0
packaging==[21](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:22).3
pexpect==4.6.0
pipx==1.1.0
protobuf==3.6.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
Pygments==2.3.1
PyGObject==3.36.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyOpenSSL==19.0.0
pyparsing==3.0.9
pyparted==3.11.2
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.7
python-debian===0.1.36ubuntu1
PyYAML==5.3.1
requests==2.[22](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:23).0
requests-unixsocket==0.2.0
SecretStorage==2.3.1
service-identity==18.1.0
simplejson==3.16.0
six==1.14.0
sos==4.3
ssh-import-id==5.10
systemd-python==[23](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:24)4
Twisted==18.9.0
ubuntu-advantage-tools==27.9
ufw==0.36
urllib3==1.[25](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:26).8
userpath==1.8.0
wadllib==1.3.3
WALinuxAgent==2.2.[46](https://github.com/sorenwacker/dvc-test-project-01/runs/7407074522?check_suite_focus=true#step:4:47)
zipp==1.0.0
zope.interface==4.7.1
dvc doctor
Platform: Python 3.9.13 on Linux-5.15.0-[101](https://github.com/sorenwacker/dvc-test-project-01/runs/7407189429?check_suite_focus=true#step:4:102)4-azure-x86_64-with-glibc2.31
Supports:
azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
gdrive (pydrive2 = 1.10.1),
gs (gcsfs = 2022.5.0),
hdfs (fsspec = 2022.5.0, pyarrow = 8.0.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
https (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
s3 (s3fs = 2022.5.0, boto3 = 1.21.21),
ssh (sshfs = 2022.6.0),
oss (ossfs = 2021.8.0),
webdav (webdav4 = 0.9.7),
webdavs (webdav4 = 0.9.7)
I'm running on Windows 10, and had the same error message when running dvc pull. The windows installation for dvc appears to be the cause for me. I removed the windows dvc installation, and used pip to install dvc instead. Had to manually add a dependency though.
py -m pip install 'dvc[gdrive]'
py -m pip install pydrive2
And after this dvc pull worked fine again.
That definitely changed the error message.
aiohttp==3.8.1
aiohttp-retry==2.5.2
aiosignal==1.2.0
appdirs==1.4.4
argcomplete==2.0.0
async-timeout==4.0.2
asyncssh==2.11.0
atpublic==3.0.1
attrs==19.3.0
Automat==0.8.0
blinker==1.4
cachetools==5.2.0
certifi==2019.11.28
cffi==1.15.1
chardet==3.0.4
charset-normalizer==2.1.0
Click==7.0
cloud-init==22.2
colorama==0.4.3
command-not-found==0.3
commonmark==0.9.1
configobj==5.0.6
constantly==15.1.0
crcmod==1.7
crit==0.0.1
cryptography==37.0.4
dbus-python==1.2.16
dictdiffer==0.9.0
diskcache==5.4.0
distro==1.4.0
distro-info==0.23ubuntu1
dpath==2.0.6
dulwich==0.20.45
dvc==2.13.0
dvc-data==0.0.23
dvc-objects==0.0.23
dvc-render==0.0.6
dvclive==0.9.0
entrypoints==0.3
flatten-dict==0.4.2
flufl.lock==7.0
frozenlist==1.3.0
fsspec==2022.5.0
ftfy==6.1.1
funcy==1.17
future==0.18.2
gitdb==4.0.9
GitPython==3.1.27
google-api-core==2.8.2
google-api-python-client==2.54.0
google-auth==2.9.1
google-auth-httplib2==0.1.0
googleapis-common-protos==1.56.4
grandalf==0.6
httplib2==0.20.4
hyperlink==19.0.0
idna==2.8
importlib-metadata==1.5.0
importlib-resources==5.8.0
incremental==16.10.1
Jinja2==2.10.1
jsonpatch==1.22
jsonpointer==2.0
jsonschema==3.2.0
keyring==18.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.3
mailchecker==4.1.18
MarkupSafe==1.1.0
more-itertools==4.2.0
multidict==6.0.2
nanotime==0.5.2
netifaces==0.10.4
networkx==2.8.5
oauth2client==4.1.3
oauthlib==3.1.0
packaging==21.3
pathspec==0.9.0
pexpect==4.6.0
phonenumbers==8.12.52
pip==20.0.2
pipx==1.1.0
protobuf==4.21.2
psutil==5.9.1
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.21
pydot==1.4.2
PyDrive2==1.13.0
pygit2==1.9.2
Pygments==2.12.0
PyGObject==3.36.0
pygtrie==2.5.0
PyHamcrest==1.9.0
PyJWT==1.7.1
pymacaroons==0.13.0
PyNaCl==1.3.0
pyOpenSSL==22.0.0
pyparsing==3.0.9
pyparted==3.11.2
pyrsistent==0.15.5
pyserial==3.4
python-apt==2.0.0+ubuntu0.20.4.7
python-benedict==0.25.2
python-dateutil==2.8.2
python-debian==0.1.36ubuntu1
python-fsutil==0.6.1
python-slugify==6.1.2
PyYAML==6.0
requests==2.22.0
requests-unixsocket==0.2.0
rich==12.5.1
rsa==4.9
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
scmrepo==0.0.25
SecretStorage==2.3.1
service-identity==18.1.0
setuptools==45.2.0
shortuuid==1.0.9
shtab==1.5.5
simplejson==3.16.0
six==1.14.0
smmap==5.0.0
sos==4.3
ssh-import-id==5.10
systemd-python==234
tabulate==0.8.10
text-unidecode==1.3
toml==0.10.2
tqdm==4.64.0
Twisted==18.9.0
typing-extensions==4.3.0
ubuntu-advantage-tools==27.9
ufw==0.36
uritemplate==4.1.1
urllib3==1.25.8
userpath==1.8.0
voluptuous==0.13.1
wadllib==1.3.3
WALinuxAgent==2.2.46
wcwidth==0.2.5
wheel==0.34.2
xmltodict==0.13.0
yarl==1.7.2
zc.lockfile==2.0
zipp==3.8.1
zope.interface==4.7.1
DVC version: 2.13.0 (pip)
---------------------------------
Platform: Python 3.8.10 on Linux-5.15.0-1014-azure-x86_64-with-glibc2.29
Supports:
gdrive (pydrive2 = 1.13.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/sda1
Repo: dvc, git
ERROR: unexpected error - Missing required setting client_config
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
Error: Process completed with exit code [255](https://github.com/sorenwacker/dvc-test-project-01/runs/7449367677?check_suite_focus=true#step:4:256).
@sorenwacker Were you able to resolve that new one? Looks like a simple misconfiguration issue.
Upgraded to pyinstaller 5.2 seems to solve this. Releasing 2.14.0 ASAP, please give it a try and let us know if that fixes it for you.
@efiop I am not sure where to look for a misconfiguration. This is my current workflow:
name: auto-testing
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-dvc@v1
- name: Get data
run: |
echo '---'
echo GDRIVE_CREDENTIALS_DATA: $GDRIVE_CREDENTIALS_DATA
echo '---'
python --version
pip uninstall dvc
pip install dvc[gdrive]
pip install pydrive2
pip list --format freeze
dvc doctor
dvc remote modify storage --local gdrive_use_service_account true
dvc remote modify storage --local gdrive_service_account_json_file_path .dvc/gdrive-access.json
dvc pull
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}
- name: Install requirements
run: |
pip install -r requirements.txt
- name: Run tests
run: python src/test.py
Here is a minimal setup that works:
name: auto-testing
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-dvc@v1
- name: Get data
run: |
pip install -U dvc[gdrive] pydrive2
dvc pull
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}
- name: Install requirements
run: |
pip install -r requirements.txt
- name: Run tests
run: python src/test.py
@sorenwacker In your first workflow, I don't see .dvc/gdrive-access.json created anywhere, which I suppose is the issue.
Great to hear it works for you now 🙂 Thanks for the feedback!
As I said, the last post is a setup that works. It is not using the key file. I added the key-file to the (private) repository just to see if that would work. The important step here is:
pip install -U dvc[gdrive] pydrive2
Bumping to pyinstaller 5.2. appears to haven't fixed the issue. Hooks on the latest build still raise warnings and a user has reported the same bug in DVC version: 2.17.0 (deb):
https://github.com/iterative/dvc-s3-repo/runs/7703612801?check_suite_focus=true#step:6:2640
Hey I'm also facing this issue at DVC version: 2.18.1 (deb). Reached out for daavoo. waiting for a fix now.
We reverted to pyinstaller 5.0 (thanks @daavoo ) for now. No time to look into this deeper, so going to close it for now.
This appears to be broken again (w/pyinstaller 5.6)
from discord context: https://discord.com/channels/485586884165107732/485596304961962003/1041680983482695751
2022-11-14 11:45:50,298 ERROR: unexpected error - Failed to authenticate GDrive: name: drive version: v2
------------------------------------------------------------
Traceback (most recent call last):
File "pydrive2/fs/spec.py", line 70, in _wrap_errors
File "pydrive2/fs/spec.py", line 150, in _service_auth
File "pydrive2/auth.py", line 101, in _decorated
File "pydrive2/auth.py", line 734, in Authorize
File "googleapiclient/_helpers.py", line 130, in positional_wrapper
File "googleapiclient/discovery.py", line 287, in build
File "googleapiclient/discovery.py", line 404, in _retrieve_discovery_doc
googleapiclient.errors.UnknownApiNameOrVersion: name: drive version: v2
...
pydrive2.fs.spec.GDriveAuthError: Failed to authenticate GDrive
------------------------------------------------------------
2022-11-14 11:45:51,251 DEBUG: Version info for developers:
DVC version: 2.34.0 (deb)
---------------------------------
Platform: Python 3.10.8 on Linux-5.15.0-1022-azure-x86_64-with-glibc2.31
We probably need explicit pyinstaller hooks for this library in either DVC or pydrive2 (or we need to contribute an actual fix upstream in pyinstaller's hook)
.deb build logs for 2.34.1 https://github.com/iterative/dvc-s3-repo/actions/runs/3449002686/jobs/5756539961
71580 INFO: Loading module hook 'hook-googleapiclient.model.py' from '/root/.pyenv/versions/3.10.8/lib/python3.10/site-packages/_pyinstaller_hooks_contrib/hooks/stdhooks'...
71635 WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.
this should be resolved by the linked pydrive patch once it is merged (and dvc-gdrive is updated)
Should note that we will probably continue to see the
WARNING: collect_data_files - skipping data collection for module 'googleapiclient.discovery' as it is not a package.
warning in the future, but it should no longer matter for us. The warning is generated due to an outdated pyinstaller.contrib hook for googleapiclient.model.
Can I have the requirements.txt file that I can use on my python 3.9 so that I get started with solving an isssue
