dvc
dvc copied to clipboard
Published
20 hours ago •
iterative
iterative
pydrive2.settings.InvalidConfigError: Missing required setting client_config
Trying to pull data from a data registry stored on GDrive using a service account in a Github action. I am getting the error
pydrive2.settings.InvalidConfigError: Missing required setting client_config
DVC doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.7.13 on Linux-5.15.0-1014-azure-x86_64-with-debian-bullseye-sid
Supports:
gdrive (pydrive2 = 1.14.0),
webhdfs (fsspec = 2022.5.0),
http (aiohttp = 3.8.1, aiohttp-retry = 2.5.2),
https (aiohttp = 3.8.1, aiohttp-retry = 2.5.2)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: gdrive
Workspace directory: ext4 on /dev/sdb1
Repo: dvc, git
Github actions yaml file
name: PyTest
on:
push:
branches: [ develop, master, tidy-up]
pull_request:
branches: [ develop, master ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: [3.6, 3.7, 3.8]
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v2
with:
python-version: ${{ matrix.python-version }}
- name: Setup DVC
uses: iterative/setup-cml@v1
- name: Pull test data with DVC
run: |
pip install -U dvc[gdrive] pydrive2
dvc doctor
dvc pull -v
env:
GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install flake8 pytest
pip install -e .
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pytest
Traceback
2022-07-22 20:17:25,976 DEBUG: Preparing to transfer data from '1q7z3osj88JNPTd-qQxQFD0w8-Cy-mV6h' to '/home/runner/work/lrg-omics/lrg-omics/.dvc/cache'
2022-07-22 20:17:25,977 DEBUG: Preparing to collect status from '/home/runner/work/lrg-omics/lrg-omics/.dvc/cache'
2022-07-22 20:17:25,977 DEBUG: Collecting status from '/home/runner/work/lrg-omics/lrg-omics/.dvc/cache'
2022-07-22 20:17:25,977 DEBUG: Preparing to collect status from '1q7z3osj88JNPTd-qQxQFD0w8-Cy-mV6h'
2022-07-22 20:17:25,977 DEBUG: Collecting status from '1q7z3osj88JNPTd-qQxQFD0w8-Cy-mV6h'
2022-07-22 20:17:25,978 DEBUG: Querying 1 hashes via object_exists
2022-07-22 20:17:26,285 ERROR: unexpected error - Missing required setting client_config
------------------------------------------------------------
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/output.py", line 852, in get_dir_cache
ocheck(self.odb, obj)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/__init__.py", line 19, in check
odb.check(obj.hash_info, **kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/objects/db.py", line 181, in check
obj.check(self, check_hash=check_hash)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/objects/file.py", line 65, in check
self._check_hash(odb)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/objects/file.py", line 72, in _check_hash
self.fs_path, self.fs, self.hash_info.name, odb.state
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/stage.py", line 92, in get_file_hash
meta, hash_info = _get_file_hash(fs_path, fs, name)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/stage.py", line 66, in _get_file_hash
info = _adapt_info(fs.info(fs_path), fs.scheme)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/fs/base.py", line [309](https://github.com/LewisResearchGroup/lrg-omics/runs/7475013326?check_suite_focus=true#step:5:310), in info
return self.fs.info(path)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/fs/local.py", line 54, in info
return self.fs.info(path)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/fsspec/implementations/local.py", line 80, in info
out = os.stat(path, follow_symlinks=False)
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/lrg-omics/lrg-omics/.dvc/cache/0d/ce0a9a9[337](https://github.com/LewisResearchGroup/lrg-omics/runs/7475013326?check_suite_focus=true#step:5:338)eac485acfc7859b1e28e.dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/cli/__init__.py", line 90, in main
ret = cmd.do_run()
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/cli/command.py", line 22, in do_run
return self.run()
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/commands/data_sync.py", line 42, in run
glob=self.args.glob,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/__init__.py", line 48, in wrapper
return f(repo, *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/pull.py", line 44, in pull
odb=odb,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/__init__.py", line 48, in wrapper
return f(repo, *args, **kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/fetch.py", line 55, in fetch
revs=revs,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/__init__.py", line 424, in used_objs
with_deps=with_deps,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/repo/index.py", line 230, in used_objs
filter_info=filter_info,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/stage/__init__.py", line 661, in get_used_objs
for odb, objs in out.get_used_objs(*args, **kwargs).items():
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/output.py", line 934, in get_used_objs
obj = self._collect_used_dir_cache(**kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/output.py", line 874, in _collect_used_dir_cache
self.get_dir_cache(jobs=jobs, remote=remote)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/output.py", line 856, in get_dir_cache
self.repo.cloud.pull([obj.hash_info], **kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data_cloud.py", line 121, in pull
verify=odb.verify,
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/transfer.py", line 155, in transfer
src, dest, obj_ids, check_deleted=False, jobs=jobs, **kwargs
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/status.py", line 187, in compare_status
src, obj_ids, index=src_index, jobs=jobs, **kwargs
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/status.py", line 138, in status
odb, index, dir_objs, name, cache_odb, jobs=jobs
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/data/status.py", line 59, in _indexed_dir_hashes
total=len(dir_missing),
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/tqdm/std.py", line 1183, in __iter__
for obj in iterable:
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/objects/db.py", line [345](https://github.com/LewisResearchGroup/lrg-omics/runs/7475013326?check_suite_focus=true#step:5:346), in list_hashes_exists
yield from itertools.compress(hashes, in_remote)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/concurrent/futures/_base.py", line 598, in result_iterator
yield fs.pop().result()
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/concurrent/futures/_base.py", line 435, in result
return self.__get_result()
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/concurrent/futures/_base.py", line [384](https://github.com/LewisResearchGroup/lrg-omics/runs/7475013326?check_suite_focus=true#step:5:385), in __get_result
raise self._exception
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/fs/base.py", line 244, in exists
return self.fs.exists(path)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/funcy/objects.py", line 50, in __get__
return prop.__get__(instance, type)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/funcy/objects.py", line 28, in __get__
res = instance.__dict__[self.fget.__name__] = self.fget(instance)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/dvc/fs/gdrive.py", line 227, in fs
gauth = GoogleAuth(settings_file="")
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/pydrive2/auth.py", line 200, in __init__
ValidateSettings(self.settings)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/pydrive2/settings.py", line 130, in ValidateSettings
_ValidateSettingsStruct(data, SETTINGS_STRUCT)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/pydrive2/settings.py", line 145, in _ValidateSettingsStruct
_ValidateSettingsElement(data, struct, key)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/pydrive2/settings.py", line 189, in _ValidateSettingsElement
_ValidateSettingsElement(data, struct, reqkey)
File "/opt/hostedtoolcache/Python/3.7.13/x64/lib/python3.7/site-packages/pydrive2/settings.py", line 166, in _ValidateSettingsElement
raise InvalidConfigError("Missing required setting %s" % key)
pydrive2.settings.InvalidConfigError: Missing required setting client_config```
You mean whether I executed
dvc remote modify myremote gdrive_use_service_account true
?
I did that locally, but not as part of my Github actions yaml file.
@sorenwacker Then that's your problem, it looks like :) Run that command in github actions and it should work too.
That surprises me. I have this workflow from a different project and this works.
name: auto-testing
on: [push]
jobs:
run:
runs-on: [ubuntu-latest]
strategy:
matrix:
python-version: [3.6, 3.7, 3.8, 3.9, 3.10]
steps:
- uses: actions/checkout@v2
- uses: iterative/setup-dvc@v1
- name: Get data
run: |
pip install -U dvc[gdrive] pydrive2
dvc pull -r storage
env:
REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GDRIVE_CREDENTIALS_DATA : ${{ secrets.GDRIVE_CREDENTIALS_DATA }}
- name: Install requirements
run: |
pip install -r requirements.txt
- name: Run tests
run: python src/test.py
Does that mean there is persistency in the GitHub repo that can be changed permanently by running a github actions such as setting this DVC settings?
That also means, if I would set this up from scratch this workflow would not work anymore?
@sorenwacker In that other project you either have that config option already set, or maybe you are not using service credentials but cached regular user credentials.