`vdirsyncer discover` on google calendar fails with "(invalid_grant) Bad Request"

[fphammerle]

[general]
status_path = "~/.local/share/vdirsyncer/status/"

[storage local]
type = "filesystem"
path = "~/Calendar/"
fileext = ".ics"

[storage google_calendar]
type = "google_calendar"
token_file = "~/.local/share/vdirsyncer/tokens/google-calendar"
# https://console.developers.google.com/apis/credentials/oauthclient/****.apps.googleusercontent.com?project=vdirsyncer-****
client_id = "secret"
client_secret = "secret"

[pair local__google_calendar]
a = "local"
b = "google_calendar"
collections = [["personal", "personal", "[email protected]"]]

$ pip freeze
atomicwrites==1.3.0
certifi==2019.11.28
chardet==3.0.4
Click==7.0
click-log==0.3.2
click-threading==0.4.4
idna==2.8
oauthlib==3.1.0
requests==2.22.0
requests-oauthlib==1.3.0
requests-toolbelt==0.9.1
urllib3==1.25.7
vdirsyncer==0.16.7
$ python --version
Python 3.6.9

ubuntu 18.04

vdirsyncer successfully synced for several months

since beginning of september 2019 I get:

$ vdirsyncer -vdebug discover
debug: Using 1 maximal workers.
Discovering collections for pair local__google_calendar
local:
  - "personal"
debug: PROPFIND https://apidata.googleusercontent.com/caldav/v2/
debug: {'User-Agent': '***.apps.googleusercontent.com', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'\n    <d:propfind xmlns:d="DAV:">\n        <d:prop>\n            <d:resourcetype />\n        </d:prop>\n    </d:propfind>\n    '
debug: Sending request...
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 204, in _print_collections
debug:     discovered = get_discovered()
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 133, in get_self
debug:     return self._discovered
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/utils.py", line 170, in __get__
debug:     obj.__dict__[self.__name__] = result = self.fget(obj)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 142, in _discovered
debug:     return handle_storage_init_error(self._cls, self._config)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 138, in _discovered
debug:     discovered = list(self._cls.discover(**self._config))
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 267, in discover
debug:     for c in self.find_collections():
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 220, in find_collections
debug:     rv = list(self._find_collections_impl(''))
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 250, in _find_collections_impl
debug:     data=self._collection_xml)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 390, in request
debug:     return http.request(method, url, session=self._session, **more)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/http.py", line 144, in request
debug:     r = func(method, url, **kwargs)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 497, in request
debug:     self.auto_refresh_url, auth=auth, **kwargs
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 446, in refresh_token
debug:     self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
debug:     self.token = parse_token_response(body, scope=scope)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
debug:     validate_token_parameters(params)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 438, in validate_token_parameters
debug:     raise_from_error(params.get('error'), params)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 405, in raise_from_error
debug:     raise cls(**kwargs)
warning: Failed to discover collections for google_calendar, use `-vdebug` to see the full traceback.
debug: PROPFIND https://apidata.googleusercontent.com/caldav/v2/
debug: {'User-Agent': '***.apps.googleusercontent.com', 'Content-Type': 'application/xml; charset=UTF-8', 'Depth': '1'}
debug: b'\n    <d:propfind xmlns:d="DAV:">\n        <d:prop>\n            <d:resourcetype />\n        </d:prop>\n    </d:propfind>\n    '
debug: Sending request...
error: Unknown error occured: (invalid_grant) Bad Request
error: Use `-vdebug` to see the full traceback.
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/utils.py", line 341, in _worker
debug:     func(wq=self)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/tasks.py", line 82, in discover_collections
debug:     rv = collections_for_pair(pair=pair, **kwargs)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 81, in collections_for_pair
debug:     _handle_collection_not_found=handle_collection_not_found
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 183, in expand_collections
debug:     _handle_collection_not_found
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 197, in _collection_from_discovered
debug:     return get_discovered()[collection]
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 133, in get_self
debug:     return self._discovered
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/utils.py", line 170, in __get__
debug:     obj.__dict__[self.__name__] = result = self.fget(obj)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 142, in _discovered
debug:     return handle_storage_init_error(self._cls, self._config)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/cli/discover.py", line 138, in _discovered
debug:     discovered = list(self._cls.discover(**self._config))
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 267, in discover
debug:     for c in self.find_collections():
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 220, in find_collections
debug:     rv = list(self._find_collections_impl(''))
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 250, in _find_collections_impl
debug:     data=self._collection_xml)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/storage/dav.py", line 390, in request
debug:     return http.request(method, url, session=self._session, **more)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/vdirsyncer/http.py", line 144, in request
debug:     r = func(method, url, **kwargs)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 497, in request
debug:     self.auto_refresh_url, auth=auth, **kwargs
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 446, in refresh_token
debug:     self.token = self._client.parse_request_body_response(r.text, scope=self.scope)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/clients/base.py", line 421, in parse_request_body_response
debug:     self.token = parse_token_response(body, scope=scope)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 431, in parse_token_response
debug:     validate_token_parameters(params)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 438, in validate_token_parameters
debug:     raise_from_error(params.get('error'), params)
debug:   File "/home/me/.local/share/virtualenvs/vdirsyncer-j4iOrP6Q/lib/python3.6/site-packages/oauthlib/oauth2/rfc6749/errors.py", line 405, in raise_from_error
debug:     raise cls(**kwargs)
error: 1 out of 1 tasks failed.

did google's api change?

similar issues: https://github.com/pimutils/vdirsyncer/issues/517 https://github.com/pimutils/vdirsyncer/issues/416

[roleohibachi]

I had this issue too. I noticed some of the traceback mentioned errors with oauth, so decided to test whether it was an authentication issue by deleting my token file. This forced the application to reauthenticate. After that, it seems to be working fine.

Could it be that if a token is unused for some period, Google deems it too old to refresh? Is that something that vdirsyncer could detect, and automatically re-initiate authentication?

In the meantime, just rm ~/.local/share/vdirsyncer/tokens/google-calendar

[erenon]

Is there any way to make the tokens last longer?

[WhyNotHugo]

Google's docs indicate that the token expires after six months.

I guess detecting that and deleting the existing token is possible -- it still requires manual intervention to re-authenticate.

[erenon]

6 months would be fine, it expired me today after 7 days.

[WhyNotHugo]

Odd. Did you by any chance reuse the token in another host?

[erenon]

I did not. Perhaps it has something to do with the fact that this is not a "verified application"?

[Kirens]

Had the same problem. Seems to be that the Publishing status was set to testing.

As stated in answers about the consent OAuth consent screen:

Authorizations by a test user will expire seven days from the time of consent. If your OAuth client requests an offline access type and receives a refresh token, that token will also expire.

After changing it to In Production I haven't had any problems, and no verification was needed.

[WhyNotHugo]

Huh, that's it then.

I guess a good fix is to just delete the token and prompt for a new authentication. Might also make sense to detect refresh_tokens that expire in a week and show a warning.

[PMunch]

Because the app requires "sensitive scopes" (because it accesses my private calendars) putting the application in production requires verification. Verification requires a written statement, a privacy policy, and a YouTube video describing how data is to be used. This is kind of a high bar, is there really no better way to perform this synchronization?

[WhyNotHugo]

You don't need to put it into production mode if it'll only be using your own account. You only need to do that to allow others to use it.

This high bar (and every-changing set of rules) is imposed by Google, along with lots of non-standard quirk which we need to deal with, there's nothing we can do about it.

I have been thinking of splitting out a proxy that does GoogleCalDav<>StandardCalDav translation (mostly to avoid having this extra complexity in vdirsyncer). But this would still require running this proxy locally, and have the same restrictions (unless you're okay with using one hosted by some third party).

[PMunch]

My problem is with the short token lifespan. Apparently you only get seven days before having to relogin when not in production. Or did I miss something?

[RaitoBezarius]

I am running into this, did someone try to use Service Accounts rather than an testing API key which expires every 7 days… ?

[Zeioth]

Same, works after deleting the token file and re-loging. A clearer error message would be welcome.

[WhyNotHugo]

Please see this comment for a solution: https://github.com/pimutils/vdirsyncer/issues/975#issuecomment-1274381163

This was fixed but the documentation was not updated.

[WhyNotHugo]

Update for the docs in https://github.com/pimutils/vdirsyncer/pull/1035