Blocked Google Calendar requests due to `invalid_grant` from expired token

[eeshaan]

Issue Summary

When stopping the Cal.com web instance and restarting it, the Google Calendar response comes back as with a 400 bad request. Instead of attempted to fetch a new token, the Google Calendar integration breaks until the user removes the calendar connection and signs back into their Google account.

Steps to Reproduce

1. Connect Google Calendar OAuth Client
2. Restart @calcom/web
3. invalid_grant on Google Calendar
4. User has to remove calendar connection and sign back into the Google account.

Technical details

cal.com v2.3.2 — though the issue has been present in previous releases as well.

@calcom/web:start: 04:43:04.176 timeZoneName WARN CalendarManager
@calcom/web:start:  Error  invalid_grant
@calcom/web:start: details:
@calcom/web:start: <ref *1> {
@calcom/web:start:   response: {
@calcom/web:start:     config: {
@calcom/web:start:       method: 'POST',
@calcom/web:start:       url: 'https://oauth2.googleapis.com/token',
@calcom/web:start:       data: 'refresh_token=[redacted]&client_id=[redacted].apps.googleusercontent.com&client_secret=[redacted]&grant_type=refresh_token',
@calcom/web:start:       headers: {
@calcom/web:start:         'Content-Type': 'application/x-www-form-urlencoded',
@calcom/web:start:         'User-Agent': 'google-api-nodejs-client/7.14.1',
@calcom/web:start:         'x-goog-api-client': 'gl-node/16.18.0 auth/7.14.1',
@calcom/web:start:         Accept: 'application/json'
@calcom/web:start:       },
@calcom/web:start:       paramsSerializer: [Function: paramsSerializer],
@calcom/web:start:       body: 'refresh_token=[redacted]&client_id=[redacted].apps.googleusercontent.com&client_secret=[redacted]&grant_type=refresh_token',
@calcom/web:start:       validateStatus: [Function: validateStatus],
@calcom/web:start:       responseType: 'json'
@calcom/web:start:     },
@calcom/web:start:     data: {
@calcom/web:start:       error: 'invalid_grant',
@calcom/web:start:       error_description: 'Token has been expired or revoked.'
@calcom/web:start:     },
@calcom/web:start:     headers: {
@calcom/web:start:       'cache-control': 'no-cache, no-store, max-age=0, must-revalidate',
@calcom/web:start:       connection: 'close',
@calcom/web:start:       'content-encoding': 'gzip',
@calcom/web:start:       'content-type': 'application/json; charset=utf-8',
@calcom/web:start:       date: 'Tue, 29 Nov 2022 04:43:04 GMT',
@calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',
@calcom/web:start:       pragma: 'no-cache',
@calcom/web:start:       server: 'scaffolding on HTTPServer2',
@calcom/web:start:       'transfer-encoding': 'chunked',
@calcom/web:start:       vary: 'Origin, X-Origin, Referer',
@calcom/web:start:       'x-content-type-options': 'nosniff',
@calcom/web:start:       'x-frame-options': 'SAMEORIGIN',
@calcom/web:start:       'x-xss-protection': '0'
@calcom/web:start:     },
@calcom/web:start:     status: 400,
@calcom/web:start:     statusText: 'Bad Request',
@calcom/web:start:     request: {
@calcom/web:start:       responseURL: 'https://oauth2.googleapis.com/token'
@calcom/web:start:     }
@calcom/web:start:   },
@calcom/web:start:   config: [Circular *1],
@calcom/web:start:   code: '400'
@calcom/web:start: }
@calcom/web:start: error stack:
@calcom/web:start: • gaxios.ts:158 _request
@calcom/web:start:     node_modules/gaxios/src/gaxios.ts:158:15
@calcom/web:start:
@calcom/web:start: • task_queues:96 processTicksAndRejections
@calcom/web:start:     node:internal/process/task_queues:96:5
@calcom/web:start:
@calcom/web:start: • oauth2client.js:174 refreshTokenNoCache
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:174:21
@calcom/web:start:
@calcom/web:start: • oauth2client.js:284 getRequestMetadataAsync
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:284:17
@calcom/web:start:
@calcom/web:start: • oauth2client.js:357 requestAsync
@calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:357:23
@calcom/web:start:
@calcom/web:start:

[libeanim]

We seem to have the same issue using calcom-docker (with cal.com v2.4.4). Were you able to fix it?

[PeerRich]

have you tried the latest here? https://hub.docker.com/r/calcom/cal.com/tags

[libeanim]

We've used the Dockerfile in the calcom/docker project to build our own image, as otherwise non-localhost domain names don't work as far as I understood.

[PeerRich]

right that makes sense. hmm that is odd

[PeerRich]

maybe @zomars knows more

[zomars]

No clue. It seems like the token gets invalidated. Maybe the encryption key resets on restart? Although that would mean password wouldn't work as well. @krumware any ideas?

[krumware]

I'm very curious as to the expiration and date portion of the request. Is this 'expires' value normal?

@calcom/web:start:       date: 'Tue, 29 Nov 2022 04:43:04 GMT',
@calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',

[krumware]

Through some googling, this could be related to a number of configuration items, not necessarily those specific to the cal.com side. I've had problems in the past when the callback URL wasn't added properly, or my environmental variable had invalid characters or extra quotes.

Quick note, non-localhost domains do work via calcom/docker

[libeanim]

Thanks for looking into it! If its a configuration issue or an issue with the callback URL then it shouldn't work after reconnecting the google account, right? But that seems to work for now, as long as we don't restart or recreate the container.

@krumware re non-localhost domains, in the Readme it says:

...there are specific requirements for providing environmental variables at build-time in order to specify a non-localhost BASE_URL. ...

and also

For Production, for the time being, please checkout the repository and build/push your own image privately.

So it seems like we cannot use the pre-built image but need to build our own?

[krumware]

That's really interesting around the restart. We'll continue to monitor.

You can use it, it's not a license restriction or anything. It's just general best practice for container immutability to not allow some file writes at runtime, and some more compliance-heavy environments may have controls affecting that. So the true "enterprise production" recommendation would be to build it (until we fix that). But otherwise for most applications it's fine to run the provided container. We'll try to make that more clear!

[libeanim]

Ok it seems that the restarting (or recreating) the container has nothing to do with it! Its that the google tokens cannot get refreshed. Had it running w/o restart and after roughly one week the google tokens expire and I see this error:

calcom    | @calcom/web:start: 22:16:18.547 ERROR [[lib] google_calendar Error refreshing google token 
calcom    | @calcom/web:start:  Error  invalid_grant
calcom    | @calcom/web:start: details:
calcom    | @calcom/web:start: <ref *1> {
calcom    | @calcom/web:start:   response: {
calcom    | @calcom/web:start:     config: {
calcom    | @calcom/web:start:       method: 'POST',
calcom    | @calcom/web:start:       url: 'https://oauth2.googleapis.com/token',
calcom    | @calcom/web:start:       data: 'refresh_token=<redacted>&client_id=<redacted>.apps.googleusercontent.com&client_secret=<redacted>&grant_type=refresh_token',
calcom    | @calcom/web:start:       headers: {
calcom    | @calcom/web:start:         'Content-Type': 'application/x-www-form-urlencoded',
calcom    | @calcom/web:start:         'User-Agent': 'google-api-nodejs-client/7.14.1',
calcom    | @calcom/web:start:         'x-goog-api-client': 'gl-node/16.19.0 auth/7.14.1',
calcom    | @calcom/web:start:         Accept: 'application/json'
calcom    | @calcom/web:start:       },
calcom    | @calcom/web:start:       paramsSerializer: [Function: paramsSerializer],
calcom    | @calcom/web:start:       body: 'refresh_token=<redacted>&client_id=<redacted>.apps.googleusercontent.com&client_secret=<redacted>&grant_type=refresh_token',
calcom    | @calcom/web:start:       validateStatus: [Function: validateStatus],
calcom    | @calcom/web:start:       responseType: 'json'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     data: {
calcom    | @calcom/web:start:       error: 'invalid_grant',
calcom    | @calcom/web:start:       error_description: 'Token has been expired or revoked.'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     headers: {
calcom    | @calcom/web:start:       'alt-svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"',
calcom    | @calcom/web:start:       'cache-control': 'no-cache, no-store, max-age=0, must-revalidate',
calcom    | @calcom/web:start:       connection: 'close',
calcom    | @calcom/web:start:       'content-encoding': 'gzip',
calcom    | @calcom/web:start:       'content-type': 'application/json; charset=utf-8',
calcom    | @calcom/web:start:       date: 'Sun, 22 Jan 2023 22:16:18 GMT',
calcom    | @calcom/web:start:       expires: 'Mon, 01 Jan 1990 00:00:00 GMT',
calcom    | @calcom/web:start:       pragma: 'no-cache',
calcom    | @calcom/web:start:       server: 'scaffolding on HTTPServer2',
calcom    | @calcom/web:start:       'transfer-encoding': 'chunked',
calcom    | @calcom/web:start:       vary: 'Origin, X-Origin, Referer',
calcom    | @calcom/web:start:       'x-content-type-options': 'nosniff',
calcom    | @calcom/web:start:       'x-frame-options': 'SAMEORIGIN',
calcom    | @calcom/web:start:       'x-xss-protection': '0'
calcom    | @calcom/web:start:     },
calcom    | @calcom/web:start:     status: 400,
calcom    | @calcom/web:start:     statusText: 'Bad Request',
calcom    | @calcom/web:start:     request: {
calcom    | @calcom/web:start:       responseURL: 'https://oauth2.googleapis.com/token'
calcom    | @calcom/web:start:     }
calcom    | @calcom/web:start:   },
calcom    | @calcom/web:start:   config: [Circular *1],
calcom    | @calcom/web:start:   code: '400'
calcom    | @calcom/web:start: }
calcom    | @calcom/web:start: error stack:
calcom    | @calcom/web:start: • gaxios.ts:158 _request
calcom    | @calcom/web:start:     node_modules/gaxios/src/gaxios.ts:158:15
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • task_queues:96 processTicksAndRejections
calcom    | @calcom/web:start:     node:internal/process/task_queues:96:5
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • oauth2client.js:174 refreshTokenNoCache
calcom    | @calcom/web:start:     node_modules/google-auth-library/build/src/auth/oauth2client.js:174:21
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • 7105.js:1116 refreshAccessToken
calcom    | @calcom/web:start:     .next/server/chunks/7105.js:1116:34
calcom    | @calcom/web:start: 
calcom    | @calcom/web:start: • 7105.js:1344 <anonymous>
calcom    | @calcom/web:start:     .next/server/chunks/7105.js:1344:3

The google credential environment variable looks like this:

GOOGLE_API_CREDENTIALS={"web":{"client_id":"<reacted>.apps.googleusercontent.com","project_id":"<redacted>","auth_uri":"https://accounts.google.com/o/oauth2/auth","token_uri":"https://oauth2.googleapis.com/token","auth_provider_x509_cert_url":"https://www.googleapis.com/oauth2/v1/certs","client_secret":"<redacted>","redirect_uris":["https://<redacted>/api/integrations/googlecalendar/callback","https://<redacted>/api/auth/callback/google"]}}

And the google console settings are:

The google cloud app is not published but in testing mode could that be a reason for this also?

[eeshaan]

Ok it seems that the restarting (or recreating) the container has nothing to do with it! Its that the google tokens cannot get refreshed. Had it running w/o restart and after roughly one week the google tokens expire

+1 on this. Even though the issue is most easily replicated with a restart, I've also observed the same behavior after some arbitrary amount of runtime (also around 1 week). Unsure on this, but I think one possibility is that the token expires when server usage spikes and the app is briefly inaccessible.

[eeshaan]

The google cloud app is not published but in testing mode could that be a reason for this also?

The same is also true in our case.

[krumware]

That's good to know!