Auto-refreshing stale OAuth token.

[michaelhogersnplm]

Hi there,

We discussed the OAuth token before in the googleCloudStorageR repo, but given that it has to do with authentication I figured this repo is a better spot to discuss it.

Steps to reproduce the problem

When using the upload function of googleCloudStorageR (gcs_upload) I encounter a long freeze during which an attempt is made to refresh the OAuth token. This happens consistently after 60 minutes after the googleCloudStorageR is loaded with it's respective environment variables.

Verbose Setting

I did not run the verbose option before, given our setup and the 60 minute period for the bug to occur it is not straightforward to do that. Happy to do it if it helps significantly with debugging.

Session Info

I cannot share the entire sessionInfo(), but hopefully the following is helpful already.

R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

> packageVersion('googleAuthR')
[1] ‘1.1.1.9000’
> packageVersion('googleCloudStorageR')
[1] ‘0.5.1.9000’
> packageVersion('gargle')
[1] ‘0.4.0.9000’

Best regards,

Michael

[MarkEdmondson1234]

Which issue were you referring to before? Was it this one? https://github.com/cloudyr/googleCloudStorageR/issues/87

It should be solved, so I will need to see some example code where it occurs to help further, and sessionInfo() would still help to see all the dependencies loaded. If you use the JSON credentials file and set via GCS_AUTH_FILE there should be no refresh request needed at all, so its probably something to do with that.

[michaelhogersnplm]

Hi @MarkEdmondson1234 ,

It was indeed that issue. Please see the sessionInfo() below.

> sessionInfo()
R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS: /usr/lib/openblas-base/libblas.so.3
LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.5.3

Our program first loads googleCloudStorageR, then calls googleAuthR::gar_auth_service with scope = getOption("googleAuthR.scopes.selected") and finally sets a global bucket with googleCloudStorageR::gcs_global_bucket.

The issue often/only seemed to happen with googleCloudStorageR::gcs_upload.

We think we fixed it now by manually authenticating every time before using gcs_upload.

Best regards,

Michael Hogers

[yonirab]

I am seeing the exact same issue. Here's my sessionInfo():

> sessionInfo()
R version 3.6.1 (2019-07-05)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C
 [9] LC_ADDRESS=C               LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

loaded via a namespace (and not attached):
[1] compiler_3.6.1
> packageVersion('googleCloudStorageR')
[1] ‘0.5.1’
> packageVersion('googleAuthR')
[1] ‘1.1.1’
> packageVersion('gargle')
[1] ‘0.3.1’

[MarkEdmondson1234]

I think this occurs if an old user created oauth token is in the same directory (such as .httr-oauth) - googleCloudStorageR is migrated to "new" gargle based auth so the JSON credentials should only be used where possible in particular when on a server, as that should have no timeout.

[yonirab]

@MarkEdmondson1234 It seems that we have the following env vars defined, pointing to credentials files that do indeed exist in the same directory:

GCS_AUTH_FILE='/secrets/sharedassets/credentials.json'
GAR_CLIENT_JSON='/secrets/sharedassets/credentials_sa_id'

When library(googleCloudStorageR) gets called, we see the following log:

Successfully auto-authenticated via /secrets/sharedassets/credentials.json

Over 2 hours later, when gcs_upload() is first called, we see:

Auto-refreshing stale OAuth token.
Request Status Code: 408
Error : lexical error: invalid char in json text.
<!DOCTYPE html> <html lang=en> 
(right here) ------^

Adding a call to gcs_auth(Sys.getenv('GCS_AUTH_FILE')) before calling gcs_upload() (as suggested by @michaelhogersnplm ) seems to solve the problem. However, per your comment above, it sounds like the correct solution would be to get rid of GAR_CLIENT_JSON and the file it points to. Is that correct? Thanks!!

[MarkEdmondson1234]

However, per your comment above, it sounds like the correct solution would be to get rid of GAR_CLIENT_JSON and the file it points to. Is that correct?

No that should be fine as is. I was referring to a user generated token that are usually called something like .httr-oauth

I thought you wouldn't need a refresh on tokens from JSON, but perhaps thats wrong given your example. I'll try a test myself, and perhaps include a re-auth within the function calls to help make it easier to work with.

[yonirab]

Thanks. We don't have a .httr-oauth file in that directory (or any other credentials files other than those referenced by GCS_AUTH_FILE and GAR_CLIENT_JSON).