How to use a service account for CI deployments

[marcosscriven]

Running clasp login sets up a .clasprc file with a token that seems to last about a week.

Is there any way to get some kind of authentication working that could work in a headless setup like CI (E.g. GitHub Travis or Bitbucket Pipelines) please?

I looked at https://script.google.com/home/usersettings which has a switch for the API, but nothing about service tokens.

---

Note from @grant, please upvote this bug! https://issuetracker.google.com/issues/36763096

[marcosscriven]

Note I tried the directions at https://developers.google.com/identity/protocols/OAuth2ServiceAccount and using the resultant JSON key in place of the .clasprc, though this didn't seem to work.

[marcosscriven]

This seems to be related to https://github.com/google/clasp/pull/28 - but although it doesn't require a browser, it still requires interaction on the command line.

[marcosscriven]

Pinging @grant on this one (as a recent committer to https://github.com/google/clasp/blob/master/src/auth.ts).

I looked at the oauth2 client used here, and it seems there is a way to set creds in an env var: https://www.npmjs.com/package/google-auth-library#loading-credentials-from-environment-variables

It even mentions the deployment use case.

Also - over in gapps, looks like there was a PR for such a request by @gunar https://github.com/danthareja/node-google-apps-script/pull/46/files

[grant]

clasp auto-refreshes access token for any clasp command when it expires (~24h?). You should only need to clasp login once. (Unless you add scopes or change users).

~/.clasprc.json has these:

• access_token
• refresh_token

This request is for using a service account rather than a user account. I think using --ownkey should already solve this, but I'll have to check more and document it.

[marcosscriven]

@grant thanks for looking into this issue.

Any kind of auto refresh would then need to persist - in the context of CI then, one would have to check if the local .clasprc file (which CI would have to be generated from secret env vars during the build) changed, and then somehow update the env vars that contain the secrets.

--own-key seems to only be about having a .clasprc file in the local directory, not using a service account JWT key of the form:

{
  "type": "service_account",
  "project_id": "project-id-xxxxxxxxx",
  "private_key_id": "xxx",
  "private_key": "-----BEGIN PRIVATE KEY-----\nxxxx\n-----END PRIVATE KEY-----\n",
  "client_email": "xxx",
  "client_id": "xxx",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/xxxxx.iam.gserviceaccount.com"
}

[marcosscriven]

To expand, the code snippet from google-auth-library is:

export CREDS='{
  "type": "service_account",
  "project_id": "your-project-id",
  "private_key_id": "your-private-key-id",
  "private_key": "your-private-key",
  "client_email": "your-client-email",
  "client_id": "your-client-id",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://accounts.google.com/o/oauth2/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "your-cert-url"
}'

And then:

const {auth} = require('google-auth-library');
 
// load the environment variable with our keys
const keysEnvVar = process.env['CREDS'];
if (!keysEnvVar) {
  throw new Error('The $CREDS environment variable was not found!');
}
const keys = JSON.parse(keysEnvVar);
 
async function main() {
  // load the JWT or UserRefreshClient from the keys
  const client = auth.fromJSON(keys);
  client.scopes = ['https://www.googleapis.com/auth/cloud-platform'];
  await client.authorize();
  const url = `https://www.googleapis.com/dns/v1/projects/${keys.project_id}`;
  const res = await client.request({url});
  console.log(res.data);
}

So maybe either just implicitly be able to read JWT tokens of this sort in a .clasprc file, or have an explicit --jwtkey option that expects the key in an env var like this (preferred).

[campionfellin]

Hey @marcosscriven does this PR https://github.com/google/clasp/pull/223 solve your issue? It removes --ownkey in favor of --creds

This changes clasp login so that it loads those creds from a json file. Does your CI pipeline specifically need it to load from an environmental variable? We can probably make that an option as well.

However, it still requires you to select the Gmail account you're authorizing the app for. I can take a look at that sample code there and see if we can get it to work. Looks like

  await client.authorize();
  const url = `https://www.googleapis.com/dns/v1/projects/${keys.project_id}`;
  const res = await client.request({url});

may be where it authorizes without opening up in the browser.

I hope this helps! Also feel free to open your own PR, or ask more questions.

[marcosscriven]

@campionfellin - it certainly looks close. It doesn't have to be from env vars - I'm chiefly thinking about Bitbucket Cloud Pipelines https://confluence.atlassian.com/bitbucket/environment-variables-794502608.html

One could easily generate a JSON file at build time, populating the secrets from the env vars. It would be handy to avoid that step though, as show in the snippet.

[marcosscriven]

Also @campionfellin - I don't think the snippet you highlighted is anything to do with the auth - it's just an example of going on to use any of the APIs (dns in this instance).

The bit that enables it is simply const client = auth.fromJSON(keys), with the keys in the format I posted.

[campionfellin]

Hey @marcosscriven it looks like rather than using auth.fromJSON(...) I just read and parsed the file manually (https://github.com/campionfellin/clasp/blob/64b5301ccef7dcc99a2d9690c6d52db708975e08/src/auth.ts#L60). I can go ahead and change that.

However, I don't think this would solve the issue of getting the access_token or refresh_token that you need in your .clasprc.json file unless client.authorize(...) is what does that.

[campionfellin]

Hey @marcosscriven , so it does look like using what's in the sample will work. client.authorize(...) is what does the work for us. It will take some time to make those changes though.

[campionfellin]

Question for @grant : is this what we want to do by default, or add a flag for it? I am afraid of taking away the user's ability to see what scopes they are authorizing.

[marcosscriven]

@campionfellin - The scopes would have be chosen by the user while generating the service account key, so I think just working by default would be fine (so long as it was documented how to use this rather than a token).

[campionfellin]

My question is more for users who don't generate service accounts. What about a flag like --use-service-account ?

[grant]

Another flag sounds OK. I'm getting a bit confused by all the discussion here, but it seems like this is just a FR for adding another flag like clasp login --service-account and changing authorize to work with service accounts.

[marcosscriven]

@grant - I'm not clear of the purpose of 'logging in' with a service account? Logging in at the moment is just about getting a token into .clasprc - we don't need that if we've already downloaded json service account credentials.

To be clear, I would expect to be able to provide service account credentials (created according to https://developers.google.com/identity/protocols/OAuth2ServiceAccount), in either a file or env var, and for all remaining API actions to simply use client.authorize().

I think it should work fine by simply inferring behaviour from the contents of .clasprc - if it's just a token, use that. If it's credentials with a private key etc., then use that instead.

[campionfellin]

Hey @marcosscriven and @grant I've done a bit of investigation today, so here's a follow up, please correct me on any things I am misunderstanding:

1. According to here: "The Google OAuth 2.0 system supports server-to-server interactions such as those between a web application and a Google service. For this scenario you need a service account, which is an account that belongs to your application instead of to an individual end user. Your application calls Google APIs on behalf of the service account, so users aren't directly involved."

It goes on to explain that this is "2-legged OAuth", as compared to "3-legged OAuth" which can act on behalf of the user but needs user permission (like the pop-up that we currently have).

My understanding of what you want is essentially for your Bitbucket pipeline to interact with Google Services, without you having to open a page to login.

I don't think that with a Service Account or JWT this will be possible, for most clasp commands.

Anyway, here's how I tested it:

In GCP, I made a service account, with full "owner" access to the entire project. I downloaded those credentials (in the same format as you have above and same as the documentation) and used them to authenticate my API calls, like here:

https://github.com/google/clasp/blob/d807a6c72886a3cadf06bda53227fcb106f6b858/src/auth.ts#L41

But instead of the oauth2client I used the one I created as a JWT Client. Now the first command I tried was clasp list, but unfortunately got an empty array as a response. Why? Well because the service account's Drive is empty, all the scripts are located in the user's Drive. Ok, so let's clasp create. Unfortunately, you're hit with this:

Error: User has not enabled the Apps Script API. Enable it by visiting https://script.google.com/home/usersettings then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.

Since most of the clasp commands either use the Drive API or the Apps Scripts API (both of which are closely linked to the user themselves), I don't think that much can actually be done as far as using service accounts with clasp.

However, if all you really need is for your pipeline to work, there is a fairly simple solution, which I'll explain in my next comment.

[campionfellin]

So this is how I would solve your CI problem specifically, though we use Travis instead of BitBucket, it should be simple to translate.

On your local machine with some real user account (yours), use clasp login like normal and click to allow clasp access. That should save the ~/.clasprc.json file on your machine. What we did (but no longer do, for other reasons) is encrypt that file into ~/.clasprc.json.enc and then use the pipeline to decrypt it before running anything. (Here is the instructions for Travis: https://docs.travis-ci.com/user/encrypting-files/)

If you find that BitBucket doesn't allow that with files, but rather environmental variables, it would be a pretty simple change here:

https://github.com/google/clasp/blob/d807a6c72886a3cadf06bda53227fcb106f6b858/src/auth.ts#L61

To either read from a file or from ENV. However, Service Accounts and JWT will still not work.

Let me know if this at least unblocks you, or if you have further questions or can help me understand your situation better.

[marcosscriven]

@campionfellin Thanks for looking into this - as it happens, I'm not looking for impersonation in my case. There's two flavours of that in OAuth - there's the 3LO (Three-legged Oauth), which allows impersonation with a pre-shared key, but still requires user interaction for the user to accept. There's a much lesser known 2LOi (Two-legged Oauth with Impersonation) - but I don't see that mentioned anywhere in Google's docs.

Anyway - for me, I do have an account I setup just for services, and I give that account the rights to run scripts and access drives that way.

All I need here is for the OAuth client clasp uses to be setup with the service account json key (as in your penultimate post), and that'll work for me.

Regardless though - the method you specify for working around it (if one needed to) is what I considered to start with, but noted the token there has about a week's validity. At which point clasp can refresh the token with the URL it has in .clasprc - but that gets written to disk.

So this can't be one way in CI - it too would have to store (securely) any changes to the token that clasp made right?

[marcosscriven]

@campionfellin - I just wasted some hours on using the service account (which should work). The crucial step (even for a service account with 'Domain-wide delegation'), was that I still had to go to the script project and 'share' it with the service user email (of the form [email protected]).

I'm pretty sure that last step is not meant to be necessary. Maybe related to https://issuetracker.google.com/issues/36763096?

@grant - As you're a member of the Google team on Github, is there any chance you can investigate this please? There's a lot of confusion around service accounts and the App Scripts API.

[marcosscriven]

To clarify, using the Python Google OAuth 2 client, this works - so long as I've 'shared' the script with the service account email:

from google.oauth2 import service_account
import googleapiclient.discovery
import json
import os

SCOPES = ['https://www.googleapis.com/auth/script.projects']
SERVICE_KEY = json.loads(os.environ['SERVICE_KEY'])

credentials = service_account.Credentials.from_service_account_info(SERVICE_KEY , scopes=SCOPES)

script = googleapiclient.discovery.build('script', 'v1', credentials=credentials)
response = script.projects().get(scriptId='myscriptid').execute()
print response

EDIT - So while this works, trying:

response = script.projects().updateContent(scriptId='myscriptid', body=body).execute()

Suddenly gives me a 403:

<HttpError 403 when requesting https://script.googleapis.com/v1/projects/myscriptid/content?alt=json returned "User has not enabled the Apps Script API. Enable it by visiting https://script.google.com/home/usersettings then retry.

Which is very peculiar given the the API is clearly being used during get operation...

[marcosscriven]

Trying to use delegation-wide service account the right way (E.g. without the 'sharing' hack I mentioned), even just reading the project fails:

SCOPES = ['https://www.googleapis.com/auth/script.projects', 'https://www.googleapis.com/auth/drive']
SERVICE_KEY = json.loads(os.environ['SERVICE_KEY'])

credentials = service_account.Credentials.from_service_account_info(SERVICE_KEY, scopes=SCOPES)
delegated_credentials = credentials.with_subject('<email>)
script = googleapiclient.discovery.build('script', 'v1', credentials=delegated_credentials)

response = script.projects().get(scriptId='<scriptId>').execute()

Fails with:

google.auth.exceptions.RefreshError: ('unauthorized_client: Client is unauthorized to retrieve access tokens using this method.', u'{\n  "error" : "unauthorized_client",\n  "error_description" : "Client is unauthorized to retrieve access tokens using this method."\n}')

Despite ensuring those API scopes have been authorized for that client ID as per https://developers.google.com/api-client-library/python/auth/service-accounts.

[marcosscriven]

Note I asked about this on Stack Overflow too https://stackoverflow.com/questions/51049548/how-can-i-publish-a-google-app-script-using-a-domain-wide-delegation-service-acc

[grant]

Hey @marcosscriven, thanks for all the investigation. I too want this feature and to reduce friction around this and a bunch of other setup.

Please upvote the linked bug and this issue so I can ask the Apps Script team to prioritize this. If there's a workaround that clasp can promote in the meantime, perhaps we can make that setup easy.

[marcosscriven]

@grant - I don't see any voting options there, but I've commented on it.

It says it's 'blocked by' https://issuetracker.google.com/issues/26400743, but I don't have view permissions on that.

[grant]

@marcosscriven It looks like this issue is being triaged by the Apps Script team. I've asked the team for an update on the issue. Unfortunately, it's a lot easier to change features in clasp than modify anything with the Apps Script tool/product.

[marcosscriven]

@grant good news! Let’s hope it gets fixed. Thanks for following for following up.

[grant]

To be honest, I don't expect this to be fixed by the team in the next 3 months, but we will see.

[aandis]

Any update on this? Is it possible to authenticate clasp using a service account credential?

[aandis]

for reference, this is how it works in gcloud

gcloud auth activate-service-account --key-file service-account-credentials.json