Unable to push images to GCR from non GKE cluster or GKE cluster under different project

[trisberg]

Expected Behavior

I can have a build push images to GCR from cluster that is not a GKE cluster within the same project as GCR

Actual Behavior

Getting auth errors. Looks like some GCR Access Control is missing when building and pushing images from non GKE cluster or a GKE cluster that is running under a different project.

Steps to Reproduce the Problem

1. Apply Kaniko build-remplate:

kubectl apply -f https://raw.githubusercontent.com/knative/build-templates/master/kaniko/kaniko.yaml

2. Create Cloud IAM service account with 'roles/storage.admin' in the Project used for GCR and json key file

export GCP_PROJECT=$(gcloud config get-value core/project)
gcloud iam service-accounts create push-image
gcloud projects add-iam-policy-binding $GCP_PROJECT \
  --member serviceAccount:push-image@$GCP_PROJECT.iam.gserviceaccount.com \
  --role roles/storage.admin
gcloud iam service-accounts keys create \
  --iam-account "push-image@$GCP_PROJECT.iam.gserviceaccount.com" \
  $HOME/push-image.json

3. Create a Kubernetes cluster using a different project and install recent Knative build and serving.

4. Create secret and k8s service account

kubectl create secret generic kaniko-secret --from-file=$HOME/kaniko.json
kubectl annotate secret kaniko-secret build.knative.dev/docker-0=https://gcr.io
kubectl apply -f kaniko-sa.yaml

where kaniko-sa.yaml is:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: build-bot
secrets:
- name: kaniko-secret

5. Create the build, I named mine kaniko-square.yaml

apiVersion: build.knative.dev/v1alpha1
kind: Build
metadata:
  name: kaniko-build
spec:
  serviceAccountName: build-bot
  source:
    git:
      url: https://github.com/trisberg/node-kaniko-square.git
      revision: master
  template:
    name: kaniko
    arguments:
    - name: IMAGE
      value: gcr.io/[your-GCR-project]/node-kaniko-square

6. Apply the build

kubectl apply -f kaniko-square.yaml

and check the build status and logs.

Additional Info

I got this:

$ kubectl logs -c build-step-build-and-push kaniko-build-pngcq
INFO[0000] Downloading base image projectriff/node-function-invoker:0.0.9-snapshot 
ERROR: logging before flag.Parse: E1030 19:57:31.499621       1 metadata.go:159] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
ERROR: logging before flag.Parse: E1030 19:57:31.504745       1 metadata.go:142] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
2018/10/30 19:57:31 No matching credentials were found, falling back on anonymous
INFO[0000] Executing 0 build triggers                   
INFO[0000] Extracting layer 0                           
INFO[0000] Extracting layer 1                           
INFO[0002] Extracting layer 2                           
INFO[0002] Extracting layer 3                           
INFO[0002] Extracting layer 4                           
INFO[0002] Extracting layer 5                           
INFO[0005] Taking snapshot of full filesystem...        
INFO[0008] ENV FUNCTION_URI /functions/square.js        
INFO[0008] Using files from context: [/workspace/square.js] 
INFO[0008] ADD square.js ${FUNCTION_URI}                
INFO[0008] Taking snapshot of files...                  
error pushing image: failed to push to destination gcr.io/cf-sandbox-trisberg/node-kaniko-square:latest: no token in bearer response:
{"errors":[{"code":"DENIED","message":"Token exchange failed for project 'cf-sandbox-trisberg'. Caller does not have permission 'storage.buckets.get'. To configure permissions, follow instructions at: https://cloud.google.com/container-registry/docs/access-control"}]}

[nader-ziada]

/assign pivotal-nader-ziada

[nader-ziada]

For Kaniko to be able to push to a Google Cloud Registry on a different project, it needs a Kubernetes secret, which contains the auth required to push the final image.

From the Kaniko docs: How to run Kaniko on a Kubernetes cluster

you must first create the secret

kubectl create secret generic kaniko-secret --from-file=<path-to-service-account.json>

And make the Pod look like this

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=<path to Dockerfile within the build context>",
            "--context=gs://<GCS bucket>/<path to .tar.gz>",
            "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"]
    volumeMounts:
      - name: kaniko-secret
        mountPath: /secret
    env:
      - name: GOOGLE_APPLICATION_CREDENTIALS
        value: /secret/kaniko-secret.json
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: kaniko-secret

but currently in Kaniko build-template, the volume and secret are not supported

we have the following options:

• create a 2nd Kaniko template that works with an external Google Cloud Registry
• change the ApplyTemplate logic in build to somehow allow the user to append to the template
• tell users to build their own custom template?

I'm able to push to a Google Cloud Registry from a PKS cluster using the following custom template: (@trisberg if you need a workaround)

apiVersion: build.knative.dev/v1alpha1
kind: BuildTemplate
metadata:
  name: kaniko-for-external-cluster
spec:
  parameters:
  - name: IMAGE
    description: The name of the image to push
  - name: DOCKERFILE
    description: Path to the Dockerfile to build.
    default: /workspace/Dockerfile

  steps:
  - name: build-and-push
    image: gcr.io/kaniko-project/executor
    args:
    - --dockerfile=${DOCKERFILE}
    - --destination=${IMAGE}
    volumeMounts:
    - name: kaniko-secret
      mountPath: /secret
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /secret/kaniko-secret.json
  volumes:
  - name: kaniko-secret
    secret:
      secretName: kaniko-secret    

@ImJasonH @shashwathi wdyt?

[trisberg]

We already create a secret for the service account, are you saying that we need an additional secret, that seems like the wrong solution? I'm having same issue using the buildpack template as well, so it seems to be a general problem that the build doesn't use the secret provided to the service account.

[nader-ziada]

The Kaniko build-template doesn't pick up the secret unless its in the format mentioned here: https://github.com/GoogleContainerTools/kaniko#kubernetes-secret

Even if you have the secret, Kaniko doesn't read it.

Using the custom template above, I can push an image to gcr from a pks cluster

[trisberg]

It seems to use the service account secret when pushing to Docker Hub, so I wonder why GCR is different.

[nader-ziada]

In the Kaniko docs here https://github.com/GoogleContainerTools/kaniko it says there are different methods of authentication based on the target registry

[philippthun]

@trisberg

... I'm having same issue using the buildpack template as well, so it seems to be a general problem that the build doesn't use the secret provided to the service account.

I neither tried Kaniko nor GCR, but had the same issue using the buildpack template on an AWS K8s cluster using ECR as docker registry.

The reason for this behavior is that the buildpack export step uses credential helpers for GCP, AWS, and Azure by default. On AWS this helper then creates a docker configuration based on the IAM role of the EC2 instance that runs the build pod.

Disabling these helpers results in the service account's secret to be used. See https://github.com/knative/build-templates/pull/83

[stevvooe]

What's the current status on this issue? I ran into a similar problem, following the kaniko example and pushing to GCR. Is the problem here with kaniko or is this a knative issue?

[jonjohnsonjr]

Following the breadcrumbs, I think this is resolved by https://github.com/knative/build-templates/pull/87

@dlorenc is that right?

[dlorenc]

Correct, it wasn't really an issue with knative or kaniko, just the interaction in between.

[stevvooe]

@dlorenc @jonjohnsonjr Thanks for the quick response! I'll see if that works and report back.

[stevvooe]

@dlorenc That worked!