GKE - v1beta2 403 forbidden

[edenreich]

What happened?

Trying to create a minimal GKE cluster on GCP with the following configurations:

---
apiVersion: container.gcp.crossplane.io/v1beta2
kind: Cluster
metadata:
  name: default-cluster
  labels:
    managed-by: crossplane
spec:
  forProvider:
    initialClusterVersion: "1.21.10-gke.2000"
    location: europe-west3
  providerConfigRef:
    name: provider-config-gcp

Fails with:

cannot create GKE cluster: googleapi: Error 403: Retry budget exhausted (5 attempts): Google Compute Engine: Required 'compute.regions.get' permission for 'projects/<project-id>/regions/europe-west3'., forbidden

How can we reproduce it?

1. Create a k3d cluster:

k3d cluster create

2. Install crossplane using helm:

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update
helm install crossplane --namespace crossplane-system --create-namespace crossplane-stable/crossplane

3. Create a service account for crossplane in GCP:

gcloud iam service-accounts create [email protected] --display-name='Crossplane' --description='Service account for managing resources using crossplane'

4. Grant this service account owner permissions on the project level(on production you want to give it only roles which you think it needs):

declare -a roles=(
  "owner"
); \
for role in "${roles[@]}"; \
do \
    gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT --member="serviceAccount:crossplane@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com" --role="roles/${role}"; \
done;

5. Create a key from that service account:

gcloud iam service-accounts keys create ./crossplane.json --iam-account=crossplane@$GOOGLE_CLOUD_PROJECT.iam.gserviceaccount.com

6. Create a kubernetes secret in crossplane-system namespace:

kubectl create secret generic --namespace crossplane-system --from-file=service_account.json=./crossplane.json provider-gcp-credentials

7. Deploy GCP provider:

cat <<EOF | kubectl apply -f -
---
apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-gcp
spec:
  package: crossplane/provider-gcp:alpha
EOF

And the provider configurations:

cat <<EOF | kubectl apply -f -
---
apiVersion: gcp.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: provider-config-gcp
spec:
  projectID: ${GOOGLE_CLOUD_PROJECT}
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: provider-gcp-credentials
      key: service_account.json
EOF

8. Create an example GKE cluster in GCP:

cat <<EOF | kubectl apply -f -
---
apiVersion: container.gcp.crossplane.io/v1beta2
kind: Cluster
metadata:
  name: default-cluster
  labels:
    managed-by: crossplane
spec:
  forProvider:
    initialClusterVersion: "1.21.10-gke.2000"
    location: europe-west3
  providerConfigRef:
    name: provider-config-gcp
EOF

What environment did it happen in?

Crossplane version: v1.7.1 Kubernetes client version: v1.21.1 Kubernetes server version: v1.22.7+k3s1 Kubernetes distribution: Rancher - k3s