Unauthorized when accessing vcluster via Ingress without SSL-Passthrough

[FrankAnk]

What happened?

I created a vcluster (with extraArgs "tls-san" and "out-kube-config-server") and also setup an Ingress without SSL-Passthrough, then used the "vcluster connect ... --server=https://myvcluster.vcluster.itservices.tank.local --service-account=developer --cluster-role=cluster-admin --update-current=false

When i try to access the cluster with "kubectl --kube-config ./kubeconfig.yaml get namesapces" i get the message "error: You must be logged in to the server (Unauthorized)"

What did you expect to happen?

i expected to see the list of namespaces in the newly created vcluster.

How can we reproduce it (as minimally and precisely as possible)?

#!/bin/bash
CLUSTER_NAME=myvcluster
CLUSTER_FQDN=${CLUSTER_NAME}.vcluster.itservices.tank.local
VALUES_FILE=./${CLUSTER_NAME}-values.$$.yaml

cat > ${VALUES_FILE} <<EOF
syncer:
  extraArgs:
  - --tls-san=${CLUSTER_FQDN}
  - --out-kube-config-server=https://${CLUSTER_FQDN}
EOF

vcluster create ${CLUSTER_NAME} \
  --namespace vcluster-${CLUSTER_NAME} \
  --create-namespace=true \
  --connect=false \
  --expose-local=false \
  --extra-values=${VALUES_FILE}
  
kubectl create --namespace vcluster-${CLUSTER_NAME} -f - <<EOF
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: ${CLUSTER_NAME}
spec:
  rules:
  - host: ${CLUSTER_FQDN}
    http:
      paths:
      - backend:
          serviceName: ${CLUSTER_NAME}
          servicePort: 443
        path: /
        pathType: ImplementationSpecific
  tls:
  - hosts: 
    - ${CLUSTER_FQDN}
    secretName: ${CLUSTER_NAME}-tls
EOF

vcluster connect ${CLUSTER_NAME} \
  --server https://${CLUSTER_FQDN} \
  --kube-config-context-name=${CLUSTER_NAME} \
  --kube-config=./${CLUSTER_NAME}.yaml \
  --service-account=kube-system/developer \
  --cluster-role=cluster-admin \
  --insecure \
  --update-current=false

Now access the vcluster:

kubectl --kubeconfig ./myvcluster.yaml get namespaces

Anything else we need to know?

The Logs of vcluster pod:

I0711 11:52:57.451347       1 syncer.go:66] event: coredns-7c68d48c87-79ttx-x-kube-system-x-myvcluster.1700c22f861bf13e: update virtual event kube-system/coredns-7c68d48c87-79ttx.1700c22f861bf13e
E0711 11:54:36.958766       1 webhook.go:154] Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:vcluster-myvcluster:vc-myvcluster" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope 
E0711 11:54:36.958982       1 authentication.go:63] "Unable to authenticate the request" err="[[invalid bearer token, token audiences [\"https://kubernetes.default.svc.cluster.local/\" \"https://kubernetes.default.svc/\" \"https://kubernetes.default/\"] is invalid for the target audiences [\"unknown\"]], tokenreviews.authentication.k8s.io is forbidden: User \"system:serviceaccount:vcluster-myvcluster:vc-myvcluster\" cannot create resource \"tokenreviews\" in API group \"authentication.k8s.io\" at the cluster scope]"

@matskiv was able to give me a workaround by using an another kubernetes version (v1.20) in the vcluster. It seems to be a problem when running vcluster on a Host version v1.18 or v1.19 and i am running on v1.18

Changing my script to using Version 1.20 solved my problem but others might not be able to run a newer version.

vcluster create ${CLUSTER_NAME} \
  --namespace vcluster-${CLUSTER_NAME} \
  --create-namespace=true \
  --connect=false \
  --kubernetes-version=1.20 \
  --expose-local=false \
  --extra-values=${VALUES_FILE}

Host cluster Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.12", GitCommit:"7cd5e9086de8ae25d6a1514d0c87bac67ca4a481", GitTreeState:"clean", BuildDate:"2020-11-12T09:11:15Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Host cluster Kubernetes distribution

On Prem Kubernetes v1.18.12

vlcuster version

$ vcluster --version
vcluster version 0.10.2

$ vcluster --version
vcluster version 0.11.0-alpha.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k3s

OS and Arch

OS: 
Arch:

[FabianKramm]

@FrankAnk thanks for creating this issue! Mhh seems like this is a problem with the audiences we generate for the token. Not sure if we can fix that for v1.18, but we will investigate.

[justpolidor]

Hello, are you sure this is only affecting Kubernetes < 1.20? I got the same issue when using Kubernetes 1.24.1 as host cluster and Kubernetes 1.23.7-k3s1 (it was working on a cluster upgraded from 1.23 to 1.24 but on this cluster , created from scratch with Kubernetes 1.24, it's not working.)

Failed to make webhook authenticator request: tokenreviews.authentication.k8s.io is forbidden: User "system:serviceaccount:vcluster-myvcluster:vc-myvcluster" cannot create resource "tokenreviews" in API group "authentication.k8s.io" at the cluster scope

Then I tried to add the grant to the cluster role:

  - apiGroups: ["authentication.k8s.io"]
    resources: ["tokenreviews"]
    verbs: ["create", "delete", "patch", "update", "get", "watch", "list"]

But then from the syncer logs I got:

E0905 10:21:08.800876       1 authentication.go:63] "Unable to authenticate the request" err="[[invalid bearer token, square/go-jose: error in cryptographic primitive], [invalid bearer token, square/go-jose: error in cryptographic primitive, webhook authentication failed]]

Could this be related to the ServiceAccount Token change that took place in Kubernetes 1.24?

[justpolidor]

Nevermind, the second issue that I got from the syncer was because ArgoCD was trying to authenticate with a wrong configuration. At the end I just needed to add that rbac grant.

[satishweb]

Move this comment as new issue: https://github.com/loft-sh/vcluster/issues/893

[matskiv]

@satishweb Sorry for the late reply. Please create a new GH issue if you are still experiencing the problem you described. I don't think it is related to this issue, as this one is specifically about some older k8s version and missing API.