vcluster connect via Ingress method does not work even with tls-san option

[ksingh7]

What happened?

I wanted to create a vcluster and access it using kubectl CLI after exporting kubeconfig file. For this i followed this guide line-by-line https://www.vcluster.com/docs/operator/external-access#ingress

vcluster does not connect via kubectl CLI

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl --kubeconfig ./kubeconfig.yaml get namespaces
error: the server doesn't have a resource type "namespaces"

What did you expect to happen?

After following the guide and setting up ingress, i expect to run all kubectl commands against the vcluster without exporting the kubeconfig file everything with every command

How can we reproduce it (as minimally and precisely as possible)?

Below you can find the ingress.yaml and values.yaml

  505  kubectl create ns my-vcluster
  506  kubectl create -f ingress2.yaml
  508  vcluster create my-vcluster -n my-vcluster -f values.yaml
  513  vcluster connect my-vcluster -n my-vcluster --server=https://my-vcluster.ksingh.localhost --service-account admin --cluster-role cluster-admin --insecure
  514  vcluster connect my-vcluster -n my-vcluster -- kubectl get ns
  515  kubectl --kubeconfig ./kubeconfig.yaml get namespaces
  516  kubectl --kubeconfig ./kubeconfig.yaml get ns

Anything else we need to know?

Here are full logs

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl get nodes
NAME                   STATUS   ROLES                  AGE    VERSION
lima-rancher-desktop   Ready    control-plane,master   142m   v1.23.5+k3s1
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl get ns
NAME              STATUS   AGE
default           Active   143m
kube-system       Active   143m
kube-public       Active   143m
kube-node-lease   Active   143m
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$


[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl create ns my-vcluster
namespace/my-vcluster created
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ cat ingress2.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
  name: vcluster-ingress
  namespace: my-vcluster
spec:
  rules:
  - host: my-vcluster.ksingh.localhost
    http:
      paths:
      - backend:
          service:
            name: my-vcluster
            port:
              number: 443
        path: /
        pathType: ImplementationSpecific
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$


[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl create -f ingress2.yaml
ingress.networking.k8s.io/vcluster-ingress created
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl get ingress -n my-vcluster
NAME               CLASS    HOSTS                          ADDRESS   PORTS   AGE
vcluster-ingress   <none>   my-vcluster.ksingh.localhost             80      8s
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$


[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ ping my-vcluster.ksingh.localhost
PING localhost (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.050 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.073 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.063 ms
64 bytes from 127.0.0.1: icmp_seq=3 ttl=64 time=0.119 ms
^C
--- localhost ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.050/0.076/0.119/0.026 ms
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$


[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ cat values.yaml
syncer:
  extraArgs:
  - --tls-san=my-vcluster.ksingh.localhost
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$



[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ vcluster create my-vcluster -n my-vcluster -f values.yaml

[info]   execute command: helm upgrade my-vcluster vcluster --repo https://charts.loft.sh --version 0.8.0 --kubeconfig /var/folders/cv/fg73pmjs3fl_w_kf0pcbzyz40000gn/T/1509164872 --namespace my-vcluster --install --repository-config='' --values /var/folders/cv/fg73pmjs3fl_w_kf0pcbzyz40000gn/T/269191084 --values values.yaml
[done] √ Successfully created virtual cluster my-vcluster in namespace my-vcluster.
- Use 'vcluster connect my-vcluster --namespace my-vcluster' to access the virtual cluster
- Use `vcluster connect my-vcluster --namespace my-vcluster -- kubectl get ns` to run a command directly within the vcluster
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ vcluster connect my-vcluster --namespace my-vcluster -- kubectl get ns
NAME              STATUS   AGE
default           Active   14s
kube-system       Active   14s
kube-public       Active   14s
kube-node-lease   Active   13s
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ vcluster connect my-vcluster -n my-vcluster --server=https://my-vcluster.ksingh.localhost --service-account admin --cluster-role cluster-admin --insecure
[info]   Create service account token for kube-system/admin
Forwarding from 127.0.0.1:10489 -> 8443
Forwarding from [::1]:10489 -> 8443
Handling connection for 10489
[done] √ Created service account kube-system/admin
[done] √ Created cluster role binding for cluster role cluster-admin
[info]   Use `vcluster connect my-vcluster -n my-vcluster -- kubectl get ns` to execute a command directly within this terminal
[done] √ Virtual cluster kube config written to: ./kubeconfig.yaml. You can access the cluster via `kubectl --kubeconfig ./kubeconfig.yaml get namespaces`
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$


[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ echo $KUBECONFIG

[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ kubectl --kubeconfig ./kubeconfig.yaml get namespaces
error: the server doesn't have a resource type "namespaces"
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ cat kubeconfig.yaml
apiVersion: v1
clusters:
- cluster:
    insecure-skip-tls-verify: true
    server: https://my-vcluster.ksingh.localhost
  name: local
contexts:
- context:
    cluster: local
    namespace: default
    user: user
  name: Default
current-context: Default
kind: Config
preferences: {}
users:
- name: user
  user:
    token: eyJhbGciOiJSUzI1NiIsImtpZCI6InJLYkpyY3AyeTRtenZuNEtpY09KU3Q3ZVB1ajdvNkdvZ2Q1bTNvM2JmVG8ifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdCJdLCJleHAiOjE5Njc0ODI3NjYsImlhdCI6MTY1MjEyMjc2NiwiaXNzIjoiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Imt1YmUtc3lzdGVtIiwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImFkbWluIiwidWlkIjoiZWRhZmVmNGYtYzA1Ny00NjM2LTlmYWYtZmNmYWFkZjE3ZDFiIn19LCJuYmYiOjE2NTIxMjI3NjYsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTphZG1pbiJ9.XxajAxDQN_uu8ptW21oCiFgGp28j9qE4KYEyjS_ncTsBYaQa0flptOhYOIhIpjELsGzRzrNSYHYrLPPZKIyOPbLNN3IN40j5G_wcNrRDrI6Wm95YfD4sY1hvSfLYif-Hm9RZ1l6Y_wBEEKlBBWqH6b16qkyXQYkqlYdkfw4dpYjJtn1hZjNevl6AWNDYbjeew2QfBL7x4Dlu9zbknYALjVirpQSUm9-6vDGWOAr4Jrnq7Aafj2-w8Viml5HZyZzP1kQca4yWzBqOn8UJBhdweJcHSkeW95CPxU2x-3kwj_pX6n4peUb2Yruevoxl4oIDLLPI5iriEsxyf2lpPFUg7Q
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ oc get nodes
NAME                   STATUS   ROLES                  AGE   VERSION
lima-rancher-desktop   Ready    control-plane,master   14m   v1.23.5+k3s1
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$ oc get ingress -A
NAMESPACE     NAME               CLASS    HOSTS                          ADDRESS   PORTS   AGE
my-vcluster   vcluster-ingress   <none>   my-vcluster.ksingh.localhost             80      10m
[rancher-desktop|default] karasing-mac:~/git/vcluster/ingress$

Host cluster Kubernetes version

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.5", GitCommit:"5c99e2ac2ff9a3c549d9ca665e7bc05a3e18f07e", GitTreeState:"clean", BuildDate:"2021-12-16T08:38:33Z", GoVersion:"go1.16.12", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5+k3s1", GitCommit:"313aaca547f030752788dce696fdf8c9568bc035", GitTreeState:"clean", BuildDate:"2022-03-31T01:02:40Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

Host cluster Kubernetes distribution

k3s using Rancher Desktop

vlcuster version

$ vcluster --version
vcluster version 0.8.0

Vcluster Kubernetes distribution(k3s(default)), k8s, k0s)

k3s

OS and Arch

OS:  MacOS
Arch: Intel

[ksingh7]

FYI this error: the server doesn't have a resource type "namespaces" error has started once i upgraded to 0.80 vcluster. Previously i was getting error of certificate mismatch which i am not able to reproduce, but that was the original problem

[pratikjagrut]

Hi, @ksingh7 sorry for the delayed reply. What ingress controller are you using?

[marwenguesmii]

@pratikjagrut Hi, I have the same problem with aws load balancer controller

[marwenguesmii]

@pratikjagrut I can provide you with all the information you need

[matskiv]

@Marwennnne and/or @ksingh7 are you able to provide logs from the vcluster pod? Both "syncer" and "vcluster" containers. Ideally delete the pod, wait until it comes up, reproduce the issue with kubectl get, and capture all logs of the container.

[lknite]

I'm seeing similar:

error: the server doesn't have a resource type "services"

In my case, I jumped to using the vcluster-k8s values file deploying my cluster with (via argocd):

    repoURL: https://charts.loft.sh
    targetRevision: 0.10.2
    chart: vcluster-k8s

    helm:
      releaseName: vc-non
      parameters:
      #- name: vcluster.image
      #  value: "rancher/k3s:v1.24.3-k3s1"
      #- name: syncer.extraArgs[0]
      #  value: "--tls-san=vc-non.root.k.home.net"
      - name: ingress.enabled
        value: "true"
      - name: ingress.ingressClassName
        value: "nginx"
      - name: ingress.host
        value: vc-non.root.k.home.net
      - name: ingress.annotations.cert-manager\.io\/issuer
        value: "cluster-adcs-issuer"
      - name: ingress.annotations.cert-manager\.io\/issuer-kind
        value: "ClusterAdcsIssuer"
      - name: ingress.annotations.cert-manager\.io\/issuer-group
        value: "adcs.certmanager.csf.nokia.com"

Everything starts up as expected. I can use 'vcluster connect vc-non' and then run commands from another shell. With my annotations I also get:

• an ingress to use with connecting to the vcluster
• valid certificates for the ingress (though the ingress generated doesn't appear to use them)

However, if I extract the kubeconfig and try to access:

vcluster connect $NAME -n $NAMESPACE --update-current=false --kube-config=$OUTPUT --server=https://$NAME.root.k.home.net --kube-config-context-name=$NAME-admin

I get:

$ k --kubeconfig ~/.kube/available/vc-non-admin.conf get nodes
Unable to connect to the server: x509: certificate is valid for ingress.local, not vc-non.root.k.home.net

If I add on '--insecure' to the vcluster connect command and then try to use the kubeconfig file:

$ k --kubeconfig ~/.kube/available/vc-non-admin.conf get nodes
error: the server doesn't have a resource type "nodes"

All commands seemingly:

error: the server doesn't have a resource type "namespaces"
error: the server doesn't have a resource type "services"

Additional

$ k version -o yaml
clientVersion:
  buildDate: "2022-05-24T12:26:19Z"
  compiler: gc
  gitCommit: 3ddd0f45aa91e2f30c70734b175631bec5b5825a
  gitTreeState: clean
  gitVersion: v1.24.1
  goVersion: go1.18.2
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-07-13T14:23:26Z"
  compiler: gc
  gitCommit: aef86a93758dc3cb2c658dd9657ab4ad4afc21cb
  gitTreeState: clean
  gitVersion: v1.24.3
  goVersion: go1.18.3
  major: "1"
  minor: "24"
  platform: linux/amd64

$ helm version
version.BuildInfo{Version:"v3.9.0", GitCommit:"7ceeda6c585217a19a1131663d8cd1f7d641b2a7", GitTreeState:"clean", GoVersion:"go1.17.5"}

[lknite]

Ok, I re-read the instructions and added '--enable-ssl-passthrough=true' to my nginx-controller and now everything is working perfectly. Recommend the original poster give that a try if they haven't already.

Also, I removed the '--insecure' when generating the kubeconfig, not needed anymore. Works without it.