Unable to connect to vcluster via Ingress

[alezander86]

What happened?

I have two EKS clusters: eks1 and eks2.

I want to use a Tekton pipeline to deploy a vcluster on eks2 using a manifest (included below).

# My vcluster.yaml / values.yaml here
vcluster:
  controlPlane:
    ingress:
      enabled: true
      host: vcluster.domain.com
    proxy:
      extraSANs:
      - vcluster.domain.com
    statefulSet:
      image:
        tag: "0.24.0"
      persistence:
        volumeClaim:
          retentionPolicy: Delete
  sync:
    fromHost:
      ingressClasses:
        enabled: true
    toHost:
      ingresses:
        enabled: true
  exportKubeConfig:
    server: https://vcluster.domain.com

In the second part of the pipeline, I want to:

connect to the vcluster via Ingress from eks1,

deploy my application inside the vcluster,

and run integration tests against it.

However, after successfully deploying the vcluster, when I try to connect to it from eks1 using the generated kubeconfig, I get the following error:

kubectl --kubeconfig ./kubeconfig.yaml get namespaces E0411 20:54:43.937345 91162 memcache.go:265] couldn't get current server API group list: Get "https://vcluster.domain.com/api?timeout=32s": tls: failed to verify certificate: x509: certificate signed by unknown authority vcluster connect vcluster --update-current=false --server=https://vcluster.domain.com Flag --update-current has been deprecated, please use "vcluster connect --print" to write the kubeconfig of the virtual cluster to stdout. 21:43:31 warn There is a newer version of vcluster: v0.24.1. Run vcluster upgrade to upgrade to the newest version.

21:43:32 done vCluster is up and running 21:43:32 done Virtual cluster kube config written to: ./kubeconfig.yaml

• Use kubectl --kubeconfig ./kubeconfig.yaml get namespaces to access the vcluster ➜ vcluster kubectl --kubeconfig ./kubeconfig.yaml get pods --all-namespaces --insecure-skip-tls-verify E0412 21:43:59.053160 42140 memcache.go:265] couldn't get current server API group list: unknown E0412 21:43:59.096156 42140 memcache.go:265] couldn't get current server API group list: unknown

kubectl --kubeconfig ./kubeconfig.yaml auth can-i create selfsubjectreviews --insecure-skip-tls-verify E0412 20:42:58.334371 34272 memcache.go:265] couldn't get current server API group list: unknown E0412 20:42:58.376349 34272 memcache.go:265] couldn't get current server API group list: unknown Warning: the server doesn't have a resource type 'selfsubjectreviews'

Error from server (Forbidden): selfsubjectaccessreviews.authorization.k8s.io is forbidden: User "system:anonymous" cannot create resource "selfsubjectaccessreviews" in API group "authorization.k8s.io" at the cluster scope

What did you expect to happen?

After deploying the vcluster and exposing it via Ingress, I expected to be able to access the Kubernetes API at https://vcluster.domain.com from eks1 using the generated kubeconfig, and use kubectl to interact with the vcluster by ingress as if it were a regular Kubernetes cluster.

How can we reproduce it (as minimally and precisely as possible)?

Deploy a vcluster on eks2 using Tekton and the provided manifest.

Expose the vcluster’s API via an Ingress controller (e.g., NGINX).

From eks1, attempt to connect to vcluster.domain.com using kubectl and the kubeconfig from vcluster connect.

Anything else we need to know?

How can I bypass or resolve the x509: certificate signed by unknown authority error (especially in a CI context)?

What is the recommended way to connect from eks1 to a vcluster running in eks2?

Host cluster Kubernetes version

$ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.10-eks-bc803b4```

</details>


### vcluster version

<details>

```console
$ vcluster --version
vcluster version 0.23.0

VCluster Config

# My vcluster.yaml / values.yaml here
vcluster:
  controlPlane:
    ingress:
      enabled: true
      host: vcluster.domain.com
    proxy:
      extraSANs:
      - vcluster.domain.com
    statefulSet:
      image:
        tag: "0.24.0"
      persistence:
        volumeClaim:
          retentionPolicy: Delete
  sync:
    fromHost:
      ingressClasses:
        enabled: true
    toHost:
      ingresses:
        enabled: true
  exportKubeConfig:
    server: https://vcluster.domain.com

[FabianKramm]

Hey @alezander86 ! Thanks for creating this issue, the problem seems to be that your ingress controller is not doing any ssl passthrough as described here: https://www.vcluster.com/docs/vcluster/manage/accessing-vcluster#expose-vcluster . If your ingress controller doesn't support this, a way to get around the tls problem will be to use exportKubeConfig.insecure: true, but then you will need to use a vCluster sa token to connect to it, check the section in the docs above "Ingress without SSL passthrough"

[cbron]

Closing as stale.