vcluster

vcluster copied to clipboard

Is your feature request related to a problem?

No

Which solution do you suggest?

In an environment with multiple vclusters, according to documentation, using ingress controller, each vcluster can be connected to using a dns URL

vcluster connect my-vcluster -n my-vcluster --server=https://my-vcluster.example.com ...

would it be possible to use a vdir to expose the cluster to the ingress controller? for instance?

vcluster connect my-vcluster -n my-vcluster --server=https://all-my-vclusters.example.com/my-vcluster ...

Which alternative solutions exist?

No response

Additional context

No response

Looks like a reasonable and simple feature, we'll add it to the backlog. Looks like all we need to do is expose the ingress path in the values.yaml under .Values.ingress.path.

Thanks. However, it seems the CLI is only able to receive a hostname:port (at least, looking at the validation code) so prob that would also need to change to accomodate a vdir part. Pls let us know if we can help with spec or validation. Thx!

I'm looking forward to this new feature ! Can you confirm that currently we can't access different Vclusters with the same ingress object (and different paths) ? It seems not to be possible to use server:https://server.com/vcluster1 in the Vcluster kubeconfig with path /vcluster1 declared in the ingress object

Hey @rohantmp I want to work on this issue, can you assign it to me?

Yes :)

@rohantmp Results after some research.

1. Just adding some path won't work because api-server would interpret everything after /, so just having path: foo won't work in the ingress.
2. Instead, something like: path: /foo(/|$)(.*) with re-write annotation nginx.ingress.kubernetes.io/rewrite-target: "/$2" would have to be used.

The problems I am facing:

• Using minikube ingress addon with --enable-ssl-passthrough=true and values.yaml:

syncer:
  extraArgs:
  - --tls-san=my-vcluster.example.com

• ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/rewrite-target: "/$2"  
  name: vcluster-ingress
  namespace: my-vcluster
spec:
  ingressClassName: nginx 
  rules:
  - host: my-vcluster.example.com
    http:
      paths:
      - backend:
          service:
            name: my-vcluster
            port: 
              number: 443
        path: /foo(/|$)(.*)
        pathType: ImplementationSpecific

And creating kubeconfig file as:

$ vcluster connect my-vcluster -n my-vcluster --update-current=false --server=https://my-vcluster.example.com/foo

$ cat kubeconfig.yaml
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyT1RFeU1qWXpNakF3SGhjTk1qTXdPREExTURrd05USXdXaGNOTXpNd09EQXlNRGt3TlRJdwpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyT1RFeU1qWXpNakF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRTnAyNEhxcHhkdEFDSjBHam8xdlJ0WjF5cDA4a2Y0SlplbWZ0UFVlbEYKZ09GQ0oxaDhiVUtyWlF5TXVuUThLZkVwb0ZmVS92MmFJcHRwQi9FVm96b0lvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVXU3WHJVNmxJRlVlNk0vQ0dlSWFPCnIzcnZybG93Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnZTNUWlFobk5Gai9YbEFyVk9qSlE4ZlBHZGt3bUtXTnMKa29Fd3NPeitCeDRDSUg2bC93WWx0M2tTbytGbjhIQ1JJR1FGSWdGeDhZTjVUMnFJYTFMcGFrMGgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://my-vcluster.example.com/foo
  name: vcluster_my-vcluster_my-vcluster_minikube
contexts:
- context:
    cluster: vcluster_my-vcluster_my-vcluster_minikube
    namespace: default
    user: vcluster_my-vcluster_my-vcluster_minikube
  name: vcluster_my-vcluster_my-vcluster_minikube
current-context: vcluster_my-vcluster_my-vcluster_minikube
kind: Config
preferences: {}
users:
- name: vcluster_my-vcluster_my-vcluster_minikube
  user:
    client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJrVENDQVRlZ0F3SUJBZ0lJZEFLY0FGSkxRcEl3Q2dZSUtvWkl6ajBFQXdJd0l6RWhNQjhHQTFVRUF3d1kKYXpOekxXTnNhV1Z1ZEMxallVQXhOamt4TWpJMk16SXdNQjRYRFRJek1EZ3dOVEE1TURVeU1Gb1hEVEkwTURndwpOREE1TURVeU1Gb3dNREVYTUJVR0ExVUVDaE1PYzNsemRHVnRPbTFoYzNSbGNuTXhGVEFUQmdOVkJBTVRESE41CmMzUmxiVHBoWkcxcGJqQlpNQk1HQnlxR1NNNDlBZ0VHQ0NxR1NNNDlBd0VIQTBJQUJHOEVwUDlQQmpCNzlTb3cKVEVXMkZRaVBSWEpZdEt3MFF5QVlXT1BsOHNKZlZ5bHd4NUhkMVNJSVlFbUxDV3JuR2xUek5iQnUvYTkyUlh4WApmWEN5dDVxalNEQkdNQTRHQTFVZER3RUIvd1FFQXdJRm9EQVRCZ05WSFNVRUREQUtCZ2dyQmdFRkJRY0RBakFmCkJnTlZIU01FR0RBV2dCU2o3ejE5UWxyT0czRFJBSVFQSzF2Y1VFZ0J2akFLQmdncWhrak9QUVFEQWdOSUFEQkYKQWlCUUw4TG5oU3N2bWQwaWNEaHZRQXM4U21Na3ZUbjdRMUdqTUN2dVEzWjM5UUloQVBzZi8xUGtDdnlhTWNZRQo1UU1oNis0OERsQ1NyRkNxLzIrblVGL29aUU54Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkekNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdFkyeHAKWlc1MExXTmhRREUyT1RFeU1qWXpNakF3SGhjTk1qTXdPREExTURrd05USXdXaGNOTXpNd09EQXlNRGt3TlRJdwpXakFqTVNFd0h3WURWUVFEREJock0zTXRZMnhwWlc1MExXTmhRREUyT1RFeU1qWXpNakF3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFRNHFNT0hFazRQQlRrdVVvMFhueWxCeFJwMG9YTWVMenppMzY5Sm1BQ2gKTE5ha0VuNnJ6Vkk0YzVqbGQvSXFOSW02VHVKbU1MaUVHRnA3bVcyTmthS2xvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVW8rODlmVUphemh0dzBRQ0VEeXRiCjNGQklBYjR3Q2dZSUtvWkl6ajBFQXdJRFNBQXdSUUloQUl4SzBKNWcvQnY1d3dBRVVtZEZUS3dSdXZGRDl6blAKZUNIWUFZV1dMRDU4QWlCZGh4VGl4NE95QUVDRlVreG9vbWxjTjEvMkpaZVhPd3dpT1lpekZmeXE5Zz09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
    client-key-data: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1IY0NBUUVFSUdNMUMyS2kzYi9rSXg4MCtHc21DVXgxb2lvd2FHZUhpUytEbENib3kyM0NvQW9HQ0NxR1NNNDkKQXdFSG9VUURRZ0FFYndTay8wOEdNSHYxS2pCTVJiWVZDSTlGY2xpMHJEUkRJQmhZNCtYeXdsOVhLWERIa2QzVgpJZ2hnU1lzSmF1Y2FWUE0xc0c3OXIzWkZmRmQ5Y0xLM21nPT0KLS0tLS1FTkQgRUMgUFJJVkFURSBLRVktLS0tLQo=

I inspected the certs, and they look correct.

$ kubectl --kubeconfig ./kubeconfig.yaml get ns -v 10
I0805 15:16:35.901381  489516 loader.go:373] Config loaded from file:  ./kubeconfig.yaml
I0805 15:16:35.901733  489516 round_trippers.go:466] curl -v -XGET  -H "Accept: application/json;g=apidiscovery.k8s.io;v=v2beta1;as=APIGroupDiscoveryList,application/json" -H "User-Agent: kubectl/v1.26.0 (linux/amd64) kubernetes/b46a3f8" 'https://my-vcluster.example.com/foo/api?timeout=32s'
I0805 15:16:35.970986  489516 round_trippers.go:495] HTTP Trace: DNS Lookup for my-vcluster.example.com resolved to [{192.168.49.2 }]
I0805 15:16:35.971217  489516 round_trippers.go:510] HTTP Trace: Dial to tcp:192.168.49.2:443 succeed
I0805 15:16:35.971542  489516 round_trippers.go:553] GET https://my-vcluster.example.com/foo/api?timeout=32s  in 69 milliseconds
I0805 15:16:35.971572  489516 round_trippers.go:570] HTTP Statistics: DNSLookup 69 ms Dial 0 ms TLSHandshake 0 ms Duration 69 ms
I0805 15:16:35.971577  489516 round_trippers.go:577] Response Headers:
I0805 15:16:36.972204  489516 with_retry.go:241] Got a Retry-After 1s response for attempt 1 to https://my-vcluster.example.com/foo/api?timeout=32s

Re routing seems to be working:

$ curl http://my-vcluster.example.com/foo/healthz #Note http
ok%

But it doesn't work with TLS somehow 🤔 . To verify it is working otherwise, I repeated these steps with just server: https://my-vcluster.example.com and the ingress configured accordingly, and it works perfectly fine in that scenario in the very same environment. Can you please help me debug this?

@rohantmp Ping

Apologies for taking time to respond, I missed the ping . Could you provide more info about what error messages you encounter when doing this?

The example in the ingress controller docs also uses the nginx.ingress.kubernetes.io/use-regex: "true" annotation. Not sure if it defaults to true though, hmm!