Webterminals: Shoot-terminal wrong attach info

[JensAc]

Hi,

we want to enable the webterminals feature in our Gardener setup. So far we were able to get the terminal feature for the (virtual) garden cluster running. For this, we needed to set the gardenTerminalHost.apiServerIngressHost to our api-server ingress host, since our dashboard wanted us to connect to null.ingress..., otherwise. Now we are trying to enable the shoot terminals, but run into the issue that the websocket our browser want to connect to (k-some-hast.ingress....) is different from the api-server-url of the shoot (api.shoot.namespace...). Of course, we cannot simple specify an api-server ingress host in this case, but we are stuck with this issue. Do you have some idea what were are missing.

Currently our dashboard (deployed via helm) comes with these values:

ingress:                                                          
  annotations:                                                    
    cert-manager.io/cluster-issuer: letsencrypt-staging           
frontendConfig:                                                   
  seedCandidateDeterminationStrategy: MinimalDistance             
  features:                                                       
    terminalEnabled: true                                         
terminal:                                                         
  container:                                                      
    image: eu.gcr.io/gardener-project/gardener/ops-toolbelt:latest
  gardenTerminalHost:                                             
    apiServerIngressHost: OUR-HOST
    secretRef:                                                    
      namespace: garden                                           
      labelSelector:                                              
      - runtime=gardenTerminalHost                                
  garden:                                                         
    operatorCredentials:                                          
      secretRef:                                                  
        name: kubeconfig                                          
        namespace: terminal-system                                
                                                                  
  bootstrap:                                                      
    disabled: false                                               
    shootDisabled: false                                          
    seedDisabled: false                                           
    gardenTerminalHostDisabled: true                              
    apiServerIngress:                                             
      annotations:                                                
        cert-manager.io/cluster-issuer:  letsencrypt-staging                         
        cert.gardener.cloud/purpose: managed                      
        kubernetes.io/ingress.class: nginx                        
        nginx.ingress.kubernetes.io/backend-protocol: HTTPS       

Our shoot-api-servers are exposed as gateway resources and not as ingresses (we are running Gardener v1.56 with ManagedIstio and ApiServerSNI enabled, which became the default recently to the best of our knowledge).

[petersutter]

Hi @JensAc, please checkout this document https://github.com/gardener/dashboard/blob/master/docs/concepts/webterminals.md#browser-trusted-certificates-for-kube-apiservers

The terminal feature requires a browser trusted certificate for the kube-apiserver (more details in the mentioned doc). In your terminal bootstrapping configuration you are using letsencrypt-staging, for which you will not get browser trusted certificates and the websocket connection to the API server will fail. If the ingress was successfully created by the bootstrapper, you will see the respective error message in your browsers console.

Now we are trying to enable the shoot terminals, but run into the issue that the websocket our browser want to connect to (k-some-hast.ingress....) is different from the api-server-url of the shoot (api.shoot.namespace...)

yes k-some-hash.ingress.... is from the ingress that is being created by the terminal bootstrapper for the kube-apiserver.

[JensAc]

Thanks for the prompt response :+1: .

Actually, I think the cause of the problem is somewhat different from what you describe, as I have imported the le-staging certificate in my browser. However, the dashboard logs tell me for the seed (and it is the same for the shoots):

info : Bootstrapping Seed hcloud-hel1-0 aborted as 'spec.secretRef' on the seed is missing
debug: Successfully bootstrapped Seed - hcloud-hel1-0 (6dfc0052-ba80-4ec5-93bc-087c0137014c)

And there are no ingresses created. However, I cannot add the spec.secretRef to the seed as the seed seems to be immutable. Moreover, we do not have any secret containing the kubeconfig for the seed, as the gardenlet is running in the seed cluster. So, do I have to create such a secret by myself? Moreover, how would I get the secretRef in the seedSpec, when the seed resource is deployed by the gardenlet?

[petersutter]

I see. Yes true, for the gardenlet spec.secretRef does not necessarily need to be set, however, as of now, it's required for the terminal feature.

You would have to to set spec.gardenlet.config.seedConfig.secretRef on the ManagedSeed resource.

apiVersion: seedmanagement.gardener.cloud/v1alpha1
kind: ManagedSeed
metadata:
  name: foo
  namespace: garden
spec:
  gardenlet:
    config:
      apiVersion: gardenlet.config.gardener.cloud/v1alpha1
      kind: GardenletConfiguration
      seedConfig:
        spec:
          secretRef:
            name: seed-foo
            namespace: garden

ref: SeedConfig, SecretRef And you would have to create this secret containing a kubeconfig data property.

[petersutter]

@JensAc having the spec.gardenlet.config.seedConfig.secretRef set for ManagedSeeds is not really required, as @rfranzke pointed out in an out of thread communication. With the introduction of the shoots/adminkubeconfig endpoint we can get the kubeconfig for ManagedSeeds using this endpoint, instead of having to set spec.gardenlet.config.seedConfig.secretRef.

I will work on a fix.

[petersutter]

/assign

[JensAc]

Sounds great. Just one other comment. The terminal feature should also work for the internal seed, right? So, this case should be handled differently, shouldn't it?

And just for the record of this issue: Setting

spec:
  secretRef:
    name: hcloud-hel1-0-kubeconfig
    namespace: garden

in the gardenlet config and creating the corresponding secret made the ingress appear (also for the internal seed).

[petersutter]

Just one other comment. The terminal feature should also work for the internal seed, right? So, this case should be handled differently, shouldn't it?

what do you mean with internal seed? The terminal does not work for seeds that cannot be reached by the dashboard and the terminal-controller-manager. This is a known limitation and you can label these seeds with seed.dashboard.gardener.cloud/network: private which will hide the terminal button in the dashboard UI and the terminal bootstrapping logic will also not be run for such seeds.

[JensAc]

I mean the seed identical to the garden cluster, when a gardenlet is deployed in the garden cluster. However, this might be a special case. Actually, I meant seeds not deployed as managedseeds. At least the first seed would not be a managed seed, or am I mistaken?

[petersutter]

Yes right, for all non-managed seeds the secretRef is still required on the Seed resource