linkerd2

linkerd2 copied to clipboard

What problem are you trying to solve?

In private GKE clusters the mutating webhook exposed by the proxy injector cannot be called out of the box.

This is because on private GKE clusters, Kubernetes is configured with --enable-aggregator-routing=true (https://github.com/kubernetes/kubernetes/issues/79739#issuecomment-509813068).

This leads to timeouts when the Kube API server tries to call the webhook:

I0619 18:43:21.906411      11 trace.go:205] Trace[1496576369]: "Call mutating webhook" configuration:linkerd-proxy-injector-webhook-config,webhook:linkerd-proxy-injector.linkerd.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:3e9dc071-bcd2-49b9-bd68-d4038c367788 (19-Jun-2022 18:43:11.886) (total time: 10020ms):
Trace[1496576369]: [10.020060667s] [10.020060667s] END

W0619 18:43:21.906521      11 dispatcher.go:180] Failed calling webhook, failing open linkerd-proxy-injector.linkerd.io: failed calling webhook "linkerd-proxy-injector.linkerd.io": failed to call webhook: Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=10s": context deadline exceeded

E0619 18:43:21.906562      11 dispatcher.go:184] failed calling webhook "linkerd-proxy-injector.linkerd.io": failed to call webhook: Post "https://linkerd-proxy-injector.linkerd.svc:443/?timeout=10s": context deadline exceeded

A workaround would be to create a firewall rule for port 8443, where the webhook is currently exposed by linkerd-proxy-injector. Other applications also suffer from this: https://github.com/elastic/cloud-on-k8s/issues/1437

By default, a private GKE cluster already creates firewall rules for port 443 though, so changing the webhook port from 8443 to 443 eliminates the need for creating such a workaround firewall rule and makes the mutating webbhook work out of the box, as intended.

How should the problem be solved?

Change the port from 8443 to 443.

Any alternatives you've considered?

Create a firewall rule to explicitly allow incoming node traffic from Kube API server on port 8443.

How would users interact with this feature?

No response

Would you like to work on this feature?

No response

This would probably have to apply to other webhooks as well. Though, in some cases we have multiple webhook containers in a single pod, so it wouldn't be possible for both to bind on 443.

For what it's worth, we already use 443 as the Service port:

:; k get svc -n linkerd linkerd-proxy-injector 
NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
linkerd-proxy-injector   ClusterIP   10.43.105.208   <none>        443/TCP   78m

But I guess that isn't enough to make GKE happy.

Unfortunately, I'm not sure that it's possible to make Linkerd run on GKE Private clusters out-of-the-box without larger changes to Linkerd's deployment structure.

And then if we made those changes, I believe we'd have to run these containers as root so that they could bind to a privileged port? Or we'd need elevated privileges of some sort?

Furthermore, we don't have any ability to run integration tests against this configuration to ensure that we stay compatible going forward.

Hey @olix0r,

correct, the service port is not enough when the cluster is configured with --enable-aggregator-routing=true, unfortunately.

What other Webhooks are there that would need to be handled similarly? I got the linkerd install working by only allowing port 8443 on the firewall rules. I still think changing this for the injector would be a good thing to do, even if there would be more webhooks that need a similar migration.

Alternatively, maybe it makes sense to clearly state this workaround in the Getting Started guide and other parts of the documentation where this is relevant.

What do you think?

Hi Guys.

I had the same problem and the solution is already documented. just add the ports 8443, 8089 and 9443 in the firewall rule of the GKE master as in the explained documentation https://linkerd.io/2.11/tasks/securing-your-cluster/#gke

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.