Errors calling webhook "linkerd-policy-validator.linkerd.io"

[novjev]

What is the issue?

Randomly getting the following error:

Internal error occurred: failed calling webhook "linkerd-policy-validator.linkerd.io": failed to call webhook: Post "https://linkerd-policy-validator.linkerd.svc:443/?timeout=10s": EOF

The admission request events are triggered by Flux dry-run calls for gateway.networking.k8s.io/HTTPRoute resources.

How can it be reproduced?

kubectl apply -f httproute.yaml --dry-run=server

Logs, error output, etc

Error from server (InternalError): error when creating "httproute.yaml": Internal error occurred: failed calling webhook "linkerd-policy-validator.linkerd.io": failed to call webhook: Post "https://linkerd-policy-validator.linkerd.svc:443/?timeout=10s": EOF

output of linkerd check -o short

linkerd check -o short
linkerd-version
---------------
‼ cli is up-to-date
    is running version 25.4.4 but the latest edge version is 25.5.5
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 25.4.4 but the latest edge version is 25.5.5
    see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
        * linkerd-destination-785d5bbc4d-46b7w (edge-25.4.4)
        * linkerd-destination-785d5bbc4d-fz56z (edge-25.4.4)
        * linkerd-destination-785d5bbc4d-gnk5k (edge-25.4.4)
        * linkerd-identity-6c8b988f89-b4558 (edge-25.4.4)
        * linkerd-identity-6c8b988f89-q57kv (edge-25.4.4)
        * linkerd-identity-6c8b988f89-xnzkl (edge-25.4.4)
        * linkerd-proxy-injector-db867d489-9x8zt (edge-25.4.4)
        * linkerd-proxy-injector-db867d489-g8knc (edge-25.4.4)
        * linkerd-proxy-injector-db867d489-qd8wg (edge-25.4.4)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
        * metrics-api-d5996ccff-c56d5 (edge-25.4.4)
        * tap-5449496548-mtv64 (edge-25.4.4)
        * tap-injector-675448bf96-7dfgh (edge-25.4.4)
        * web-6ff5c79dc-n9rpz (edge-25.4.4)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √

Environment

• Kubernetes version: 1.32.3
• Cluster environment: AKS
• Linkerd version: edge-25.4.4

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[harti2006]

Running into the same issue

[adleong]

This looks like there are sporadic timeouts when calling the validating webhook in the policy-controller. Are you able to correlate resource usage with these timeouts? e.g. did they happen when the node was under heavy utilization or resource constrained? Have you looked at the policy-controller logs when these timeouts occur?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[harti2006]

This issue is still very relevant (in the latest recommended release edge-2025.8.5)

[alpeb]

Any further information you can provide to help diagnose this, like the policy-controller logs?

[harti2006]

Hi @alpeb , I can see the following logs in the "policy" container, when such an error happens:

linkerd-destination-7d85f54f74-h7zgp policy 2025-11-24T08:14:10.499510Z INFO server{port=9443}:conn{client.ip=10.70.3.251 client.port=39062}: kubert::server: Connection lost error=read header from client timeout linkerd-destination-7d85f54f74-7twd4 policy 2025-11-24T08:14:45.139216Z INFO server{port=9443}:conn{client.ip=10.70.2.209 client.port=41286}: kubert::server: Connection lost error=read header from client timeout