TcpSocket probes are meaningless with meshed pods

[chicocvenancio]

What is the issue?

Meshed pods have all ports redirected to linkerd-proxy sidecar and as such all ports are open at the TCP level. This means probes with TCPSocketAction are meaningless, as they will succeed as long as linkerd-proxy is alive, independent if the main container is dead.

How can it be reproduced?

While this deployment will fail liveness probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox-fails
  labels:
    app: busybox-fails
spec:
  replicas: 3
  selector:
    matchLabels:
      app: busybox-fails
  template:
    metadata:
      labels:
        app: busybox-fails
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        command:
        - sleep
        args:
        - "3600"
        livenessProbe:
          tcpSocket:
            port: 8080

Injecting linkerd is enough to make the probes pass:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
  labels:
    app: busybox
spec:
  replicas: 3
  selector:
    matchLabels:
      app: busybox
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      labels:
        app: busybox
    spec:
      containers:
      - name: busybox
        image: busybox:latest
        command:
        - sleep
        args:
        - "3600"
        livenessProbe:
          tcpSocket:
            port: 8080

Logs, error output, etc

Seems non-applicable.

output of linkerd check -o short

Linkerd core checks
===================


Status check results are √

Linkerd extensions checks
=========================


Status check results are √


Status check results are √

Environment

• Kubernetes v1.21.6-gke.1500 on GKE

Possible solution

As discussed in Slack, the ideal solution here is to passthrough kubelet traffic to the pod without redirecting to linkerd-proxy container, but at a minimum we should have this very clearly documented so developers are not surprised.

Additional context

No response

Would you like to work on fixing this bug?

no

[adleong]

I would recommend using the --skip-inbound-ports feature to bypass the proxy for these TcpSocket probes. Skipped inbound ports can be configured through the CLI, as a Helm value, or via annotations.

[bdols]

I also encountered this on v1.21.5+k3s2

[jseiser]

We hit this as well in our active/passive setup for ActiveMQ Classic. Enabling linkerd causes everything to show ready, even when its not.