k9s icon indicating copy to clipboard operation
k9s copied to clipboard

port forward cannot stay up

Open masterkain opened this issue 7 months ago • 3 comments

since some versions (I'm at 0.50.6 but I believe it started around 0.50) port forwards simply cannot stay up, they are getting deleted after few seconds most of the times.

I encountered this on both my machines.

I'm port porwarding a cnpg postgres service

Name:                     eeevp-db-cluster-rw
Namespace:                eeevp
Labels:                   cnpg.io/cluster=eeevp-db-cluster
Annotations:              cnpg.io/operatorVersion: 1.25.1
Selector:                 cnpg.io/cluster=eeevp-db-cluster,cnpg.io/instanceRole=primary
Type:                     ClusterIP
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.43.48.254
IPs:                      10.43.48.254
Port:                     postgres  5432/TCP
TargetPort:               5432/TCP
Endpoints:                10.42.0.104:5432
Session Affinity:         None
Internal Traffic Policy:  Cluster
Events:                   <none>

forwarding goes away as soon as I use the port (mapped to 5433 locally), say for a pg_dump operation.

😗 PortForward failed for eeevp/eeevp-db-cluster-1|postgres|5433:5432: lost connection to pod. Deleting!

masterkain avatar May 15 '25 01:05 masterkain

@masterkain I can seem to repro this ;( I've left pf running all night, hitting the url every 10s, pf remained up. Does this happen only on the pg pod or others too?

derailed avatar May 18 '25 14:05 derailed

I'm experiencing this same issue, specifically when connecting to my CNPG Postgres pod - it doesn't occur with other pods.

I was on k9s version 0.40.10 initially, upgraded to 0.50.6 and the issue is still occurring.

I can create the port forward, perform 1 interaction with the database (e.g. refreshing database tables via datagrip), that succeeds but then the tunnel terminates immediately after with 😗 PortForward failed for immich/db-1|postgres|5432:5432: lost connection to pod. Deleting!

ZakShearman avatar May 23 '25 20:05 ZakShearman

@ZakShearman Thank you for the details!! Still can't seem to repro using helm pg chart v16.6.3 ;( Port forward stay up just fine when running psql

So wondering....

  1. What pg version are you running and how was it installed?
  2. Perhaps unsetting KUBECTL_PORT_FORWARD_WEBSOCKETS ?
  3. Would be great to know under which k9s version this stopped working

derailed avatar Jun 08 '25 13:06 derailed

This issue is stale because it has been open for 30 days with no activity.

github-actions[bot] avatar Jul 09 '25 02:07 github-actions[bot]

I dug into this a bit deeper and wanted to report back with a reproducible scenario and what looks like the root cause. Short version: the tunnel drops because kubectl’s port-forward doesn’t like the TLS connection profile that CloudNativePG exposes; K9s just inherits the failure because it shells out to kubectl.

Environment

  • Kubernetes server: v1.33.2 (k3s distribution)
  • kubectl client: v1.34.1
  • CloudNativePG operator: 1.27.0
  • PostgreSQL image: ghcr.io/cloudnative-pg/postgresql:17.4
  • K9s: 0.50.6 (but the same behaviour occurs when running kubectl directly)

Repro steps

  1. kubectl port-forward -n <namespace> svc/<cnpg-primary-service> 5433:5432
  2. Run any psql/pg_dump with the default sslmode=prefer.
  3. The query completes once, then kubectl immediately exits and prints
    failed to execute portforward … read tcp4 127.0.0.1:<local>->127.0.0.1:5432: read: connection reset by peer
    

Extra checks:

  • Switching to kubectl port-forward pod/<primary-pod> hits the same error.
  • Forcing SPDY by exporting KUBECTL_PORT_FORWARD_WEBSOCKETS=false does not change the behaviour.
  • Verbose kubectl -v9 logs confirm the API server upgrades the connection, the client query runs, and the stream ends with the reset shown above.

What stops the flapping If I force plaintext on the client side—PGSSLMODE=disable or sslmode=disable in the connection string—the port-forward stays up indefinitely, even through large pg_dump transfers. The in-cluster traffic still terminates at the CloudNativePG pod, so the workaround only affects the hop between the local tool and kubectl.

Given that behaviour, I don’t think K9s itself needs a code change here. The failure originates in the same kubectl command that K9s runs under the hood. Anyone hitting this can work around it by disabling SSL just for the forwarded connection until kubectl (or CloudNativePG) addresses the TLS interaction.

masterkain avatar Oct 05 '25 00:10 masterkain