linkerd2 icon indicating copy to clipboard operation
linkerd2 copied to clipboard
verified publisher icon linkerd

Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

Open wamak9 opened this issue 1 year ago • 3 comments

What is the issue?

I keep seeing os error 111 issue on linkerd and have no idea how to fix it.

linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

More logs below. IPs seen in the logs are releated to self-hosted Prometheus. There are two sets on Prometheus running, one is from Linkerd and one is self hosted by us. Tried to add bunch of annotations

  annotations:
    config.linkerd.io/default-inbound-policy: all-unauthenticated
    config.linkerd.io/image-pull-policy: Always
    config.linkerd.io/proxy-outbound-connect-timeout: "5"

How can it be reproduced?

Current version running is stable-2.14.9. Enable linkerd on one of the namespace and then deploy. Deploy prometheus on different namepace with no linkeerd annotation.

Logs, error output, etc

Linkerd enabled POD logs.

2024-06-06T14:02:51.497752933Z [ 61784.863882s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.192.187:45472}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:02:53.065641026Z [ 61786.431859s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.195.24:42250 server.addr=10.109.194.85:4143
2024-06-06T14:02:54.178812465Z [ 61787.544994s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.195.24:39080}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:02:57.564499057Z [ 61790.930762s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:59806 server.addr=10.109.194.85:4143
2024-06-06T14:02:58.072767476Z [ 61791.439007s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:49474 server.addr=10.109.194.85:4143
2024-06-06T14:03:03.539011114Z [ 61796.905198s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.194.225:34054}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:06.497366974Z [ 61799.863570s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.192.187:37222}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:08.065163606Z [ 61801.431478s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.195.24:34458 server.addr=10.109.194.85:4143
2024-06-06T14:03:09.178458960Z [ 61802.544615s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.195.24:52422}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:12.567208334Z [ 61805.933426s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:58710 server.addr=10.109.194.85:4143
2024-06-06T14:03:13.073033574Z [ 61806.439218s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:56226 server.addr=10.109.194.85:4143
2024-06-06T14:03:18.539286848Z [ 61811.905464s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.194.225:40056}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:21.498128401Z [ 61814.864275s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.192.187:50618}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:23.065716727Z [ 61816.431909s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.195.24:50206 server.addr=10.109.194.85:4143
2024-06-06T14:03:24.178703304Z [ 61817.544816s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.195.24:45960}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:27.565232830Z [ 61820.931438s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:50358 server.addr=10.109.194.85:4143
2024-06-06T14:03:28.073418766Z [ 61821.439587s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:38512 server.addr=10.109.194.85:4143
2024-06-06T14:03:33.539307360Z [ 61826.905511s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.194.225:37682}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:36.498069332Z [ 61829.864335s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.192.187:55184}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:38.065723027Z [ 61831.431949s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.195.24:59672 server.addr=10.109.194.85:4143
2024-06-06T14:03:39.178306799Z [ 61832.544496s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.195.24:52204}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T14:03:42.566024937Z [ 61835.932270s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:34180 server.addr=10.109.194.85:4143
2024-06-06T14:03:43.072400780Z [ 61836.438591s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:43040 server.addr=10.109.194.85:4143

linkerd-destination logs

2024-06-06T11:01:45.527253157Z time="2024-06-06T11:01:45Z" level=info msg="running version stable-2.14.9"
2024-06-06T11:01:45.527306758Z time="2024-06-06T11:01:45Z" level=info msg="starting admin server on :9996"
2024-06-06T11:01:45.527683660Z time="2024-06-06T11:01:45Z" level=info msg="Using default opaque ports: map[25:{} 587:{} 3306:{} 4444:{} 5432:{} 6379:{} 9300:{} 11211:{}]"
2024-06-06T11:01:45.577414572Z time="2024-06-06T11:01:45Z" level=warning msg="failed to register Prometheus gauge Desc{fqName: \"job_cache_size\", help: \"Number of items in the client-go job cache\", constLabels: {cluster=\"local\"}, variableLabels: []}: duplicate metrics collector registration attempted"
2024-06-06T11:01:45.577762874Z time="2024-06-06T11:01:45Z" level=info msg="waiting for caches to sync"
2024-06-06T11:01:46.278942767Z time="2024-06-06T11:01:46Z" level=info msg="caches synced"
2024-06-06T11:01:46.278963767Z time="2024-06-06T11:01:46Z" level=info msg="waiting for caches to sync"
2024-06-06T11:01:46.479453623Z time="2024-06-06T11:01:46Z" level=info msg="caches synced"
2024-06-06T11:01:46.479476623Z time="2024-06-06T11:01:46Z" level=info msg="waiting for caches to sync"
2024-06-06T11:01:46.580583557Z time="2024-06-06T11:01:46Z" level=info msg="caches synced"
2024-06-06T11:01:46.580609057Z time="2024-06-06T11:01:46Z" level=info msg="starting gRPC server on :8086"
2024-06-06T11:02:14.227039126Z time="2024-06-06T11:02:14Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T11:02:14.227061327Z time="2024-06-06T11:02:14Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T11:27:59.866809601Z time="2024-06-06T11:27:59Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T11:27:59.871358521Z time="2024-06-06T11:27:59Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:19:56.992769784Z time="2024-06-06T12:19:56Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:20:53.725740222Z time="2024-06-06T12:20:53Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:23:39.890411717Z time="2024-06-06T12:23:39Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:23:56.058165008Z time="2024-06-06T12:23:56Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:24:15.286391663Z time="2024-06-06T12:24:15Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:37:10.770923303Z time="2024-06-06T12:37:10Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:39:40.200961684Z time="2024-06-06T12:39:40Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:43:37.669790348Z time="2024-06-06T12:43:37Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:45:09.760380028Z time="2024-06-06T12:45:09Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T12:48:29.842386056Z time="2024-06-06T12:48:29Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T13:02:24.699553018Z time="2024-06-06T13:02:24Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"
2024-06-06T13:02:43.467367168Z time="2024-06-06T13:02:43Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462"

linkerd Proxy logs

Defaulted container "linkerd-proxy" out of: linkerd-proxy, destination, sp-validator, policy, linkerd-init (init)
2024-06-06T11:01:39.532896292Z [     0.001881s]  INFO ThreadId(01) linkerd2_proxy: release 2.210.4 (5a910be) by linkerd on 2023-11-22T17:01:46Z
2024-06-06T11:01:39.533797393Z [     0.002752s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
2024-06-06T11:01:39.534630694Z [     0.003570s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
2024-06-06T11:01:39.534645594Z [     0.003586s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
2024-06-06T11:01:39.534649094Z [     0.003589s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
2024-06-06T11:01:39.534652694Z [     0.003591s]  INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
2024-06-06T11:01:39.534656194Z [     0.003594s]  INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
2024-06-06T11:01:39.534664494Z [     0.003596s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
2024-06-06T11:01:39.534667994Z [     0.003599s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via localhost:8086
2024-06-06T11:01:39.535228695Z [     0.004176s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:39.554482718Z [     0.023458s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity id=linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local
2024-06-06T11:01:39.644521926Z [     0.113388s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:39.848110472Z [     0.316943s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:40.270987949Z [     0.739771s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:40.771837348Z [     1.240706s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:41.272759547Z [     1.741663s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:41.773640646Z [     2.242598s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:42.274516244Z [     2.743467s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:42.775443144Z [     3.244340s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:43.276569556Z [     3.745309s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:43.777472594Z [     4.246312s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:44.278418133Z [     4.747232s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:44.780177977Z [     5.249157s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:45.281776819Z [     5.750557s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:45.782566657Z [     6.251406s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:46.283488295Z [     6.752283s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:46.784334333Z [     7.253158s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:47.286039677Z [     7.755010s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:47.787507619Z [     8.256423s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]
2024-06-06T11:01:48.288459857Z [     8.757288s]  WARN ThreadId(01) watch{port=4191}:controller{addr=localhost:8090}:endpoint{addr=127.0.0.1:8090}: linkerd_reconnect: Failed to connect error=endpoint 127.0.0.1:8090: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

Linkerd Policy Logs

 linkerd-destination-7fbd959544-f6kpk -n linkerd -c policy
2024-06-06T11:01:48.294603796Z 2024-06-06T11:01:48.294380Z  INFO linkerd_policy_controller: Lease already exists, no need to create it
2024-06-06T11:01:48.299913029Z 2024-06-06T11:01:48.299761Z  INFO grpc{port=8090}: linkerd_policy_controller: policy gRPC server listening addr=0.0.0.0:8090
2024-06-06T11:01:48.605395043Z 2024-06-06T11:01:48.605148Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-t27zz}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T11:01:48.605421943Z 2024-06-06T11:01:48.605177Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-t27zz}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T11:06:17.398630916Z 2024-06-06T11:06:17.398329Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:06:20.742492818Z 2024-06-06T11:06:20.742265Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:06:43.331433686Z 2024-06-06T11:06:43.331082Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:11:18.346663559Z 2024-06-06T11:11:18.346410Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:11:20.666748723Z 2024-06-06T11:11:20.666472Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:11:33.341829499Z 2024-06-06T11:11:33.341609Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:11:33.343025401Z 2024-06-06T11:11:33.342818Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:11:33.343944203Z 2024-06-06T11:11:33.343794Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:15:36.278244184Z 2024-06-06T11:15:36.277927Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:16:05.295769996Z 2024-06-06T11:16:05.295625Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:16:05.787488410Z 2024-06-06T11:16:05.787225Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:20:43.271326046Z 2024-06-06T11:20:43.271013Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:21:05.692689311Z 2024-06-06T11:21:05.692500Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:21:18.369883721Z 2024-06-06T11:21:18.369736Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:25:26.291677397Z 2024-06-06T11:25:26.291517Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:25:33.209261617Z 2024-06-06T11:25:33.208992Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:26:08.364672652Z 2024-06-06T11:26:08.364489Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:30:10.948518034Z 2024-06-06T11:30:10.948315Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:30:45.787450297Z 2024-06-06T11:30:45.787192Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:30:45.826712617Z 2024-06-06T11:30:45.826434Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:30:55.703409507Z 2024-06-06T11:30:55.703292Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:35:16.301389076Z 2024-06-06T11:35:16.301022Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:35:23.260278653Z 2024-06-06T11:35:23.260096Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:35:36.191636641Z 2024-06-06T11:35:36.191425Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:40:00.958819115Z 2024-06-06T11:40:00.958697Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:40:01.440901906Z 2024-06-06T11:40:01.440781Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:40:15.505510479Z 2024-06-06T11:40:15.505247Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:40:45.752405390Z 2024-06-06T11:40:45.752166Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:44:45.664638129Z 2024-06-06T11:44:45.664413Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:49:00.564402785Z 2024-06-06T11:49:00.564130Z  INFO serverauthorizations: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777383951 (777388465): Expired
2024-06-06T11:49:02.891293267Z 2024-06-06T11:49:02.891063Z  INFO authorizationpolicies: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777379769 (777388515): Expired
2024-06-06T11:49:51.461938005Z 2024-06-06T11:49:51.461766Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:50:16.235593017Z 2024-06-06T11:50:16.235453Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:50:21.245912723Z 2024-06-06T11:50:21.245768Z  INFO meshtlsauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777378036 (777388477): Expired
2024-06-06T11:53:40.865031263Z 2024-06-06T11:53:40.864829Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:54:16.686759657Z 2024-06-06T11:54:16.686536Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:54:21.689614894Z 2024-06-06T11:54:21.689279Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777385220 (777388478): Expired
2024-06-06T11:54:39.985348235Z 2024-06-06T11:54:39.985177Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:55:21.257504554Z 2024-06-06T11:55:21.257292Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:58:58.367168758Z 2024-06-06T11:58:58.366911Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T11:59:25.683175952Z 2024-06-06T11:59:25.683011Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T11:59:41.469525977Z 2024-06-06T11:59:41.469270Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:03:25.872998107Z 2024-06-06T12:03:25.872794Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:03:53.372850817Z 2024-06-06T12:03:53.372526Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:04:04.698892263Z 2024-06-06T12:04:04.698643Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:04.698916763Z 2024-06-06T12:04:04.698673Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:04.711185297Z 2024-06-06T12:04:04.711037Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:04.711205498Z 2024-06-06T12:04:04.711060Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:04.733837362Z 2024-06-06T12:04:04.733721Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:04.733857062Z 2024-06-06T12:04:04.733747Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:06.597825659Z 2024-06-06T12:04:06.597623Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:06.597859759Z 2024-06-06T12:04:06.597655Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:15.635856882Z 2024-06-06T12:04:15.635620Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:15.635885882Z 2024-06-06T12:04:15.635649Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:16.684837496Z 2024-06-06T12:04:16.684667Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:16.684858496Z 2024-06-06T12:04:16.684699Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:19.668418281Z 2024-06-06T12:04:19.668255Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:19.668471381Z 2024-06-06T12:04:19.668299Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:19.685482371Z 2024-06-06T12:04:19.685225Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:19.685511971Z 2024-06-06T12:04:19.685257Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:04:29.997367091Z 2024-06-06T12:04:29.997199Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:05:27.879089570Z 2024-06-06T12:05:27.878896Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:05:27.879113470Z 2024-06-06T12:05:27.878926Z  WARN pods:apply{ns=rmplatform name=forecast-service-78f97f8d7f-6wnzn}: linkerd_policy_controller_k8s_index::inbound::pod: invalid probe path error=invalid format path="health-check?type=alive"
2024-06-06T12:09:15.692854259Z 2024-06-06T12:09:15.692626Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:09:40.104394626Z 2024-06-06T12:09:40.104196Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:13:15.883973936Z 2024-06-06T12:13:15.883673Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:13:21.012481288Z 2024-06-06T12:13:21.012248Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:13:58.928645562Z 2024-06-06T12:13:58.928419Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:15:06.467868444Z 2024-06-06T12:15:06.467679Z  INFO networkauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777416000 (777424520): Expired
2024-06-06T12:15:06.517568904Z 2024-06-06T12:15:06.517269Z  INFO serverauthorizations: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777415582 (777424521): Expired
2024-06-06T12:15:18.511644407Z 2024-06-06T12:15:18.511538Z  WARN httproutes.policy.linkerd.io: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:15:18.513770913Z 2024-06-06T12:15:18.513639Z  WARN meshtlsauthentications: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:15:18.514321714Z 2024-06-06T12:15:18.514242Z  WARN pods: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:15:35.690883076Z 2024-06-06T12:15:35.690772Z  WARN meshtlsauthentications: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:15:47.390236327Z 2024-06-06T12:15:47.389963Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777414655 (777426271): Expired
2024-06-06T12:15:47.413151897Z 2024-06-06T12:15:47.413022Z  INFO meshtlsauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777418107 (777426271): Expired
2024-06-06T12:15:47.440330380Z 2024-06-06T12:15:47.440235Z  INFO authorizationpolicies: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777417886 (777426272): Expired
2024-06-06T12:15:47.478405997Z 2024-06-06T12:15:47.478280Z  INFO servers: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777420115 (777426272): Expired
2024-06-06T12:16:04.320879731Z 2024-06-06T12:16:04.320763Z  WARN httproutes.policy.linkerd.io: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:16:04.321729334Z 2024-06-06T12:16:04.321553Z  WARN servers: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:16:04.332365268Z 2024-06-06T12:16:04.332231Z  WARN services: kube_client::client: eof in poll: error reading a body from connection: error reading a body from connection: unexpected EOF during chunk size line
2024-06-06T12:16:04.883815838Z 2024-06-06T12:16:04.883500Z  INFO servers: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777426414 (777426984): Expired
2024-06-06T12:16:04.885085042Z 2024-06-06T12:16:04.884925Z  INFO services: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777414423 (777426076): Expired
2024-06-06T12:16:04.957142073Z 2024-06-06T12:16:04.956950Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777416962 (777426988): Expired
2024-06-06T12:19:28.416621738Z 2024-06-06T12:19:28.416477Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:19:28.514727496Z 2024-06-06T12:19:28.514468Z  INFO serverauthorizations: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777424524 (777431439): Expired
2024-06-06T12:20:06.547000849Z 2024-06-06T12:20:06.546772Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:20:06.615500301Z 2024-06-06T12:20:06.615243Z  INFO networkauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777424735 (777432263): Expired
2024-06-06T12:20:16.142167373Z 2024-06-06T12:20:16.141916Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:20:28.674233123Z 2024-06-06T12:20:28.674063Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:20:47.511533898Z 2024-06-06T12:20:47.511323Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:20:47.535927386Z 2024-06-06T12:20:47.535658Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:20:47.583804758Z 2024-06-06T12:20:47.583603Z  INFO authorizationpolicies: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777426416 (777433307): Expired
2024-06-06T12:21:04.930239330Z 2024-06-06T12:21:04.929985Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:21:04.964350469Z 2024-06-06T12:21:04.964065Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:23:59.574846395Z 2024-06-06T12:23:59.574512Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:23:59.681645184Z 2024-06-06T12:23:59.681336Z  INFO serverauthorizations: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777431550 (777437510): Expired
2024-06-06T12:24:41.364637145Z 2024-06-06T12:24:41.364384Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:00.416086828Z 2024-06-06T12:25:00.415917Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:02.942062993Z 2024-06-06T12:25:02.941868Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:10.075898281Z 2024-06-06T12:25:10.075741Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:16.046403942Z 2024-06-06T12:25:16.046225Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:48.952645346Z 2024-06-06T12:25:48.952403Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:25:54.231405536Z 2024-06-06T12:25:54.231142Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777427147 (777440002): Expired
2024-06-06T12:28:59.691395980Z 2024-06-06T12:28:59.691159Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:29:39.120923640Z 2024-06-06T12:29:39.120713Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:29:45.631078852Z 2024-06-06T12:29:45.630835Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:30:54.239005766Z 2024-06-06T12:30:54.238752Z  INFO httproutes.policy.linkerd.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:33:23.047866513Z 2024-06-06T12:33:23.047500Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:33:28.182198861Z 2024-06-06T12:33:28.181936Z  INFO serverauthorizations: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777437609 (777449915): Expired
2024-06-06T12:34:16.806200732Z 2024-06-06T12:34:16.805882Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:34:26.373132231Z 2024-06-06T12:34:26.372957Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:34:31.467332616Z 2024-06-06T12:34:31.467067Z  INFO networkauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777439602 (777451281): Expired
2024-06-06T12:34:45.010556395Z 2024-06-06T12:34:45.010377Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:34:50.139471530Z 2024-06-06T12:34:50.139232Z  INFO servers: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777440186 (777451675): Expired
2024-06-06T12:35:01.095308083Z 2024-06-06T12:35:01.095083Z  INFO services: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:38:02.902431904Z 2024-06-06T12:38:02.902303Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:39:01.740286124Z 2024-06-06T12:39:01.740094Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:39:21.569384996Z 2024-06-06T12:39:21.569217Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:43:29.206446034Z 2024-06-06T12:43:29.206310Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:44:06.812732074Z 2024-06-06T12:44:06.812489Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:44:11.884168159Z 2024-06-06T12:44:11.884002Z  INFO meshtlsauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777455204 (777464089): Expired
2024-06-06T12:44:14.413028870Z 2024-06-06T12:44:14.412885Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:47:47.913824905Z 2024-06-06T12:47:47.913699Z  INFO serverauthorizations: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:48:29.211067377Z 2024-06-06T12:48:29.210475Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:48:34.267077056Z 2024-06-06T12:48:34.266838Z  INFO networkauthentications: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777451363 (777469802): Expired
2024-06-06T12:48:39.794898846Z 2024-06-06T12:48:39.794610Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:48:44.946292607Z 2024-06-06T12:48:44.945962Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777455404 (777470030): Expired
2024-06-06T12:49:11.894486140Z 2024-06-06T12:49:11.894363Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:53:05.166395204Z 2024-06-06T12:53:05.166196Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:53:17.571772206Z 2024-06-06T12:53:17.571464Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:53:56.588415289Z 2024-06-06T12:53:56.588215Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:53:59.138464841Z 2024-06-06T12:53:59.138109Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T12:57:42.892839989Z 2024-06-06T12:57:42.892607Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:58:39.206768245Z 2024-06-06T12:58:39.206461Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T12:58:44.272308275Z 2024-06-06T12:58:44.272051Z  INFO authorizationpolicies: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777469652 (777483014): Expired
2024-06-06T13:02:42.899102788Z 2024-06-06T13:02:42.898971Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:02:50.176771944Z 2024-06-06T13:02:50.176545Z  INFO networkauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:03:14.873931587Z 2024-06-06T13:03:14.873604Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T13:03:20.620364765Z 2024-06-06T13:03:20.620176Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T13:03:46.598863877Z 2024-06-06T13:03:46.598573Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:07:49.843722494Z 2024-06-06T13:07:49.843577Z  INFO authorizationpolicies: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T13:13:04.907069780Z 2024-06-06T13:13:04.906740Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:13:36.608248209Z 2024-06-06T13:13:36.608080Z  INFO servers: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:17:22.913973674Z 2024-06-06T13:17:22.913767Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:27:17.326437339Z 2024-06-06T13:27:17.326182Z  INFO meshtlsauthentications: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T13:36:52.929380825Z 2024-06-06T13:36:52.929063Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out
2024-06-06T13:36:58.076826456Z 2024-06-06T13:36:58.076580Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=error returned by apiserver during watch: too old resource version: 777522074 (777533810): Expired
2024-06-06T13:41:33.744611849Z 2024-06-06T13:41:33.744390Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: Connection reset by peer (os error 104)
2024-06-06T13:56:08.754288809Z 2024-06-06T13:56:08.754153Z  INFO httproutes.gateway.networking.k8s.io: kubert::errors: stream failed error=watch stream failed: Error reading events stream: error reading a body from connection: error reading a body from connection: timed out

output of linkerd check -o short

linkerd-version
---------------
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    unsupported version channel: stable-2.14.9
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-7fbd959544-f6kpk (stable-2.14.9)
	* linkerd-identity-7cf66888f7-bb68b (stable-2.14.9)
	* linkerd-proxy-injector-64c5976b47-h7hg7 (stable-2.14.9)
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-7fbd959544-f6kpk running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-55c8db4654-zsvt2 (stable-2.14.9)
	* prometheus-68ddcbf849-x4kxs (stable-2.14.9)
	* tap-5c6788c47d-pz5qm (stable-2.14.9)
	* tap-injector-674d8d486f-mq4zx (stable-2.14.9)
	* web-5f8cd8d88f-xbnq6 (stable-2.14.9)
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-55c8db4654-zsvt2 running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints

Status check results are √

linkerd check
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API

kubernetes-version
------------------
√ is running the minimum Kubernetes API version

linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ control plane pods are ready
√ cluster networks contains all pods
√ cluster networks contains all services

linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ proxy-init container runs as root user if docker container runtime is used

linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor

linkerd-webhooks-and-apisvc-tls
-------------------------------
√ proxy-injector webhook has valid cert
√ proxy-injector cert is valid for at least 60 days
√ sp-validator webhook has valid cert
√ sp-validator cert is valid for at least 60 days
√ policy-validator webhook has valid cert
√ policy-validator cert is valid for at least 60 days

linkerd-version
---------------
√ can determine the latest version
‼ cli is up-to-date
    unsupported version channel: stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-cli for hints

control-plane-version
---------------------
√ can retrieve the control plane version
‼ control plane is up-to-date
    unsupported version channel: stable-2.14.9
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints
‼ control plane and cli versions match
    control plane running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
√ control plane proxies are healthy
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-7fbd959544-f6kpk (stable-2.14.9)
	* linkerd-identity-7cf66888f7-bb68b (stable-2.14.9)
	* linkerd-proxy-injector-64c5976b47-h7hg7 (stable-2.14.9)
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
    linkerd-destination-7fbd959544-f6kpk running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-cp-proxy-cli-version for hints

linkerd-viz
-----------
√ linkerd-viz Namespace exists
√ can initialize the client
√ linkerd-viz ClusterRoles exist
√ linkerd-viz ClusterRoleBindings exist
√ tap API server has valid cert
√ tap API server cert is valid for at least 60 days
√ tap API service is running
√ linkerd-viz pods are injected
√ viz extension pods are running
√ viz extension proxies are healthy
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-55c8db4654-zsvt2 (stable-2.14.9)
	* prometheus-68ddcbf849-x4kxs (stable-2.14.9)
	* tap-5c6788c47d-pz5qm (stable-2.14.9)
	* tap-injector-674d8d486f-mq4zx (stable-2.14.9)
	* web-5f8cd8d88f-xbnq6 (stable-2.14.9)
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-55c8db4654-zsvt2 running stable-2.14.9 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints
√ prometheus is installed and configured correctly
√ viz extension self-check

Status check results are √

Environment

Kubernetes version: v1.27.7
Cluster environment: AKS
Host OS: Linux
Linkerd version: stable-2.14.9

Possible solution

N/A

Additional context

So, I tried to add linkerd to Monitoring namespace which is running Prometheus. But that did not help and I am seeing way too many error logs in Prometheus on the same OS error.

I did add an authorization policy, so our self installed Prometheus can scrape metrics.

apiVersion: policy.linkerd.io/v1alpha1
kind: AuthorizationPolicy
metadata:
  name: prometheus-admin-federate
  namespace: linkerd-viz
spec:
  targetRef:
    group: policy.linkerd.io
    kind: Server
    name: prometheus-admin
  requiredAuthenticationRefs:
    - group: policy.linkerd.io
      kind: NetworkAuthentication
      name: kubelet

Would you like to work on fixing this bug?

maybe

wamak9 avatar Jun 06 '24 14:06 wamak9

Hey @wamak9,

Going to walk through the errors one-by-one to explain what they mean.

1. linkerd enabled pod logs:

2024-06-06T14:03:42.566024937Z [ 61835.932270s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.192.187:34180 server.addr=10.109.194.85:4143
2024-06-06T14:03:43.072400780Z [ 61836.438591s]  INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated error.sources=[direct connections must be mutually authenticated] client.addr=10.109.194.225:43040 server.addr=10.109.194.85:4143

This indicates that the client of this service (10.109.192.187:34180) is not mTLS'd but is trying to connect directly to the proxy's inbound port 4143. Only another mTLS participant can initiate connections to 4143. Typically, we might see this if the server is marked as opaque or is a multicluster gateway. If it opaque, then the client is either not injected with a proxy (in which case it should send directly to the proxy's port anyway) or misconfigured. It's hard to tell without more information.

2. linkerd destination logs:

2024-06-06T13:02:43.467367168Z time="2024-06-06T13:02:43Z" level=error msg="failed to find LINKERD2_PROXY_INBOUND_LISTEN_ADDR environment variable in any container for given pod spec" addr=":8086" component=endpoint-profile-translator context-ns=retail context-pod=tapi-api-7c47bcf658-bcpqb remote="10.109.194.212:47462

An endpoint is marked as injected (probably has a control plane label) but it is not. Something went wrong where either the pod received the proxy with incomplete configuration, or the pod has been improperly annotated. This is not a typical occurence. Are you using any other features such as native sidecars?

3. linkerd proxy logs: these logs are benign. They're from the destination service's proxy and it simply signals it cannot establish a connection since the socket has been bound yet. The proxy starts first before the destination container warms up, these only happen within the first 10-20s of the proxy's lifetime.

4. policy controller logs: seems like the API server's conn is a bit wonky? I'm not sure this is necessarily related to what you're seeing. Is that a common occurence or is it just in a limited time interval when this happens.

I think it's helpful to isolate the logs that are relevant here, and that's probably the first set of logs. errno 111 simply means a connection cannot be established. Typically, it's because the socket's not listening. Sometimes you might see these out in the wild but if it's not a recurrence, or if it doesn't directly impact your traffic and services, it's safe to ignore.

From your original description:

IPs seen in the logs are releated to self-hosted Prometheus.

Where exactly? The one not being mutually authenticated, or...

For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod.

annotations: config.linkerd.io/default-inbound-policy: all-unauthenticated config.linkerd.io/image-pull-policy: Always config.linkerd.io/proxy-outbound-connect-timeout: "5"

On your self-hosted prometheus? Is it injected or not injected? If it's not, then the annotation won't have anything to configure. Can you do a kubectl get pods so we can see how your set-up looks like?

mateiidavid avatar Jun 11 '24 11:06 mateiidavid

IPs seen in the logs are releated to self-hosted Prometheus.

Where exactly? The one not being mutually authenticated, or...

For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod.

annotations: config.linkerd.io/default-inbound-policy: all-unauthenticated config.linkerd.io/image-pull-policy: Always config.linkerd.io/proxy-outbound-connect-timeout: "5"

On your self-hosted prometheus? Is it injected or not injected? If it's not, then the annotation won't have anything to configure. Can you do a kubectl get pods so we can see how your set-up looks like?

My self-hosted Prometheus is running on the same cluster and Linkerd is not injected on the Prometheus. So there are in total 300 pods which has linkerd injected and we see these errors on almost every single one of them.

k get pods -n linkerd
NAME                                      READY   STATUS    RESTARTS   AGE
linkerd-destination-7fbd959544-f6kpk      4/4     Running   0          7d4h
linkerd-identity-7cf66888f7-bb68b         2/2     Running   0          7d12h
linkerd-proxy-injector-64c5976b47-h7hg7   2/2     Running   0          7d20h


k get pods -n linkerd-viz
NAME                            READY   STATUS    RESTARTS       AGE
metrics-api-55c8db4654-zsvt2    2/2     Running   2 (7d4h ago)   7d4h
prometheus-68ddcbf849-x4kxs     2/2     Running   0              7d4h
tap-5c6788c47d-pz5qm            2/2     Running   2 (7d4h ago)   7d4h
tap-injector-674d8d486f-mq4zx   2/2     Running   0              7d4h
web-5f8cd8d88f-xbnq6            2/2     Running   0              7d4h

For the connection refused in your linkerd enabled pod, is the server listening. Can you confirm that? Would be useful what is supposed to happen in that pod.

I am not sure what this means, when you say server listening are we talking about proxy and init ?

 klogs e-yw0qc1c3x6kl5n1ifx-78854d4c58gcnxx -n default -c linkerd-init
2024-06-12T06:35:56.821771350Z time="2024-06-12T06:35:56Z" level=info msg="/sbin/iptables-save -t nat"
2024-06-12T06:35:56.918594899Z time="2024-06-12T06:35:56Z" level=info msg="# Generated by iptables-save v1.8.8 on Wed Jun 12 06:35:56 2024\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\nCOMMIT\n# Completed on Wed Jun 12 06:35:56 2024\n"
2024-06-12T06:35:56.918749699Z time="2024-06-12T06:35:56Z" level=info msg="/sbin/iptables -t nat -N PROXY_INIT_REDIRECT"
2024-06-12T06:35:56.920661900Z time="2024-06-12T06:35:56Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_REDIRECT -p tcp --match multiport --dports 4190,4191,8090 -j RETURN -m comment --comment proxy-init/ignore-port-4190,4191,8090/1718174156"
2024-06-12T06:35:56.922512901Z time="2024-06-12T06:35:56Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_REDIRECT -p tcp -j REDIRECT --to-port 4143 -m comment --comment proxy-init/redirect-all-incoming-to-proxy-port/1718174156"
2024-06-12T06:35:57.017541950Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A PREROUTING -j PROXY_INIT_REDIRECT -m comment --comment proxy-init/install-proxy-init-prerouting/1718174156"
2024-06-12T06:35:57.019529751Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -N PROXY_INIT_OUTPUT"
2024-06-12T06:35:57.021036252Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -j RETURN -m comment --comment proxy-init/ignore-proxy-user-id/1718174156"
2024-06-12T06:35:57.022927253Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_OUTPUT -o lo -j RETURN -m comment --comment proxy-init/ignore-loopback/1718174156"
2024-06-12T06:35:57.025502354Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_OUTPUT -p tcp --match multiport --dports 8090 -j RETURN -m comment --comment proxy-init/ignore-port-8090/1718174156"
2024-06-12T06:35:57.118326702Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A PROXY_INIT_OUTPUT -p tcp -j REDIRECT --to-port 4140 -m comment --comment proxy-init/redirect-all-outgoing-to-proxy-port/1718174156"
2024-06-12T06:35:57.120459303Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables -t nat -A OUTPUT -j PROXY_INIT_OUTPUT -m comment --comment proxy-init/install-proxy-init-output/1718174156"
2024-06-12T06:35:57.122458304Z time="2024-06-12T06:35:57Z" level=info msg="/sbin/iptables-save -t nat"
2024-06-12T06:35:57.218168653Z time="2024-06-12T06:35:57Z" level=info msg="# Generated by iptables-save v1.8.8 on Wed Jun 12 06:35:57 2024\n*nat\n:PREROUTING ACCEPT [0:0]\n:INPUT ACCEPT [0:0]\n:OUTPUT ACCEPT [0:0]\n:POSTROUTING ACCEPT [0:0]\n:PROXY_INIT_OUTPUT - [0:0]\n:PROXY_INIT_REDIRECT - [0:0]\n-A PREROUTING -m comment --comment \"proxy-init/install-proxy-init-prerouting/1718174156\" -j PROXY_INIT_REDIRECT\n-A OUTPUT -m comment --comment \"proxy-init/install-proxy-init-output/1718174156\" -j PROXY_INIT_OUTPUT\n-A PROXY_INIT_OUTPUT -m owner --uid-owner 2102 -m comment --comment \"proxy-init/ignore-proxy-user-id/1718174156\" -j RETURN\n-A PROXY_INIT_OUTPUT -o lo -m comment --comment \"proxy-init/ignore-loopback/1718174156\" -j RETURN\n-A PROXY_INIT_OUTPUT -p tcp -m multiport --dports 8090 -m comment --comment \"proxy-init/ignore-port-8090/1718174156\" -j RETURN\n-A PROXY_INIT_OUTPUT -p tcp -m comment --comment \"proxy-init/redirect-all-outgoing-to-proxy-port/1718174156\" -j REDIRECT --to-ports 4140\n-A PROXY_INIT_REDIRECT -p tcp -m multiport --dports 4190,4191,8090 -m comment --comment \"proxy-init/ignore-port-4190,4191,8090/1718174156\" -j RETURN\n-A PROXY_INIT_REDIRECT -p tcp -m comment --comment \"proxy-init/redirect-all-incoming-to-proxy-port/1718174156\" -j REDIRECT --to-ports 4143\nCOMMIT\n# Completed on Wed Jun 12 06:35:57 2024\n"

This is not a typical occurence. Are you using any other features such as native sidecars? I am not aware of such thing

wamak9 avatar Jun 13 '24 15:06 wamak9 

2024-06-06T14:03:06.497366974Z [ 61799.863570s]  INFO ThreadId(01) inbound:server{port=80}:rescue{client.addr=10.109.192.187:37222}: linkerd_app_core::errors::respond: HTTP/1.1 request failed error=error trying to connect: Connection refused (os error 111) error.sources=[Connection refused (os error 111)]

This indicates that the proxy is trying to connect to port 80 in the local pod. Does the workload have a server that is listening on port 80?

If not, this could indicate that there are clients (port scanners?) in your network trying to connect to arbitrary ports across an IP range, or it could indicate a misconfigured client. If you expect your application to serve traffic on port 80, then I would suggest looking at the application server to see if it is healthy.

olix0r avatar Sep 10 '24 17:09 olix0r

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Dec 19 '24 01:12 stale[bot]