deployments-k8s
deployments-k8s copied to clipboard
Published
20 hours ago •
networkservicemesh
networkservicemesh
NSM in kubeadm cluster
Question
Hello i am using the basic examples for the NSM in a kind cluster which they work fine, but when i try the same examples on my "production" cluster, some of the pods are stack in the init state or even when they are running they are not working properly.
Examples
Lets try an example of kernel to IP to kernel connection on a Kind cluster.
ns-kernel2ip2kernel alpine 2/2 Running 0
ns-kernel2ip2kernel nse-kernel-5787687848-5dc4z 1/1 Running 0
I can ping from the NSC to NSE and vice versa without a problem.
On the other hand, in my "production" cluster, the alpine pod is stack on the init state.
ns-kernel2ip2kernel alpine 0/2 Init:0/1
ns-kernel2ip2kernel nse-kernel-78dd9c4dfd-rq42f 1/1 Running
The logs of the alpine pod are the following:
kubectl logs -n ns-kernel2ip2kernel alpine cmd-nsc-init:
This application is configured via the environment. The following environment
variables can be used:
KEY TYPE DEFAULT REQUIRED DESCRIPTION
NSM_NAME String cmd-nsc-init Name of the client
NSM_DIAL_TIMEOUT Duration 5s timeout to dial NSMgr
NSM_REQUEST_TIMEOUT Duration 15s timeout to request NSE
NSM_CONNECT_TO URL unix:///var/lib/networkservicemesh/nsm.io.sock url to connect to
NSM_MAX_TOKEN_LIFETIME Duration 10m maximum lifetime of tokens
NSM_NETWORK_SERVICES Comma-separated list of URL A list of Network Service Requests
NSM_AWARENESS_GROUPS Comma-separated list of Comma-separated list of URL Awareness groups for mutually aware NSEs
NSM_LOG_LEVEL String INFO Log level
NSM_OPEN_TELEMETRY_ENDPOINT String otel-collector.observability.svc.cluster.local:4317 OpenTelemetry Collector Endpoint
NSM_METRICS_EXPORT_INTERVAL Duration 10s interval between mertics exports
time="2024-05-16T22:28:36Z" level=info msg="Starting NetworkServiceMesh Client ..."
May 16 22:28:36.935 [INFO] [cmd:[/bin/app]] rootConf: &{Name:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0 DialTimeout:5s RequestTimeout:15s ConnectTo:{Scheme:unix Opaque: User: Host: Path:/var/lib/networkservicemesh/nsm.io.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:} MaxTokenLifetime:10m0s NetworkServices:[{Scheme:kernel Opaque: User: Host:kernel2ip2kernel Path:/nsm-1 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}] AwarenessGroups:[] LogLevel:TRACE OpenTelemetryEndpoint:otel-collector.observability.svc.cluster.local:4317 MetricsExportInterval:10s}
May 16 22:28:38.177 [INFO] [cmd:[/bin/app]] sVID: "spiffe://k8s.nsm/ns/ns-kernel2ip2kernel/pod/alpine"
May 16 22:28:38.177 [INFO] [cmd:[/bin/app]] NSC: Connecting to Network Service Manager unix:///var/lib/networkservicemesh/nsm.io.sock
May 16 22:28:38.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:28:38.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1) request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2) request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2) ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4) ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:28:38.189 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1) Environment variable NODE_NAME is not set. Skipping.
May 16 22:28:38.189 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2) Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3) request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5) ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1) request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8) ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:28:38.189 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9) ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11) ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:28:38.190 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12) ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1) rpc error: code = DeadlineExceeded desc = context deadline exceeded; Error returned from api/pkg/api/networkservice/networkServiceClient.Request; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/common.go:211; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:57;github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/excludedprefixes/client.go:115; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/authorize/client.go:77; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60;
May 16 22:28:53.188 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1) request-response=null
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1) metadata deleted
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.188 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1) request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2) request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2) ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:28:53.396 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4) ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:28:53.396 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1) Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:28:53.396 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2) Environment variable NODE_NAME is not set. Skipping.
May 16 22:28:53.397 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3) request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5) ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:28:53.403 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:28:53.405 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1) request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:28:53.405 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8) ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9) ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:28:53.406 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11) ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:28:53.412 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12) ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1) rpc error: code = DeadlineExceeded desc = context deadline exceeded; Error returned from api/pkg/api/networkservice/networkServiceClient.Request; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/common.go:211; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:57;github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/excludedprefixes/client.go:115; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/authorize/client.go:77; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60;
May 16 22:29:08.393 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1) request-response=null
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded
May 16 22:29:08.393 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1) metadata deleted
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.411 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.412 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: cannot support any of the requested mechanism
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1) request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2) request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2) ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4) ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:29:08.613 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1) Environment variable NODE_NAME is not set. Skipping.
May 16 22:29:08.613 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2) Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3) request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5) ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1) request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8) ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:29:08.613 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9) ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11) ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:29:08.614 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12) ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12.1) rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed; Error returned from api/pkg/api/networkservice/networkServiceClient.Request;github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.logError; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/common.go:211; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:57; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/excludedprefixes.(*excludedPrefixesClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/excludedprefixes/client.go:115; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/common/authorize.(*authorizeClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/authorize/client.go:77; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*beginTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:55; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*traceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:49; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60; github.com/networkservicemesh/sdk/pkg/networkservice/core/trace/traceverbose.(*endTraceClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/traceverbose/client.go:81; github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request; /go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:60;
May 16 22:29:23.617 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.1) request-response=null
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.617 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.4) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [DEBU] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [connID:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [metadata:client] [type:networkService] (3.1) metadata deleted
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.3) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.618 [ERRO] [cmd:[/bin/app]] [retryClient:Request] try attempt has failed: Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = 0. An error during select forwawrder forwarder-vpp-dvtct --> Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = DeadlineExceeded desc = context deadline exceeded: all forwarders have failed: cannot support any of the requested mechanism
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request()
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.1) request={"connection":{"id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0","network_service":"kernel2ip2kernel"},"mechanism_preferences":[{"cls":"LOCAL","type":"KERNEL","parameters":{"name":"nsm-1"}}]}
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (1.2) request-diff={"connection":{"path":{"path_segments":{"+0":{"name":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0","id":"alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0"}}}}}
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (2) ⎆ sdk/pkg/networkservice/common/begin/beginClient.Request()
May 16 22:29:23.818 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (3) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4) ⎆ sdk/pkg/networkservice/common/clientinfo/clientInfo.Request()
May 16 22:29:23.819 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.1) Environment variable NODE_NAME is not set. Skipping.
May 16 22:29:23.819 [WARN] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.2) Environment variable CLUSTER_NAME is not set. Skipping.
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (4.3) request-diff={"connection":{"labels":{"+podName":"alpine"}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (5) ⎆ sdk-sriov/pkg/networkservice/common/token/multitoken/tokenClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (6) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (7.1) request-diff={"connection":{"path":{"path_segments":{"0":{"metrics":{"+client_interface":"KERNEL/nsm-1"}}}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8) ⎆ sdk/pkg/networkservice/common/mechanisms/kernel/kernelMechanismClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (8.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"+inodeURL":"file:///proc/thread-self/ns/net"}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (9) ⎆ sdk/pkg/networkservice/common/authorize/authorizeClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (10.1) request-diff={"mechanism_preferences":{"0":{"parameters":{"inodeURL":"inode://4/4026533768"}}}}
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (11) ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesClient.Request()
May 16 22:29:23.819 [TRAC] [id:alpine-40babba5-dccc-403a-9e5f-340b151a2ca0-0] [type:networkService] (12) ⎆ api/pkg/api/networkservice/networkServiceClient.Request()
Context
the same error logs are similar with other examples, i suspect that it has to do that i use eBPF instead of kube-proxy (iptables)?
- Kubernetes Version: kubeadm 1.28.3 - ubuntu 22.0.4
- 1 master, 2 worker nodes
- CNI: Cilium without kube-proxy
