sdk
sdk copied to clipboard
Published
20 hours ago •
networkservicemesh
networkservicemesh
connectServer Request fails with context canceled error during refresh period
Expected Behavior
The refresh or healing should work seamlessly when endpoint and client are connected and traffic is running fine among those pods.
Current Behavior
The forwarder's connectServer.Request() fails with context canceled error for unknown reason, this would cause timeout handler to invoke Close of forwarder which would teardown the connection between client and endpoint.
Failure Information (for bugs)
Steps to Reproduce
- Bring up NSC and NSE pods.
- Keep the setup untouched for a while and possibly run the traffic between client and endpoint pods.
- You get the idea...
Context
- Kubernetes Version:
- etc.
Failure Logs
Forwarder logs:
Aug 16 08:29:57.503 [INFO] [name:forwarder-ovs-bvjm6] (1.1) request={"connection":{"id":"b2dd2988-2b73-457e-80eb-82724a67505f","network_service":"icmp-responder","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"dst_ip":"10.10.10.33","dst_port":"4789","src_ip":"10.10.10.22","src_port":"4789","vni":"9070652"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-cb878b7cf-xg2sb","id":"nsc-kernel-cb878b7cf-xg2sb-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.m1KXC1e4COyI0wBO0oXcG8LoX509f7KoP-5FGGfmloZJ15yGq6NAB2keEPQnCRm36V1N4pHn4AZ4OtScCvwlrg","expires":{"seconds":1629103031,"nanos":904438128}},{"name":"nsmgr-52tmw","id":"97560283-dd11-4eda-b1c1-0627ff771784","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.4xq-CNZ1zXG726xOH8BQ-KtTWGxQKq_ydcmafVT9OzCEgGhIZSyMkiYkd5vedgsip6WcQBCiLp-s024ZMX1XBA","expires":{"seconds":1629103032,"nanos":4033021}},{"name":"forwarder-ovs-grmg5","id":"bc44d0f4-0fd6-48e7-951c-a0f17adeb113","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5MTAsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.LmEigJIPaPFEzkHUEwpVwokY4QeC0IhBSJYsptWQLrSRN9h2u_e8anQkA5oQIptgnh_9Wrd-r3kXvS0cyQKA_Q","expires":{"seconds":1629104910}},{"name":"nsmgr-52tmw","id":"c063a9eb-788b-4dec-9b82-63e45fa66aab","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.eZAQacvJML1NCpEpvoIk4IgVKcpPUQOV-DiIxjq5Qm_j34K8z-9adD_NnyOyqYgz34k2HxoCFG6ZyZe4XseO7g","expires":{"seconds":1629103197,"nanos":398897352}},{"name":"nsmgr-7xdp2","id":"b2dd2988-2b73-457e-80eb-82724a67505f","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.Jo1DIzMvDdDu4ut-9H6_3kFgdUQDxcuZIYcR7M-5_xSnJEsQXEujRuw9t0nR9huOwBEeEqZXhVDyrsk4CY5f4g","expires":{"seconds":1629103197,"nanos":402083304}},{"name":"forwarder-ovs-bvjm6","id":"68c8c60b-aae3-4c7f-a736-ce4a35db732c","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5OTIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.BNQ2Mic83a98rEs8HX_AqSbW8emgZGqUYNEPxdgRhIPj9xlflE1Pf2Cnr7DPugUJr97s0NgE1BhFOFOi2LpAyA","expires":{"seconds":1629104992}},{"name":"nsmgr-7xdp2","id":"13f9d5fa-c5b8-4259-af22-45fdd02bc580","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MTAzMDMyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.W3dcWtrmws-OSAPZ_5UQ-iqtS-Q_frDxezWuheXgkaSFNzvzlo6MiFt5Cw0kkyN-Xv8viyKzkfjSqsQWVJyrkw","expires":{"seconds":1629103032,"nanos":304457798}},{"name":"nse-kernel-945c556db-vbblb","id":"b33cdb74-06dd-4ce0-8669-00227aea9209","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMwMzIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.8HOZ20O3yN13zVF0RDpzYxiB_znBpJ8y2NFXkQWglxegTqmen25b8RXUo5mNDXv_EVHNlUR5-FZ6877ks4sruA","expires":{"seconds":1629103032,"nanos":307027675}}]},"network_service_endpoint_name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
Aug 16 08:29:57.503 [INFO] [name:forwarder-ovs-bvjm6] (1.2) request-diff={"connection":{"id":"68c8c60b-aae3-4c7f-a736-ce4a35db732c","path":{"index":5,"path_segments":{"4":{"expires":{"nanos":502907557},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.f-1MkV7Iak4pEtUUQNOBxsLjtVidl7hCpPDnHgLq1ZbuF7MIRFk1J00IKXV7eFbMpc3PA01cp_gWp63aWFTNBQ"}}}}} span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (2) ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (3) ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (4) ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (5) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
Aug 16 08:29:57.504 [INFO] [name:forwarder-ovs-bvjm6] (6) ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (7) ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (7.1) request-diff={"connection":{"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDQ5OTIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.dDNFKVHghBhHuphf7KROTaiCIcQncbCF-AFWbpcJn5SLWJYUj5axmwgUaK5S3gyiBPFaYGPioRR5NtkS0Osq2g"}}}}} span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (8) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
Aug 16 08:29:57.505 [INFO] [name:forwarder-ovs-bvjm6] (9) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (11) ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (12) ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (13) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (14) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
Aug 16 08:29:57.506 [INFO] [name:forwarder-ovs-bvjm6] (15) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (16) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (17) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (18) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (19) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
Aug 16 08:29:57.507 [INFO] [name:forwarder-ovs-bvjm6] (20) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (21) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (22) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.508 [INFO] [name:forwarder-ovs-bvjm6] (23) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
Aug 16 08:29:57.509 [ERRO] [name:forwarder-ovs-bvjm6] (23.1) context canceled
Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:60
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress.(*ipaddressServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/server.go:64
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes.(*routesServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors.(*ipNeighborsServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/server.go:37
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu.(*mtuServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69 span=406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:321e4097129c2390:32b6d8b3862ab7df:1
Aug 16 08:29:57.509 [INFO] [name:forwarder-ovs-bvjm6] (22.1) response=null span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.510 [ERRO] [name:forwarder-ovs-bvjm6] (22.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:32b6d8b3862ab7df:7abea4adbbac6e28:1
Aug 16 08:29:57.511 [ERRO] [name:forwarder-ovs-bvjm6] (21.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:7abea4adbbac6e28:24ccaa524beaae0a:1
Aug 16 08:29:57.512 [ERRO] [name:forwarder-ovs-bvjm6] (20.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:24ccaa524beaae0a:5161149fda649b5b:1
Aug 16 08:29:57.513 [ERRO] [name:forwarder-ovs-bvjm6] (19.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5161149fda649b5b:1bac7b46eabeb2d2:1
Aug 16 08:29:57.514 [ERRO] [name:forwarder-ovs-bvjm6] (18.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1bac7b46eabeb2d2:449d9837ab879a96:1
Aug 16 08:29:57.515 [ERRO] [name:forwarder-ovs-bvjm6] (17.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:449d9837ab879a96:05242dd3140a274b:1
Aug 16 08:29:57.516 [ERRO] [name:forwarder-ovs-bvjm6] (16.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:05242dd3140a274b:66dbcb9e112799fe:1
Aug 16 08:29:57.516 [ERRO] [name:forwarder-ovs-bvjm6] (15.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:66dbcb9e112799fe:67a200715d6291bc:1
Aug 16 08:29:57.517 [ERRO] [name:forwarder-ovs-bvjm6] (14.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:67a200715d6291bc:27aeea0d0c771228:1
Aug 16 08:29:57.518 [ERRO] [name:forwarder-ovs-bvjm6] (13.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:27aeea0d0c771228:28ce5cdd432321ee:1
Aug 16 08:29:57.518 [ERRO] [name:forwarder-ovs-bvjm6] (12.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:28ce5cdd432321ee:0d1c3b85f6bcca6f:1
Aug 16 08:29:57.519 [ERRO] [name:forwarder-ovs-bvjm6] (11.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0d1c3b85f6bcca6f:5333ce0813706551:1
Aug 16 08:29:57.521 [ERRO] [name:forwarder-ovs-bvjm6] (10.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5333ce0813706551:191949718b4ad4c5:1
Aug 16 08:29:57.521 [ERRO] [name:forwarder-ovs-bvjm6] (9.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:191949718b4ad4c5:20e005e6da164b4e:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (8.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:20e005e6da164b4e:47292580b1921315:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (7.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:47292580b1921315:60118e58fcd9b69e:1
Aug 16 08:29:57.522 [ERRO] [name:forwarder-ovs-bvjm6] (6.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:60118e58fcd9b69e:79fe37457dce6fa3:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (5.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:79fe37457dce6fa3:266fcaf4a3af11f2:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (4.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:266fcaf4a3af11f2:1c10e9ede3bf742a:1
Aug 16 08:29:57.523 [ERRO] [name:forwarder-ovs-bvjm6] (3.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1c10e9ede3bf742a:52338a96b3ce83e9:1
Aug 16 08:29:57.524 [ERRO] [name:forwarder-ovs-bvjm6] (2.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:52338a96b3ce83e9:56edd9f2d5712279:1
Aug 16 08:29:57.524 [ERRO] [name:forwarder-ovs-bvjm6] (1.3) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:56edd9f2d5712279:6dbba770b04a0379:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:6dbba770b04a0379:77df86fd1c366362:1
NSMgr logs:
Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (1.2) request-diff={"connection":{"id":"b2dd2988-2b73-457e-80eb-82724a67505f","path":{"index":4,"path_segments":{"3":{"expires":{"nanos":398897352,"seconds":1629103197},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.eZAQacvJML1NCpEpvoIk4IgVKcpPUQOV-DiIxjq5Qm_j34K8z-9adD_NnyOyqYgz34k2HxoCFG6ZyZe4XseO7g"}}}}} span=406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (2) ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
Aug 16 08:29:57.400 [INFO] [name:nsmgr-7xdp2] (3) ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (4) ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (5) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (6) ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
Aug 16 08:29:57.401 [INFO] [name:nsmgr-7xdp2] (7) ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.402 [INFO] [name:nsmgr-7xdp2] (7.1) request-diff={"connection":{"path":{"path_segments":{"4":{"expires":{"nanos":402083304,"seconds":1629103197},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkxMDMxOTcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.Jo1DIzMvDdDu4ut-9H6_3kFgdUQDxcuZIYcR7M-5_xSnJEsQXEujRuw9t0nR9huOwBEeEqZXhVDyrsk4CY5f4g"}}}}} span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (8) ⎆ sdk/pkg/networkservice/core/adapters/clientToServer.Request() span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.1) Environment variable CLUSTER_NAME is not set. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.2) The label nodeName was already assigned to dl380-004. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [WARN] [name:nsmgr-7xdp2] (8.3) Environment variable POD_NAME is not set. Skipping. span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (9) ⎆ sdk/pkg/networkservice/common/discover/discoverCandidatesServer.Request() span=406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (10) ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Find() span=406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (10.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (11) ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Find() span=406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (11.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (12) ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Find() span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.403 [INFO] [name:nsmgr-7xdp2] (12.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"}} span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.404 [INFO] [name:nsmgr-7xdp2] (12.2) response={"ClientStream":{"ClientStream":{}}} span=406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0b2a7e94aaec15b8:1165320eb7e92320:1
Aug 16 08:29:57.404 [INFO] [name:nsmgr-7xdp2] (13) ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (13.1) response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:336a9d5e80f2e612:0b2a7e94aaec15b8:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (12) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (12.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (11) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (11.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:6978090a448e08ca:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:59732a0aa84259f3:1165320eb7e92320:1
Aug 16 08:29:57.408 [INFO] [name:nsmgr-7xdp2] (13) ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4abe12adcda1a071:0b2a7e94aaec15b8:1
Aug 16 08:29:57.410 [ERRO] [name:nsmgr-7xdp2] (13.1) EOF
Error returned from api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryFindClient).Recv
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/trace/nse_registry.go:52
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindClientToServer
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:107
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:52
github.com/networkservicemesh/sdk/pkg/registry/common/connect.(*connectNSEServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/common/connect/nse_server.go:98
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/trace/nse_registry.go:174
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_server.go:71
github.com/networkservicemesh/sdk/pkg/registry/common/clienturl.(*clientURLNSEServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/common/clienturl/nse_server.go:49
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/trace/nse_registry.go:174
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindServerToClient
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:132
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:85
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_client.go:49
github.com/networkservicemesh/sdk/pkg/registry/common/querycache.(*queryCacheNSEClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/common/querycache/nse_client.go:58
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*callNextNSEClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_next_client.go:35
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_server.go:71
github.com/networkservicemesh/sdk/pkg/registry/common/localbypass.(*localBypassNSEServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/common/localbypass/server.go:77
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryServer).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_server.go:62
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.nseFindServerToClient
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:132
github.com/networkservicemesh/sdk/pkg/registry/core/adapters.(*networkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/adapters/nse_registry.go:85
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/registry/common/serialize.(*serializeNSEClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/common/serialize/nse_client.go:50
github.com/networkservicemesh/sdk/pkg/registry/core/next.(*nextNetworkServiceEndpointRegistryClient).Find
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/next/nse_registry_client.go:47
github.com/networkservicemesh/sdk/pkg/networkservice/common/discover.(*discoverCandidatesServer).discoverNetworkServiceEndpoint
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/discover/server.go:145
github.com/networkservicemesh/sdk/pkg/networkservice/common/discover.(*discoverCandidatesServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/discover/server.go:61
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/adapters.(*callNextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/adapters/client_to_server.go:53 span=406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4eb03cbc01f5ad16:0b2a7e94aaec15b8:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:1165320eb7e92320:5b54b23309b8047f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5b54b23309b8047f:08c02d56e082bd1d:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (2) ⎆ sdk/pkg/registry/common/clienturl/clientURLNSEServer.Find() span=37c32fda70cfd60a:774519f395ee7ba6:37c32fda70cfd60a:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (2.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:774519f395ee7ba6:37c32fda70cfd60a:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (3) ⎆ sdk/pkg/registry/common/connect/connectNSEServer.Find() span=37c32fda70cfd60a:477fa58f6514c26b:774519f395ee7ba6:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (3.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:477fa58f6514c26b:774519f395ee7ba6:1
Aug 16 08:29:57.410 [INFO] [name:nsmgr-7xdp2] (10) ⎆ sdk/pkg/networkservice/common/roundrobin/selectEndpointServer.Request() span=406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4) ⎆ sdk/pkg/registry/common/connect/grpcNSEClient.Find() span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4.1) find={"network_service_endpoint":{"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb"},"watch":true} span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.410 [INFO] [cmd:Nsmgr] (4.2) response={"ClientStream":{"ClientStream":{}}} span=37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:0b9b73968be5bc1c:477fa58f6514c26b:1
Aug 16 08:29:57.411 [INFO] [cmd:Nsmgr] (5) ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (11) ⎆ sdk/pkg/networkservice/common/excludedprefixes/excludedPrefixesServer.Request() span=406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (12) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
Aug 16 08:29:57.411 [INFO] [name:nsmgr-7xdp2] (13) ⎆ sdk/pkg/networkservice/common/interpose/interposeServer.Request() span=406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5.1) response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:072eb83ff79856ba:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [name:nsmgr-7xdp2] (14) ⎆ sdk/pkg/networkservice/common/filtermechanisms/filterMechanismsServer.Request() span=406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
Aug 16 08:29:57.500 [INFO] [name:nsmgr-7xdp2] (15) ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:7f9fa62075e341fe:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:638a2882e63ead96:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5) ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (5.1) response={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:0d13a554f914f1c0:0b9b73968be5bc1c:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (4.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.500 [INFO] [cmd:Nsmgr] (3) ⎆ sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send() span=37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
Aug 16 08:29:57.501 [INFO] [cmd:Nsmgr] (3.1) network service endpoint={"name":"0c19a95a-fdc8-4dd5-b6a4-1ac0e0260a89-nse-kernel-945c556db-vbblb","network_service_names":["icmp-responder"],"network_service_labels":{"icmp-responder":{"labels":{"nodeName":"dl380-003"}}},"url":"tcp://192.168.185.26:5001","expiration_time":{"seconds":1629102657,"nanos":176048862}} span=37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:713c0a89d0077871:774519f395ee7ba6:1
2021/08/16 08:29:57 Reporting span 37c32fda70cfd60a:41644ae95fa9f421:477fa58f6514c26b:1
Aug 16 08:29:57.501 [INFO] [cmd:Nsmgr] (5) ⎆ api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv() span=37c32fda70cfd60a:15a165de952b654c:0b9b73968be5bc1c:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (16) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (17) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (18) ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (19) ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (20) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
Aug 16 08:29:57.501 [INFO] [name:nsmgr-7xdp2] (21) ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (22) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (23) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (24) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.502 [INFO] [name:nsmgr-7xdp2] (25) ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:77df86fd1c366362:3e7fb8bed835e4e9:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (25.1) rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled
Error returned from api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:62
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/null.(*nullClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/null/client.go:43
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/recvfd/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/heal.(*healClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/heal/client.go:106
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/utils/metadata.(*metaDataClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/utils/metadata/client.go:45
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69 span=406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3e7fb8bed835e4e9:32bd5e99745b57f7:1
Aug 16 08:29:57.525 [INFO] [name:nsmgr-7xdp2] (24.1) response=null span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (24.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:32bd5e99745b57f7:0ef2bb586c60e040:1
Aug 16 08:29:57.525 [ERRO] [name:nsmgr-7xdp2] (23.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:0ef2bb586c60e040:50d728211e48ca9f:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (22.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:50d728211e48ca9f:2f40b5a0da0f00a3:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (21.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:2f40b5a0da0f00a3:5e5acdd1a7dbc069:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (20.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5e5acdd1a7dbc069:548bf9a8352571ff:1
Aug 16 08:29:57.526 [ERRO] [name:nsmgr-7xdp2] (19.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:548bf9a8352571ff:4692d25faa6611cf:1
Aug 16 08:29:57.527 [ERRO] [name:nsmgr-7xdp2] (18.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4692d25faa6611cf:12d66789fd896f67:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/serialize/serializeClient.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.600 [ERRO] [name:nsmgr-7xdp2] (17.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:12d66789fd896f67:38a7ac8a3d90e738:1
Aug 16 08:29:57.601 [ERRO] [name:nsmgr-7xdp2] (16.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:38a7ac8a3d90e738:54d359380244f3b3:1
Aug 16 08:29:57.602 [ERRO] [name:nsmgr-7xdp2] (15.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:54d359380244f3b3:5ea8f8ccdea881e0:1
Aug 16 08:29:57.603 [ERRO] [name:nsmgr-7xdp2] (14.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5ea8f8ccdea881e0:3460afd7c9dc355f:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/filtermechanisms/filterMechanismsServer.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.604 [ERRO] [name:nsmgr-7xdp2] (13.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3460afd7c9dc355f:033390a7d018b022:1
Aug 16 08:29:57.604 [ERRO] [name:nsmgr-7xdp2] (12.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:033390a7d018b022:4a597879c775d5e9:1
Aug 16 08:29:57.605 [ERRO] [name:nsmgr-7xdp2] (11.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:4a597879c775d5e9:3a302fe2e7c85311:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (10.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3a302fe2e7c85311:08c02d56e082bd1d:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (9.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:08c02d56e082bd1d:23793339b1f8e6c7:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (8.4) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:23793339b1f8e6c7:367bfc5f5ae265e6:1
Aug 16 08:29:57.606 [ERRO] [name:nsmgr-7xdp2] (7.2) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:367bfc5f5ae265e6:13f49d421954e6c0:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (6.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:13f49d421954e6c0:5be08f649388c434:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (5.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:5be08f649388c434:3158c52cd1f82f1c:1
2021/08/16 08:29:57 ERROR: error reporting Jaeger span "sdk/pkg/networkservice/common/monitor/monitorServer.Request": write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (4.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3158c52cd1f82f1c:3e2fa8b80cb704d3:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (3.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:3e2fa8b80cb704d3:016f320a847004f0:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (2.1) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:016f320a847004f0:00a4f28db9521d81:1
Aug 16 08:29:57.607 [ERRO] [name:nsmgr-7xdp2] (1.3) Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:00a4f28db9521d81:70606533bfc55800:1
2021/08/16 08:29:57 Reporting span 406cdaa5e60e81a2:70606533bfc55800:41caa70e1b77d793:1
2021/08/16 08:29:59 Reporting span 36ba8cd186c2dc89:36ba8cd186c2dc89:0000000000000000:1
2021/08/16 08:29:59 ERROR: failed to flush Jaeger spans to server: write udp 127.0.0.1:58831->127.0.0.1:6831: write: connection refused
2021/08/16 08:30:06 Reporting span 7d10e663f026ed3f:7d10e663f026ed3f:0000000000000000:1
@Bolodya1997 as discussed, i tried with latest NSM images and issue still exists.
Here is the forwarder log:
Aug 17 12:54:42.219[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.1) request={"connection":{"id":"a5052de1-cb38-44c0-ae11-edb4730ae064","network_service":"icmp-responder","mechanism":{"cls":"REMOTE","type":"VXLAN","parameters":{"dst_ip":"10.10.10.33","dst_port":"4789","src_ip":"10.10.10.22","src_port":"4789","vni":"9832580"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.0uXakzic0NC44YVPyyDA_32jy9qQrIShPT5E5Hva0KWZOSBts0FthOtLu7JcuiHrQ_XQr30oF7ByUrpvm4s48Q","expires":{"seconds":1629205482,"nanos":103258984}},{"name":"forwarder-ovs-vc7qb","id":"df936088-582e-41db-8205-32e8d8b849b6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.YRK2sbi52Tk7OfIGhocjZrHq_nOAXlgCg9-tceigiyK5bsAHMAe_1oMBmZswke2eNVpMn1bcF7mdHUzMH3PuHg","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"5be19c57-bff5-467d-be43-6eb2853c83e2","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.UlrMLPTy01M7Bjbded8Fr1O-hRtwa1-bweunvEfZInYy_fGxBx37Osrwvw42Zckz4lSpkS-JLuKzdJTdWBm7tg","expires":{"seconds":1629205482,"nanos":205443660}},{"name":"nsmgr-8hs7f","id":"a5052de1-cb38-44c0-ae11-edb4730ae064","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.d2D6DxI7rJZhU0AQsb-jiohPN8mBbxeFmofCdJ5hiEF4uTW5uEzOJzO30MmHgMwL8nmVmU3k6XJO6Agji4d3GA","expires":{"seconds":1629205482,"nanos":208094292}},{"name":"forwarder-ovs-jck5d","id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.3R2_rMVDKSdrYY5Isrh48BK2zVelfNuf8Zu7GCNJOmCZVJFmfSPYFBW6hWKqvemaTwcHqvIeIZTvraOIV1_vKQ","expires":{"seconds":1629207419}},{"name":"nsmgr-8hs7f","id":"fbcd7003-f6ad-471b-9bad-f212e54566cd","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1MzYxLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.CH4-Zhfd-pWO14H7pVgoBXsvBbXLrKlwHUR0Zd9--LCcUMO-OsxqJtOlzg9VlkbVeoph0Ad8duTd1MkqYNFX1Q","expires":{"seconds":1629205361,"nanos":420051348}},{"name":"nse-kernel-6444cdcc55-jjjfv","id":"356b02b4-7bc0-45fe-86e2-70ccd4358632","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDUzNjEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.duPMH9KJYAS75akCaUIc0wHssP73AcBQgq_B6P96_lcyoKnmi2ocXQ64X0DrqnhhbjZqS9uOKgFbDIQFiJk92g","expires":{"seconds":1629205361,"nanos":422670822}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.2) request-diff={"connection":{"id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","path":{"index":5,"path_segments":{"4":{"expires":{"nanos":217344406},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.il2Xax2TBZ6CATGWNuQvF4D5O-nlv7LYTkfhw8tzl-lShp3OM-J5qVk0mmzJ5_Xfkko-lNrGakfI3nf0GTYrvQ"}}}}} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
Aug 17 12:54:42.219[36m [INFO] [name:forwarder-ovs-jck5d] [0m(2) ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=03e9f52b95487f7f:3d9c60d29dff250f:7a7cce123fb62752:1
Aug 17 12:54:42.220[36m [INFO] [name:forwarder-ovs-jck5d] [0m(3) ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=03e9f52b95487f7f:609c63bb3c8de1a7:3d9c60d29dff250f:1
Aug 17 12:54:42.221[36m [INFO] [name:forwarder-ovs-jck5d] [0m(4) ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=03e9f52b95487f7f:76f00b136766d430:609c63bb3c8de1a7:1
Aug 17 12:54:42.221[36m [INFO] [name:forwarder-ovs-jck5d] [0m(5) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=03e9f52b95487f7f:6ee58b63ac81944e:76f00b136766d430:1
Aug 17 12:54:42.222[36m [INFO] [name:forwarder-ovs-jck5d] [0m(6) ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=03e9f52b95487f7f:7190c8ee82847f23:6ee58b63ac81944e:1
Aug 17 12:54:42.222[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7) ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
Aug 17 12:54:42.222[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7.1) request-diff={"connection":{"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.oqz1j0DMd4uHsb2LyuRoCFJe9pcskLy5lZ-H5_6Jpc-xNBBhFeumRVba_R57FzpCWD3W-bn0PwrKBHR__AoWAw"}}}}} span=03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
Aug 17 12:54:42.222[36m [INFO] [name:forwarder-ovs-jck5d] [0m(8) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=03e9f52b95487f7f:209c9732b7709b77:485fb6841c630633:1
Aug 17 12:54:42.222[36m [INFO] [name:forwarder-ovs-jck5d] [0m(9) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=03e9f52b95487f7f:303a460c70ffe464:209c9732b7709b77:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=03e9f52b95487f7f:04996bd5df684e5c:303a460c70ffe464:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(11) ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=03e9f52b95487f7f:12753f2931c02cb3:04996bd5df684e5c:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(12) ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=03e9f52b95487f7f:5fff97c6694d58a1:12753f2931c02cb3:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(13) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=03e9f52b95487f7f:25468e3012761c18:5fff97c6694d58a1:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(14) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=03e9f52b95487f7f:2a271d7f7444fb04:25468e3012761c18:1
Aug 17 12:54:42.223[36m [INFO] [name:forwarder-ovs-jck5d] [0m(15) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=03e9f52b95487f7f:6365ab165c9d2342:2a271d7f7444fb04:1
Aug 17 12:54:42.224[36m [INFO] [name:forwarder-ovs-jck5d] [0m(16) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=03e9f52b95487f7f:720a4aff192ac769:6365ab165c9d2342:1
Aug 17 12:54:42.224[36m [INFO] [name:forwarder-ovs-jck5d] [0m(17) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=03e9f52b95487f7f:63d6a7283737ab3c:720a4aff192ac769:1
Aug 17 12:54:42.224[36m [INFO] [name:forwarder-ovs-jck5d] [0m(18) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=03e9f52b95487f7f:4cfad252feee66ca:63d6a7283737ab3c:1
Aug 17 12:54:42.225[36m [INFO] [name:forwarder-ovs-jck5d] [0m(19) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=03e9f52b95487f7f:09678d30a096de49:4cfad252feee66ca:1
Aug 17 12:54:42.225[36m [INFO] [name:forwarder-ovs-jck5d] [0m(20) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=03e9f52b95487f7f:584d665bbbbc6573:09678d30a096de49:1
Aug 17 12:54:42.225[36m [INFO] [name:forwarder-ovs-jck5d] [0m(21) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=03e9f52b95487f7f:08776bc38c65c580:584d665bbbbc6573:1
Aug 17 12:54:42.225[36m [INFO] [name:forwarder-ovs-jck5d] [0m(22) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=03e9f52b95487f7f:0e18efecba32b8ea:08776bc38c65c580:1
Aug 17 12:54:42.226[36m [INFO] [name:forwarder-ovs-jck5d] [0m(23) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
Aug 17 12:54:42.226[36m [INFO] [name:forwarder-ovs-jck5d] [0m(24) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=03e9f52b95487f7f:7e05a0d199774e24:527ba635996fad7d:1
Aug 17 12:54:42.226[36m [INFO] [name:forwarder-ovs-jck5d] [0m(25) ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=03e9f52b95487f7f:541f71b4a17c3c8f:7e05a0d199774e24:1
Aug 17 12:54:42.227[36m [INFO] [name:forwarder-ovs-jck5d] [0m(26) ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=03e9f52b95487f7f:19fb894ed9f7f635:541f71b4a17c3c8f:1
Aug 17 12:54:42.227[36m [INFO] [name:forwarder-ovs-jck5d] [0m(27) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=03e9f52b95487f7f:3b56478b9a5c05c8:19fb894ed9f7f635:1
Aug 17 12:54:42.227[36m [INFO] [name:forwarder-ovs-jck5d] [0m(28) ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=03e9f52b95487f7f:2eee0b6eb144c851:3b56478b9a5c05c8:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(29) ⎆ sdk/pkg/networkservice/common/mechanismtranslation/mechanismTranslationClient.Request() span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(29.1) request-diff={"connection":{"mechanism":{"cls":"LOCAL","parameters":{"+inodeURL":"file:///proc/51245/fd/12","+name":"icmp-respo-356b","-dst_ip":"10.10.10.33","-dst_port":"4789","-src_ip":"10.10.10.22","-src_port":"4789","-vni":"9832580"},"type":"KERNEL"}},"mechanism_preferences":{"-0":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}}} span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(30) ⎆ sdk-ovs/pkg/networkservice/l2ovsconnect/l2ConnectClient.Request() span=03e9f52b95487f7f:7cd300ebba7d3cee:3fbd6ab211997370:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(31) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=03e9f52b95487f7f:0ff81ec2ba0aa6bf:7cd300ebba7d3cee:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(32) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuClient.Request() span=03e9f52b95487f7f:562cd69da38879ba:0ff81ec2ba0aa6bf:1
Aug 17 12:54:42.228[36m [INFO] [name:forwarder-ovs-jck5d] [0m(33) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsClient.Request() span=03e9f52b95487f7f:089e439bf8c15434:562cd69da38879ba:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(34) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesClient.Request() span=03e9f52b95487f7f:0ef229f0482264e0:089e439bf8c15434:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(35) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressClient.Request() span=03e9f52b95487f7f:5e3a1e0835a4996e:0ef229f0482264e0:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(36) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectClient.Request() span=03e9f52b95487f7f:40d6a179174cf83f:5e3a1e0835a4996e:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(37) ⎆ sdk-ovs/pkg/networkservice/mechanisms/kernel/kernelClient.Request() span=03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(37.1) request-diff={"mechanism_preferences":{"+0":{"cls":"LOCAL","type":"KERNEL"}}} span=03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
Aug 17 12:54:42.229[36m [INFO] [name:forwarder-ovs-jck5d] [0m(38) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=03e9f52b95487f7f:155b9120674b1516:4b0811a7a2465bff:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(39) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanClient.Request() span=03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(39.1) request-diff={"mechanism_preferences":{"+1":{"cls":"REMOTE","type":"VXLAN"}}} span=03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(40) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniClient.Request() span=03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(40.1) request-diff={"mechanism_preferences":{"1":{"parameters":{"+src_ip":"10.10.10.33","+src_port":"4789"}}}} span=03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(41) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=03e9f52b95487f7f:5500bc4a788a7161:7086d6c41d5b7ad4:1
Aug 17 12:54:42.230[36m [INFO] [name:forwarder-ovs-jck5d] [0m(42) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
Aug 17 12:54:42.231[36m [INFO] [name:forwarder-ovs-jck5d] [0m(42.1) request-diff={"connection":{"mechanism":{"parameters":{"inodeURL":"inode://3/4026535837"}}}} span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
Aug 17 12:54:42.231[36m [INFO] [name:forwarder-ovs-jck5d] [0m(43) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=03e9f52b95487f7f:164d524467bbb39d:48252471ad377f27:1
Aug 17 12:54:42.231[36m [INFO] [name:forwarder-ovs-jck5d] [0m(44) ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
Aug 17 12:54:42.403[36m [INFO] [name:forwarder-ovs-jck5d] [0m(44.1) response={"id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","network_service":"icmp-responder","mechanism":{"cls":"LOCAL","type":"KERNEL","parameters":{"inodeURL":"inode://3/4026535837","name":"icmp-respo-356b"}},"context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":5,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.0uXakzic0NC44YVPyyDA_32jy9qQrIShPT5E5Hva0KWZOSBts0FthOtLu7JcuiHrQ_XQr30oF7ByUrpvm4s48Q","expires":{"seconds":1629205482,"nanos":103258984}},{"name":"forwarder-ovs-vc7qb","id":"df936088-582e-41db-8205-32e8d8b849b6","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.YRK2sbi52Tk7OfIGhocjZrHq_nOAXlgCg9-tceigiyK5bsAHMAe_1oMBmZswke2eNVpMn1bcF7mdHUzMH3PuHg","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"5be19c57-bff5-467d-be43-6eb2853c83e2","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.UlrMLPTy01M7Bjbded8Fr1O-hRtwa1-bweunvEfZInYy_fGxBx37Osrwvw42Zckz4lSpkS-JLuKzdJTdWBm7tg","expires":{"seconds":1629205482,"nanos":205443660}},{"name":"nsmgr-8hs7f","id":"a5052de1-cb38-44c0-ae11-edb4730ae064","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.il2Xax2TBZ6CATGWNuQvF4D5O-nlv7LYTkfhw8tzl-lShp3OM-J5qVk0mmzJ5_Xfkko-lNrGakfI3nf0GTYrvQ","expires":{"seconds":1629205482,"nanos":217344406}},{"name":"forwarder-ovs-jck5d","id":"dadfb4ac-a7ae-4be8-9fcf-f60fcd0a9009","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.C5l2Q_DY_ttt-IIiQHBfbfZKwnsZ1bKhj5Cb--lYac9oz-TvCLMUpDG_x4QHJ-NJOG4jx8LaZR6b_paxcxMc1w","expires":{"seconds":1629207419}},{"name":"nsmgr-8hs7f","id":"fbcd7003-f6ad-471b-9bad-f212e54566cd","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1NDgyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.s-zXOGagdPesvWqpJcHp_3TnFMLEJv4dqcziSgSKYl4r1-I9n-L5DF9b9ViDbtyUMPNiTCNZqoP-G3c3-S20PQ","expires":{"seconds":1629205482,"nanos":301924123}},{"name":"nse-kernel-6444cdcc55-jjjfv","id":"356b02b4-7bc0-45fe-86e2-70ccd4358632","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.QE09R28E5dsyC7t95845MkkvvBRNb04NfSN0kXm1ITfFCCwJLANvbJY5MpeJWW52J43b5T3YzMu5LSi7dMMq9Q","expires":{"seconds":1629205482,"nanos":305529376}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"} span=03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5eebd95730e3f5c3:164d524467bbb39d:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:164d524467bbb39d:48252471ad377f27:1
Aug 17 12:54:42.403[36m [INFO] [name:forwarder-ovs-jck5d] [0m(42.2) response-diff={"mechanism":{"parameters":{"inodeURL":"file:///proc/51245/fd/12"}}} span=03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:48252471ad377f27:5500bc4a788a7161:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5500bc4a788a7161:7086d6c41d5b7ad4:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7086d6c41d5b7ad4:0dfe92a2ba8c3001:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0dfe92a2ba8c3001:155b9120674b1516:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:155b9120674b1516:4b0811a7a2465bff:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:4b0811a7a2465bff:40d6a179174cf83f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:40d6a179174cf83f:5e3a1e0835a4996e:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5e3a1e0835a4996e:0ef229f0482264e0:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0ef229f0482264e0:089e439bf8c15434:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:089e439bf8c15434:562cd69da38879ba:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:562cd69da38879ba:0ff81ec2ba0aa6bf:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0ff81ec2ba0aa6bf:7cd300ebba7d3cee:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7cd300ebba7d3cee:3fbd6ab211997370:1
Aug 17 12:54:42.406[36m [INFO] [name:forwarder-ovs-jck5d] [0m(29.2) response-diff={"mechanism":{"cls":"REMOTE","parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+src_ip":"10.10.10.22","+src_port":"4789","+vni":"9832580","-inodeURL":"file:///proc/51245/fd/12","-name":"icmp-respo-356b"},"type":"VXLAN"}} span=03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3fbd6ab211997370:2eee0b6eb144c851:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:2eee0b6eb144c851:3b56478b9a5c05c8:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3b56478b9a5c05c8:19fb894ed9f7f635:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:19fb894ed9f7f635:541f71b4a17c3c8f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:541f71b4a17c3c8f:7e05a0d199774e24:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7e05a0d199774e24:527ba635996fad7d:1
Aug 17 12:54:42.407[36m [INFO] [name:forwarder-ovs-jck5d] [0m(23.1) request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+src_ip":"10.10.10.22","+src_port":"4789","+vni":"9832580","-inodeURL":"inode://3/4026535837","-name":"icmp-respo-356b"},"type":"VXLAN"},"path":{"path_segments":{"5":{"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.C5l2Q_DY_ttt-IIiQHBfbfZKwnsZ1bKhj5Cb--lYac9oz-TvCLMUpDG_x4QHJ-NJOG4jx8LaZR6b_paxcxMc1w"},"6":{"expires":{"nanos":301924123,"seconds":1629205482},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9ucy1jNWxtcS9zYS9kZWZhdWx0IiwiZXhwIjoxNjI5MjA1NDgyLCJzdWIiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQifQ.s-zXOGagdPesvWqpJcHp_3TnFMLEJv4dqcziSgSKYl4r1-I9n-L5DF9b9ViDbtyUMPNiTCNZqoP-G3c3-S20PQ"},"7":{"expires":{"nanos":305529376,"seconds":1629205482},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODIsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.QE09R28E5dsyC7t95845MkkvvBRNb04NfSN0kXm1ITfFCCwJLANvbJY5MpeJWW52J43b5T3YzMu5LSi7dMMq9Q"}}}},"mechanism_preferences":{"-1":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.33","src_port":"4789"}},"0":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:527ba635996fad7d:0e18efecba32b8ea:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:0e18efecba32b8ea:08776bc38c65c580:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:08776bc38c65c580:584d665bbbbc6573:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:584d665bbbbc6573:09678d30a096de49:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:09678d30a096de49:4cfad252feee66ca:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:4cfad252feee66ca:63d6a7283737ab3c:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:63d6a7283737ab3c:720a4aff192ac769:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:720a4aff192ac769:6365ab165c9d2342:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:6365ab165c9d2342:2a271d7f7444fb04:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:2a271d7f7444fb04:25468e3012761c18:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:25468e3012761c18:5fff97c6694d58a1:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:5fff97c6694d58a1:12753f2931c02cb3:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:12753f2931c02cb3:04996bd5df684e5c:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:04996bd5df684e5c:303a460c70ffe464:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:303a460c70ffe464:209c9732b7709b77:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:209c9732b7709b77:485fb6841c630633:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:485fb6841c630633:7190c8ee82847f23:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7190c8ee82847f23:6ee58b63ac81944e:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:6ee58b63ac81944e:76f00b136766d430:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:76f00b136766d430:609c63bb3c8de1a7:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:609c63bb3c8de1a7:3d9c60d29dff250f:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:3d9c60d29dff250f:7a7cce123fb62752:1
Aug 17 12:54:42.411[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.3) response-diff={"id":"a5052de1-cb38-44c0-ae11-edb4730ae064","path":{"index":4}} span=03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:7a7cce123fb62752:25e1e78f96c18120:1
2021/08/17 12:54:42 Reporting span 03e9f52b95487f7f:25e1e78f96c18120:66a47f186fd3a3c9:1
2021/08/17 12:54:45 Reporting span 7adb6f3bf789e1b8:5204e92bfe118ef2:6803c6dd3d17691d:1
Aug 17 12:54:46.118[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.1) request={"connection":{"id":"e43e7390-fb55-467a-a3c7-1dc1a338c644","network_service":"icmp-responder","context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODUsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.OKq5NDUFeRtTLp--fJKxJjosBFigpvJNwjLPRTWbpZxcRav2FAgERXRVxTe9LlYcjn59omnI2pM9IkLwfhaJGA","expires":{"seconds":1629205485,"nanos":912920715}},{"name":"forwarder-ovs-vc7qb","id":"18bd5275-1853-4e09-9ad6-929771551ec5","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.-syXmY0abGHOc8BNLDz82Kkn3zo4yWB3qrf1AV4kAyf7pAdAxsj9otAxDCSy5l77AdiTsvLjBR6_52bc6sRQgQ","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"98e62743-eaae-48ca-ac89-175813a316db","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.EmajpYu5XjSh4jBIklQhKuNIkmTIPPryDVDtFLz-xSHCkrmaA8VnfhV-5Lud7vg38rGC1eakgc45Xja0oeTpsg","expires":{"seconds":1629205486,"nanos":107575748}},{"name":"nsmgr-8hs7f","id":"e43e7390-fb55-467a-a3c7-1dc1a338c644","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.wcBkDpU9T5R0_IEPnoHe2lP00YfCnpJlYige_GY4bFo8MZ3zi5HWWrfTbLUdI1cvHiYZJUve4LOTjdG_Psek1A","expires":{"seconds":1629205486,"nanos":109801924}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.2) request-diff={"connection":{"id":"b128d4d5-6d06-4842-8f01-226e27d5fa92","path":{"index":5,"path_segments":{"+5":{"name":"forwarder-ovs-jck5d","id":"b128d4d5-6d06-4842-8f01-226e27d5fa92"},"4":{"expires":{"nanos":118220347},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODYsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.J483oxn2_J8Q1aKsAQxqfiQ4SXweSIm2q5Y3BxdQ9YeniSVRxxH3skthnlk4EoC6Jhn5bzfpHYSFEtZTi7wtqw"}}}}} span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(2) ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(3) ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(4) ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(5) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
Aug 17 12:54:46.119[36m [INFO] [name:forwarder-ovs-jck5d] [0m(6) ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7) ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7.1) request-diff={"connection":{"path":{"path_segments":{"5":{"expires":{"seconds":1629207419},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.oZ4hn9Dkoakhn31vELixi-vxITRRT7TworlZ6UBlRg8kl9vFFc1zE4jYwwjGsFEXTUKQ-U0G1sdJx9AE4VNTpw"}}}}} span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(8) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(9) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(11) ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
Aug 17 12:54:46.120[36m [INFO] [name:forwarder-ovs-jck5d] [0m(12) ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(13) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(14) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(14.1) request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(15) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(15.1) request-diff={"connection":{"mechanism":{"parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+vni":"9070652"}}}} span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [0m(16) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.1) connection not established, calling add span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.2) inside add method span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.3) mech is vxlan span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.121[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.4) ovs tunnel name: v10101022 span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.125[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.5) ovs tunnel port number: 1 span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.125[36m [INFO] [name:forwarder-ovs-jck5d] [0m(17) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
Aug 17 12:54:46.125[36m [INFO] [name:forwarder-ovs-jck5d] [0m(18) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
Aug 17 12:54:46.125[36m [INFO] [name:forwarder-ovs-jck5d] [0m(19) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
Aug 17 12:54:46.125[36m [INFO] [name:forwarder-ovs-jck5d] [0m(20) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
Aug 17 12:54:46.126[36m [INFO] [name:forwarder-ovs-jck5d] [0m(21) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
Aug 17 12:54:46.126[36m [INFO] [name:forwarder-ovs-jck5d] [0m(22) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.127[36m [INFO] [name:forwarder-ovs-jck5d] [0m(23) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
Aug 17 12:54:46.130[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(23.1) context canceled
Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:60
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress.(*ipaddressServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/server.go:64
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes.(*routesServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors.(*ipNeighborsServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/server.go:37
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:96
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:71
github.com/networkservicemesh/sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu.(*mtuServer).Request
/build/local/nsm-sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/server.go:63
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/server.go:56
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextServer).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/server.go:69 span=4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:40c0eac4ee59079b:6cbfe09b01f3e371:1
Aug 17 12:54:46.130[36m [INFO] [name:forwarder-ovs-jck5d] [0m(22.1) response=null span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.131[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(22.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6cbfe09b01f3e371:2634afd39f918e0a:1
Aug 17 12:54:46.131[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(21.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2634afd39f918e0a:0e1ad8e73b5b3040:1
Aug 17 12:54:46.132[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(20.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:0e1ad8e73b5b3040:1e238d9ec9e46614:1
Aug 17 12:54:46.133[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(19.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:1e238d9ec9e46614:3a39ddd59c3543a1:1
Aug 17 12:54:46.134[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(18.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:3a39ddd59c3543a1:6cb11cfe293b337a:1
Aug 17 12:54:46.135[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(17.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6cb11cfe293b337a:18731484d8cb069a:1
Aug 17 12:54:46.135[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(16.6) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:18731484d8cb069a:2d3015ecf4a6f046:1
Aug 17 12:54:46.136[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(15.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2d3015ecf4a6f046:65033774651b755c:1
Aug 17 12:54:46.137[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(14.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled span=4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:65033774651b755c:2d18dd97d11ad99e:1
Aug 17 12:54:46.137[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(13.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:2d18dd97d11ad99e:5c4cf9be59362cd2:1
Aug 17 12:54:46.138[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(12.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5c4cf9be59362cd2:514ef3a009bd68a0:1
Aug 17 12:54:46.138[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(11.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:514ef3a009bd68a0:5478a5aa3b554c33:1
Aug 17 12:54:46.139[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(10.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5478a5aa3b554c33:5d0c719d14f3e756:1
Aug 17 12:54:46.139[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(9.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5d0c719d14f3e756:6dc82ce73e7c9687:1
Aug 17 12:54:46.139[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(8.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6dc82ce73e7c9687:0b9d560c035ab0ba:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(7.2) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:0b9d560c035ab0ba:73299f1cb4d4cf6e:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(6.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:73299f1cb4d4cf6e:6b2367eafb964e45:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(5.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:6b2367eafb964e45:3e37c3a7f3af12e5:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(4.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:3e37c3a7f3af12e5:5c1c10f40b476b08:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(3.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:5c1c10f40b476b08:509094a0ffc76466:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(2.1) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:509094a0ffc76466:30badedf05149b36:1
Aug 17 12:54:46.140[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(1.3) Error returned from sdk/pkg/networkservice/common/connect/connectServer.Request: context canceled: cannot support any of the requested mechanism span=4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:30badedf05149b36:12746015f0ef5b22:1
2021/08/17 12:54:46 Reporting span 4db0233badb1acd3:12746015f0ef5b22:324032673fe775c5:1
2021/08/17 12:54:46 Reporting span 7adb6f3bf789e1b8:4814765c13ec3c96:2a5db85f96e876c5:1
Aug 17 12:54:47.810[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathServer.Request() span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.810[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.1) request={"connection":{"id":"6bc38cb1-39ef-4cfb-93ca-1e840236ad90","network_service":"icmp-responder","context":{"ip_context":{"src_ip_addrs":["172.16.1.101/32"],"dst_ip_addrs":["172.16.1.100/32"],"src_routes":[{"prefix":"172.16.1.100/32"}],"dst_routes":[{"prefix":"172.16.1.101/32"}]},"dns_context":{}},"labels":{"nodeName":"dl380-004"},"path":{"index":4,"path_segments":[{"name":"nsc-kernel-7779d5fd5c-ngp8f","id":"nsc-kernel-7779d5fd5c-ngp8f-0","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODEsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zLWM1bG1xL3NhL2RlZmF1bHQifQ.jaXcx2zqyY8aXlP5Nz2ENu-UVmd3THTZsR7QPwXOwrwOwI0l4jcxM99ObcVQT5AmpCDyFW5h-ZXbLz-FRYmy_w","expires":{"seconds":1629205481,"nanos":737548685}},{"name":"nsmgr-lqdlf","id":"adab98c8-1029-4019-8ed0-6a9693ea440a","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.2uj7IrFsA2UgOHfEf0jLwIT27dadMx-Drj7WoIiXJ3yraFc6OhqfOc9NredYIpeW5Staxx4FJ5WtJW4oO07Elw","expires":{"seconds":1629205487,"nanos":403397355}},{"name":"forwarder-ovs-vc7qb","id":"fb061d27-f765-45e1-acc4-da8334770b57","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDczNDcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.VLYeMnlma2vMDSNYDclUOpN9YKsNZtTA8VpHJkbG3cD4DBiUMdPS5yi7hnj8VRsnNJ6hTqBYkCfVs9HjnJW0aw","expires":{"seconds":1629207347}},{"name":"nsmgr-lqdlf","id":"58856609-3808-4045-a5c7-ec4be63fe53e","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.rnxn8bx1zTBEAKNHixiFlPvGOlKcIICj1gQWhMEzqJ7pzkP002psWXOHk6OnuUHreFokYlk5NnvfUv4XZtgOZg","expires":{"seconds":1629205487,"nanos":707657276}},{"name":"nsmgr-8hs7f","id":"6bc38cb1-39ef-4cfb-93ca-1e840236ad90","token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.W1DcrnJ6gHpD60SAydPBlpNEDpU1pN-vk4myhgHB7HVrChXhYBTdW61cbRaT8SHZ4q7y9BE5Bqj9u-nJ71Q6Vg","expires":{"seconds":1629205487,"nanos":803101542}}]},"network_service_endpoint_name":"149a1d2f-a622-4a11-bf2b-7f156fc64763-nse-kernel-6444cdcc55-jjjfv","payload":"ETHERNET"},"mechanism_preferences":[{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}]} span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.810[36m [INFO] [name:forwarder-ovs-jck5d] [0m(1.2) request-diff={"connection":{"id":"4d8cd1ff-e0ca-43a7-8d11-d5c2786b4339","path":{"index":5,"path_segments":{"+5":{"name":"forwarder-ovs-jck5d","id":"4d8cd1ff-e0ca-43a7-8d11-d5c2786b4339"},"4":{"expires":{"nanos":810141575},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDU0ODcsInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.cKDOhHFqUHeHExw7bgCcJXAoVgOv4iIjMTF3o8stEC-vpUks7yrLr0vpfDkbjb93QVR3Dd0K4tJPHkojr-U4Ig"}}}}} span=33171ea980340c55:3d7f562b904c926f:4082ea13a7356c43:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(2) ⎆ sdk/pkg/networkservice/common/authorize/authorizeServer.Request() span=33171ea980340c55:32e55e6ab5639b47:3d7f562b904c926f:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(3) ⎆ sdk/pkg/networkservice/common/serialize/serializeServer.Request() span=33171ea980340c55:444982e0184ab51c:32e55e6ab5639b47:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(4) ⎆ sdk/pkg/networkservice/common/timeout/timeoutServer.Request() span=33171ea980340c55:17f3af5424e75d46:444982e0184ab51c:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(5) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=33171ea980340c55:0e18be1c08053317:17f3af5424e75d46:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(6) ⎆ sdk/pkg/networkservice/common/monitor/monitorServer.Request() span=33171ea980340c55:3196cd10c66043c0:0e18be1c08053317:1
Aug 17 12:54:47.811[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7) ⎆ sdk/pkg/networkservice/common/updatetoken/updateTokenServer.Request() span=33171ea980340c55:5f159c641dd77dab:3196cd10c66043c0:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(7.1) request-diff={"connection":{"path":{"path_segments":{"5":{"expires":{"seconds":1629207419},"token":"eyJhbGciOiJFUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJzcGlmZmU6Ly9leGFtcGxlLm9yZy9ucy9uc20tc3lzdGVtL3NhL2RlZmF1bHQiLCJleHAiOjE2MjkyMDc0MTksInN1YiI6InNwaWZmZTovL2V4YW1wbGUub3JnL25zL25zbS1zeXN0ZW0vc2EvZGVmYXVsdCJ9.hsPYSiqGiEBiDT-0pITG2S8ISnyDfj8cipLtlcWfw8JaAu2FFJkQTH_qSBanSVXLJGpd-dOd7V341HjYHqUfAw"}}}}} span=33171ea980340c55:5f159c641dd77dab:3196cd10c66043c0:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(8) ⎆ sdk/pkg/networkservice/utils/metadata/metadataServer.Request() span=33171ea980340c55:0fa9d5dee39ba2e4:5f159c641dd77dab:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(9) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDServer.Request() span=33171ea980340c55:0b846146600577ad:0fa9d5dee39ba2e4:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(10) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDServer.Request() span=33171ea980340c55:7d03570f9b5cbeac:0b846146600577ad:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(11) ⎆ sdk/pkg/networkservice/common/clienturl/clientURLServer.Request() span=33171ea980340c55:6a59f03cba01225d:7d03570f9b5cbeac:1
Aug 17 12:54:47.812[36m [INFO] [name:forwarder-ovs-jck5d] [0m(12) ⎆ sdk/pkg/networkservice/common/heal/healServer.Request() span=33171ea980340c55:57d1cd41864aa672:6a59f03cba01225d:1
Aug 17 12:54:47.813[36m [INFO] [name:forwarder-ovs-jck5d] [0m(13) ⎆ sdk/pkg/networkservice/common/mechanisms/mechanismsServer.Request() span=33171ea980340c55:1251af30344580fc:57d1cd41864aa672:1
Aug 17 12:54:47.813[36m [INFO] [name:forwarder-ovs-jck5d] [0m(14) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=33171ea980340c55:3f42381edf9545b1:1251af30344580fc:1
Aug 17 12:54:47.814[36m [INFO] [name:forwarder-ovs-jck5d] [0m(14.1) request-diff={"connection":{"mechanism":{"cls":"REMOTE","parameters":{"+src_ip":"10.10.10.22","+src_port":"4789"},"type":"VXLAN"}}} span=33171ea980340c55:3f42381edf9545b1:1251af30344580fc:1
Aug 17 12:54:47.814[36m [INFO] [name:forwarder-ovs-jck5d] [0m(15) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniServer.Request() span=33171ea980340c55:1b8c28d57904c3ae:3f42381edf9545b1:1
Aug 17 12:54:47.814[36m [INFO] [name:forwarder-ovs-jck5d] [0m(15.1) request-diff={"connection":{"mechanism":{"parameters":{"+dst_ip":"10.10.10.33","+dst_port":"4789","+vni":"4273948"}}}} span=33171ea980340c55:1b8c28d57904c3ae:3f42381edf9545b1:1
Aug 17 12:54:47.814[36m [INFO] [name:forwarder-ovs-jck5d] [0m(16) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanServer.Request() span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.1) connection not established, calling add span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.2) inside add method span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.3) mech is vxlan span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.815[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.4) ovs tunnel name: v10101022 span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [vxlanServer:Request] [0m(16.5) ovs tunnel port number: 1 span=33171ea980340c55:3799f63ec171df92:1b8c28d57904c3ae:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [0m(17) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectServer.Request() span=33171ea980340c55:313a64fee9b5e6dd:3799f63ec171df92:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [0m(18) ⎆ sdk/pkg/networkservice/core/next/nextServer.Request() span=33171ea980340c55:334ae01a91c6bcd3:313a64fee9b5e6dd:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [0m(19) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuServer.Request() span=33171ea980340c55:18348afba4d1f762:334ae01a91c6bcd3:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [0m(20) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsServer.Request() span=33171ea980340c55:59055e4e99dbac64:18348afba4d1f762:1
Aug 17 12:54:47.820[36m [INFO] [name:forwarder-ovs-jck5d] [0m(21) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesServer.Request() span=33171ea980340c55:13ff63e6a1848755:59055e4e99dbac64:1
Aug 17 12:54:47.821[36m [INFO] [name:forwarder-ovs-jck5d] [0m(22) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressServer.Request() span=33171ea980340c55:1341f03843aa6ff3:13ff63e6a1848755:1
Aug 17 12:54:47.821[36m [INFO] [name:forwarder-ovs-jck5d] [0m(23) ⎆ sdk/pkg/networkservice/common/connect/connectServer.Request() span=33171ea980340c55:14a9c468f8fe2450:1341f03843aa6ff3:1
2021/08/17 12:54:47 [DEBUG] passed clientURL: unix:///var/lib/networkservicemesh/nsm.io.sock
Aug 17 12:54:47.824[36m [INFO] [name:forwarder-ovs-jck5d] [0m(24) ⎆ sdk/pkg/networkservice/common/updatepath/updatePathClient.Request() span=33171ea980340c55:44514a22ea533702:14a9c468f8fe2450:1
Aug 17 12:54:47.824[36m [INFO] [name:forwarder-ovs-jck5d] [0m(25) ⎆ sdk/pkg/networkservice/common/serialize/serializeClient.Request() span=33171ea980340c55:2f94fab5a51a35e3:44514a22ea533702:1
Aug 17 12:54:47.824[36m [INFO] [name:forwarder-ovs-jck5d] [0m(26) ⎆ sdk/pkg/networkservice/common/refresh/refreshClient.Request() span=33171ea980340c55:40e4f68002ce365f:2f94fab5a51a35e3:1
Aug 17 12:54:47.824[36m [INFO] [name:forwarder-ovs-jck5d] [0m(27) ⎆ sdk/pkg/networkservice/utils/metadata/metaDataClient.Request() span=33171ea980340c55:025c6799c9f4c167:40e4f68002ce365f:1
Aug 17 12:54:47.824[36m [INFO] [name:forwarder-ovs-jck5d] [0m(28) ⎆ sdk/pkg/networkservice/common/heal/healClient.Request() span=33171ea980340c55:66522b1ac5378601:025c6799c9f4c167:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(29) ⎆ sdk/pkg/networkservice/common/mechanismtranslation/mechanismTranslationClient.Request() span=33171ea980340c55:4031871f635f2e70:66522b1ac5378601:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(29.1) request-diff={"connection":{"mechanism":{"cls":"","parameters":{"-dst_ip":"10.10.10.33","-dst_port":"4789","-src_ip":"10.10.10.22","-src_port":"4789","-vni":"4273948"},"type":""}},"mechanism_preferences":{"-0":{"cls":"REMOTE","type":"VXLAN","parameters":{"src_ip":"10.10.10.22","src_port":"4789"}}}} span=33171ea980340c55:4031871f635f2e70:66522b1ac5378601:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(30) ⎆ sdk-ovs/pkg/networkservice/l2ovsconnect/l2ConnectClient.Request() span=33171ea980340c55:37df39c6dfb0b273:4031871f635f2e70:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(31) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=33171ea980340c55:6a6b938b0aa13cac:37df39c6dfb0b273:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(32) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/mtu/mtuClient.Request() span=33171ea980340c55:18d89b93e3c6e52d:6a6b938b0aa13cac:1
Aug 17 12:54:47.825[36m [INFO] [name:forwarder-ovs-jck5d] [0m(33) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipneighbors/ipNeighborsClient.Request() span=33171ea980340c55:0d89ae6f2c30831e:18d89b93e3c6e52d:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(34) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/routes/routesClient.Request() span=33171ea980340c55:0074594f9572df25:0d89ae6f2c30831e:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(35) ⎆ sdk-kernel/pkg/kernel/networkservice/connectioncontextkernel/ipcontext/ipaddress/ipaddressClient.Request() span=33171ea980340c55:171e7a38568f99a3:0074594f9572df25:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(36) ⎆ sdk-kernel/pkg/kernel/networkservice/inject/injectClient.Request() span=33171ea980340c55:3e31d42098a89dc3:171e7a38568f99a3:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(37) ⎆ sdk-ovs/pkg/networkservice/mechanisms/kernel/kernelClient.Request() span=33171ea980340c55:4576ebaf4460b8ea:3e31d42098a89dc3:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(37.1) request-diff={"mechanism_preferences":{"+0":{"cls":"LOCAL","type":"KERNEL"}}} span=33171ea980340c55:4576ebaf4460b8ea:3e31d42098a89dc3:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(38) ⎆ sdk/pkg/networkservice/core/next/nextClient.Request() span=33171ea980340c55:7fe9db03dd431cf1:4576ebaf4460b8ea:1
Aug 17 12:54:47.826[36m [INFO] [name:forwarder-ovs-jck5d] [0m(39) ⎆ sdk-ovs/pkg/networkservice/mechanisms/vxlan/vxlanClient.Request() span=33171ea980340c55:10ccfd079b8fc35f:7fe9db03dd431cf1:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(39.1) request-diff={"mechanism_preferences":{"+1":{"cls":"REMOTE","type":"VXLAN"}}} span=33171ea980340c55:10ccfd079b8fc35f:7fe9db03dd431cf1:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(40) ⎆ sdk/pkg/networkservice/common/mechanisms/vxlan/vni/vniClient.Request() span=33171ea980340c55:30eb541b70728ea6:10ccfd079b8fc35f:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(40.1) request-diff={"mechanism_preferences":{"1":{"parameters":{"+src_ip":"10.10.10.33","+src_port":"4789"}}}} span=33171ea980340c55:30eb541b70728ea6:10ccfd079b8fc35f:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(41) ⎆ sdk/pkg/networkservice/common/mechanisms/recvfd/recvFDClient.Request() span=33171ea980340c55:4bbde622745cab01:30eb541b70728ea6:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(42) ⎆ sdk/pkg/networkservice/common/mechanisms/sendfd/sendFDClient.Request() span=33171ea980340c55:5622e789510d63da:4bbde622745cab01:1
Aug 17 12:54:47.827[36m [INFO] [name:forwarder-ovs-jck5d] [0m(43) ⎆ sdk/pkg/networkservice/common/null/nullClient.Request() span=33171ea980340c55:2c79c16386cda3ee:5622e789510d63da:1
Aug 17 12:54:47.828[36m [INFO] [name:forwarder-ovs-jck5d] [0m(44) ⎆ api/pkg/api/networkservice/networkServiceClient.Request() span=33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
Aug 17 12:54:48.009[31m [ERRO] [name:forwarder-ovs-jck5d] [0m(44.1) rpc error: code = Unknown desc = Error returned from api/pkg/api/networkservice/networkServiceClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/ipam/point2pointipam/ipamServer.Request: IPPool is empty
Error returned from api/pkg/api/networkservice/networkServiceClient.Request
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:62
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/null.(*nullClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/null/client.go:43
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/sendfd.(*sendFDClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/sendfd/client.go:54
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/recvfd.(*recvFDClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/recvfd/client.go:55
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/common/mechanisms/vxlan/vni.(*vniClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/common/mechanisms/vxlan/vni/client.go:53
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*beginTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:58
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:69
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk/pkg/networkservice/core/trace.(*endTraceClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/trace/client.go:99
github.com/networkservicemesh/sdk/pkg/networkservice/core/next.(*nextClient).Request
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/networkservice/core/next/client.go:71
github.com/networkservicemesh/sdk-ovs/pkg/networkservice/mechanisms/vxlan.(*vxlanClient).Request
/build/local/sdk-ovs/pkg/networkservice/mechanisms/vxlan/client.go:59 span=33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
2021/08/17 12:54:48 Reporting span 33171ea980340c55:037ec6a58d9fb28b:2c79c16386cda3ee:1
Aug 17 12:54:48.009[36m [INFO] [name:forwarder-ovs-jck5d] [0m(43.1) response=null span=33171ea980340c55:2c79c16386cda3ee:5622e789510d63da:1
@Bolodya1997 please ignore the above log, looks that happened because of NSM Mgr pod crash.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 54m Successfully assigned nsm-system/nsmgr-8hs7f to dl380-003
Warning Unhealthy 32m kubelet, dl380-003 Liveness probe failed: OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "argument list too long": unknown
Normal Created 30m (x5 over 54m) kubelet, dl380-003 Created container nsmgr
Normal Pulled 30m (x5 over 54m) kubelet, dl380-003 Container image "ghcr.io/networkservicemesh/ci/cmd-nsmgr:634a640" already present on machine
Normal Started 30m (x5 over 54m) kubelet, dl380-003 Started container nsmgr
Warning Unhealthy 28m (x2 over 35m) kubelet, dl380-003 Readiness probe failed:
Warning Unhealthy 25m (x3 over 34m) kubelet, dl380-003 Liveness probe failed:
Warning Unhealthy 7m15s kubelet, dl380-003 Liveness probe failed: failed to initialize tls credentials with spiffe. error=context deadline exceeded
Warning Unhealthy 7m15s kubelet, dl380-003 Readiness probe errored: rpc error: code = Unknown desc = container not running (eaeb4e6197ef4a15e1deb64074326c4fc9d0ed4fbad1201cacf7d913bb2d22c7)
Warning BackOff 5m27s (x60 over 34m) kubelet, dl380-003 Back-off restarting failed container
would keep you posted if the issue is seen again with latest image..
Hey, Seems I have the same issue. Running EKS 1.21 with Calico and VPP without huge pages.
Running a simple setup of 1 nse - 2 nsc's
Everything seems to be working fine, but I see lots of error messages from all the nsm-components as well as healing not working properly (not sure if its related or another issue).
-
nsmgr has error
Error returned from api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv: rpc error: code = Canceled desc = context canceled -
forwarder-vpp has error
Error returned from api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv: rpc error: code = Canceled desc = context canceled -
simple nse has error
Error returned from api/pkg/api/registry/networkServiceEndpointRegistryFindClient.Recv: rpc error: code = Canceled desc = context canceled -
cmd-nsc injected in clients app sometimes has error
Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = Error returned from sdk/pkg/networkservice/common/connect/connectClient.Request: rpc error: code = Unknown desc = failed to dial unix:///proc/1/fd/21: context deadline exceeded -
in-memory registry sometimes has error
Error returned from sdk/pkg/registry/core/streamcontext/networkServiceEndpointRegistryFindServer.Send: rpc error: code = Unavailable desc = transport is closing
Attached logs per component
forwarder.txt nse.txt nsc.txt nsmgr.txt
EDIT: Seems the errors occur in an interval of 30s-40s. I am guessing this is some kind of refresh mechanism?
EDIT2: I tried to restart nse - it shows a lot of registry errors 1 nsc - has 3 interfaces connected to the nse 2 nsc - has 2 interfaces which are not connected anywhere
Attached logs after nse restart:
nsc2-after-nse-restart.txt nse-after-restart.txt nsc-after-nse-restart.txt
ip addr for nse
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN qlen 500
link/[65534]
inet 10.10.99.25/32 scope global eth0
valid_lft forever preferred_lft forever
3: icmp-respo-2ed1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
link/ether 02:fe:65:c5:a7:b2 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.96/32 scope global icmp-respo-2ed1
valid_lft forever preferred_lft forever
inet 172.16.1.98/32 scope global icmp-respo-2ed1
valid_lft forever preferred_lft forever
inet6 fe80::fe:65ff:fec5:a7b2/64 scope link
valid_lft forever preferred_lft forever
4: icmp-respo-e5a3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
link/ether 02:fe:a9:44:60:53 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.100/32 scope global icmp-respo-e5a3
valid_lft forever preferred_lft forever
inet 172.16.1.98/32 scope global icmp-respo-e5a3
valid_lft forever preferred_lft forever
inet 172.16.1.96/32 scope global icmp-respo-e5a3
valid_lft forever preferred_lft forever
inet6 fe80::fe:a9ff:fe44:6053/64 scope link
valid_lft forever preferred_lft forever
ip addr for nsc 1
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN qlen 500
link/[65534]
inet 10.10.158.37/32 scope global eth0
valid_lft forever preferred_lft forever
34: ovpn-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN qlen 1000
link/ether 02:fe:b3:bf:db:a4 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.101/32 scope global ovpn-1
valid_lft forever preferred_lft forever
inet 172.16.1.99/32 scope global ovpn-1
valid_lft forever preferred_lft forever
inet 172.16.1.97/32 scope global ovpn-1
valid_lft forever preferred_lft forever
inet6 fe80::fe:b3ff:febf:dba4/64 scope link
valid_lft forever preferred_lft forever
ip addr for nsc 2
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 8981 qdisc mq state UNKNOWN group default qlen 500
link/none
inet 10.10.61.79/32 scope global eth0
valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 10000
link/none
inet 10.20.30.1/24 scope global tun0
valid_lft forever preferred_lft forever
26: ovpn-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8951 qdisc mq state UNKNOWN group default qlen 1000
link/ether 02:fe:1c:ac:62:d1 brd ff:ff:ff:ff:ff:ff
inet 172.16.1.97/32 scope global ovpn-1
valid_lft forever preferred_lft forever
inet 172.16.1.99/32 scope global ovpn-1
valid_lft forever preferred_lft forever
inet6 fe80::fe:1cff:feac:62d1/64 scope link
valid_lft forever preferred_lft forever
I was running v1.1.1 for a week and then upgraded to v1.2.0 a couple of days ago
@yuraxdrumz And you are seeing the same issue in v1.2.0 after the upgrade?
@edwarnicke Yep. I just checked and registry, nsmgr, nsc, nsc-init and forwarder are all on latest images. Same issues.
@yuraxdrumz
Canceled desc = context canceled error is expected from registry. In short, each refresh cancels heal and schedules a new one with a new context. I think we can change log level for this message.
Started to look into this.
Q1: Could you point at examples or steps that you're using to reproduce this?
Q2: What do you see in terms of datapath? Are all interfaces correct? Is ping working?
- Steps
- Created an eks cluster with calico and vpp without hugepages and a node group of 4 nodes - https://projectcalico.docs.tigera.io/getting-started/kubernetes/vpp/getting-started
- Added spire server and agent not in crd mode - spire.zip
- Installed the nsm control plane - https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/basic
- Changed the admission webhook to be hostNetwork so master nodes can reach the pod due to calico cni not running on master
- Installed basic example for nse and nsc - https://github.com/networkservicemesh/deployments-k8s/tree/main/examples/use-cases/Kernel2Kernel
- Installed in memory registry instead of k8s registry
- Tried scaling nsc's from 1 to 2
- Tried scaling nse's from 1 to 3
- In terms of data path, I see a very strange behavior:
- If I leave a fresh install with 2 nsc's and 1 nse, everything pings and works as it should for a while
- If I restart any of the nsc's or nce's, I always see multiple interfaces like in the code I pasted a few comments above and ping works for some interfaces and for some it does not
- I left a fresh install running over the course of the weekend and I saw the following logs today, I am guessing the IPPOOL error has something to do with why I see the errors and although the interfaces get cleaned up, it seems the ip addresses are still taken, so healing does not work properly. nse-refresh-failed.txt nsmgr-refresh-failed.txt forwarder-refresh-failed.txt nsc-refresh-failed.txt
After going through https://github.com/networkservicemesh/deployments-k8s/issues/1999, I see my use case of multiple interfaces with same address is caused by multiple nse's that overwrite each other, so that explains one issue.
I think that the problem is not that we get "context cancelled" errors but that the code which tries to ignore them isn't working. For example, in the log messages that I get, one of the lines in the backtrace is:
github.com/networkservicemesh/sdk/pkg/registry/core/trace.(*traceNetworkServiceEndpointRegistryFindClient).Recv;
/go/pkg/mod/github.com/networkservicemesh/[email protected]/pkg/registry/core/trace/nse_registry.go:56;
...which is https://github.com/networkservicemesh/sdk/blob/v1.5.0/pkg/registry/core/trace/nse_registry.go#L56
The message is rpc error: code = Canceled desc = context canceled; so I think that the log message should be prevented by https://github.com/networkservicemesh/sdk/blob/v1.5.0/pkg/registry/core/trace/nse_registry.go#L53 but the equality comparison is failing so the error gets logged.
https://github.com/golang/go/wiki/ErrorValueFAQ#how-should-i-change-my-error-handling-code-to-work-with-the-new-features indicates that we should be using "errors.Is()" for this purpose instead of "==" so this change might reduce the logspam. I'm just getting into NSM so I haven't set up a dev environment yet but I should be able to take a look in the next couple of weeks unless someone else gets to it sooner.
