edgemesh icon indicating copy to clipboard operation
edgemesh copied to clipboard
verified publisher icon kubeedge

edgemesh-agent throw invalid memory address or nil pointer dereference,then exit

Open taoyonggang opened this issue 2 years ago • 4 comments

What happened:

edgemesh-agent start failed when using kubeflow.

this is the logs:

 E0613 10:20:24.945193       1 util.go:116] destination rule object knative-serving/knative .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945236       1 util.go:116] destination rule object kubeflow/metadata-grpc-service .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945262       1 util.go:116] destination rule object kubeflow/ml-pipeline .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945283       1 util.go:116] destination rule object kubeflow/ml-pipeline-minio .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945305       1 util.go:116] destination rule object kubeflow/ml-pipeline-mysql .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945326       1 util.go:116] destination rule object kubeflow/ml-pipeline-ui .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945355       1 util.go:116] destination rule object kubeflow/ml-pipeline-visualizationserver .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945375       1 util.go:116] destination rule object kubeflow-user-example-com/ml-pipeline-visualizationserver .Spec.TrafficPolicy.LoadBalancer is nil

 E0613 10:20:24.945412       1 loadbalancer.go:578] unsupported loadBalance policy 

 E0613 10:20:24.986106       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)

 goroutine 365 [running]:

 k8s.io/apimachinery/pkg/util/runtime.logPanic({0x20733e0, 0x3d55320})

 	/code/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d

 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0016ee8f0})

 	/code/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75

 panic({0x20733e0, 0x3d55320})

 	/usr/local/go/src/runtime/panic.go:1038 +0x215

 github.com/kubeedge/edgemesh/pkg/loadbalancer.(*LoadBalancer).OnDestinationRuleUpdate(0xc000192600, 0xc001606c70, 0xc00000b0e0)

 	/code/pkg/loadbalancer/loadbalancer.go:544 +0x497

 github.com/kubeedge/edgemesh/pkg/loadbalancer.(*LoadBalancer).handleUpdateDestinationRule(0xc00137f380, {0x237fce0, 0xc0018fb888}, {0x237fce0, 0xc00000b0e0})

 	/code/pkg/loadbalancer/loadbalancer.go:267 +0x57

 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)

 	/code/vendor/k8s.io/client-go/tools/cache/controller.go:238

 k8s.io/client-go/tools/cache.(*processorListener).run.func1()

 	/code/vendor/k8s.io/client-go/tools/cache/shared_informer.go:785 +0x127

 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f8e39bc19f0)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67

 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001228738, {0x292d5c0, 0xc0013792c0}, 0x1, 0xc001237da0)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6

 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0xe, 0xc001228788)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89

 k8s.io/apimachinery/pkg/util/wait.Until(...)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90

 k8s.io/client-go/tools/cache.(*processorListener).run(0xc001328f80)

 	/code/vendor/k8s.io/client-go/tools/cache/shared_informer.go:781 +0x6b

 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a

 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88

 panic: runtime error: invalid memory address or nil pointer dereference [recovered]

 	panic: runtime error: invalid memory address or nil pointer dereference

 [signal SIGSEGV: segmentation violation code=0x1 addr=0x38 pc=0x1d78e97]

 

 goroutine 365 [running]:

 k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0016ee8f0})

 	/code/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8

 panic({0x20733e0, 0x3d55320})

 	/usr/local/go/src/runtime/panic.go:1038 +0x215

 github.com/kubeedge/edgemesh/pkg/loadbalancer.(*LoadBalancer).OnDestinationRuleUpdate(0xc000192600, 0xc001606c70, 0xc00000b0e0)

 	/code/pkg/loadbalancer/loadbalancer.go:544 +0x497

 github.com/kubeedge/edgemesh/pkg/loadbalancer.(*LoadBalancer).handleUpdateDestinationRule(0xc00137f380, {0x237fce0, 0xc0018fb888}, {0x237fce0, 0xc00000b0e0})

 	/code/pkg/loadbalancer/loadbalancer.go:267 +0x57

 k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate(...)

 	/code/vendor/k8s.io/client-go/tools/cache/controller.go:238

 k8s.io/client-go/tools/cache.(*processorListener).run.func1()

 	/code/vendor/k8s.io/client-go/tools/cache/shared_informer.go:785 +0x127

 k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7f8e39bc19f0)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67

 k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001228738, {0x292d5c0, 0xc0013792c0}, 0x1, 0xc001237da0)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6

 k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0, 0x3b9aca00, 0x0, 0xe, 0xc001228788)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89

 k8s.io/apimachinery/pkg/util/wait.Until(...)

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90

 k8s.io/client-go/tools/cache.(*processorListener).run(0xc001328f80)

 	/code/vendor/k8s.io/client-go/tools/cache/shared_informer.go:781 +0x6b

 k8s.io/apimachinery/pkg/util/wait.(*Group).Start.func1()

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:73 +0x5a

 created by k8s.io/apimachinery/pkg/util/wait.(*Group).Start

 	/code/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:71 +0x88

What you expected to happen: it is running .

How to reproduce it (as minimally and precisely as possible): ..... Anything else we need to know?: ...... Environment:

  • EdgeMesh version: 1.9.0 - 1.13.0

  • Kubernetes version (use kubectl version): 1.21

  • KubeEdge version(e.g. cloudcore --version and edgecore --version): 1.9.2

  • Cloud nodes Environment:
    • Hardware configuration (e.g. lscpu):
    • OS (e.g. cat /etc/os-release):
    • Kernel (e.g. uname -a):
    • Go version (e.g. go version):
    • Others:
  • Edge nodes Environment:
    • edgecore version (e.g. edgecore --version): 1.9.2
    • Hardware configuration (e.g. lscpu):
    • OS (e.g. cat /etc/os-release):
    • Kernel (e.g. uname -a):
    • Go version (e.g. go version):
    • Others:

taoyonggang avatar Jun 13 '23 02:06 taoyonggang

Can I see your destinationrule file?

Poorunga avatar Jun 13 '23 03:06 Poorunga

how get it ?

taoyonggang avatar Jun 13 '23 04:06 taoyonggang

cloud agents is good. edge agents is bad.

taoyonggang avatar Jun 13 '23 04:06 taoyonggang

You can get it by kubectl get destinationrule -o yaml

Poorunga avatar Jun 26 '23 03:06 Poorunga