swift
swift copied to clipboard
Published
20 hours ago •
kubepack
kubepack
A question about 'Out of Memory'
Hello . you guys:
I have deploy a tiller and swift into my k8s cluster. It seem everything is OK.
But the swift pod will been killed about every 2 hours.
The kernel log located in path "/var/log" :
Apr 12 08:43:14 VM_133_12_centos kernel: Out of memory: Kill process 6593 (swift) score 1470 or sacrifice child Apr 12 08:43:14 VM_133_12_centos kernel: Out of memory: Kill process 6611 (swift) score 1470 or sacrifice child Apr 12 10:10:49 VM_133_12_centos kernel: Out of memory: Kill process 20516 (swift) score 1455 or sacrifice child Apr 12 10:56:59 VM_133_12_centos kernel: Out of memory: Kill process 11716 (swift) score 1457 or sacrifice child Apr 12 10:56:59 VM_133_12_centos kernel: Out of memory: Kill process 11737 (swift) score 1457 or sacrifice child Apr 12 11:43:03 VM_133_12_centos kernel: Out of memory: Kill process 27510 (swift) score 1457 or sacrifice child Apr 12 13:14:57 VM_133_12_centos kernel: Out of memory: Kill process 11044 (swift) score 1457 or sacrifice child Apr 12 13:14:57 VM_133_12_centos kernel: Out of memory: Kill process 11075 (swift) score 1458 or sacrifice child
Tiller version:
Client: &version.Version{SemVer:"v2.12.0", GitCommit:"d325d2a9c179b33af1a024cdb5a4472b6288016a", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.13.0", GitCommit:"79d07943b03aea2b76c12644b4b54733bc5958d6", GitTreeState:"clean"}
Swift version
appscode/swift:0.11.0
Kubernetes version
Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-tke.3", GitCommit:"53e244be925234190938376fe8637189b6caf125", GitTreeState:"clean", BuildDate:"2018-12-04T04:10:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.5-tke.3", GitCommit:"53e244be925234190938376fe8637189b6caf125", GitTreeState:"clean", BuildDate:"2018-12-04T04:10:15Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
I will post the more clues as soon as I find them.
Thanks.
The memory utilization metric of swift pod: `sum(container_memory_usage_bytes{namespace="kube-system", pod_name="swift-swift-c5d5c7bf7-65k9v", container_name!="POD", container_name!=""}) by (container_name)
{container_name="swift"} | 6474194944 it is about 6.4G. `
Out of memory: Kill process 11075 (swift) score 1458 or sacrifice child messages-Apr 12 13:14:57 VM_133_12_centos kernel: Killed process 11075 (swift) total-vm:10307388kB, anon-rss:3658748kB, file-rss:4kB, shmem-rss:0kB
At every time the swift has been killed, almost it's memory utilization is about 10307388kB.
can you run the command
netstat | wc -l? Should I execute this command into the container? Or just on the host?
can you run the command
netstat | wc -l?
For host:
[root@VM_133_12_centos log]# netstat | wc -l 447
For container:
[root@VM_133_12_centos log]# ks exec -it swift-swift-c5d5c7bf7-65k9v sh ~ $ netstat Active Internet connections (w/o servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 swift-swift-c5d5c7bf7-65k9v:33574 kubernetes.default.svc.cluster.local:https ESTABLISHED tcp 0 9 localhost:57582 localhost:9855 ESTABLISHED tcp 0 9 localhost:9855 localhost:57582 ESTABLISHED Active UNIX domain sockets (w/o servers) Proto RefCnt Flags Type State I-Node Path
the output of pstack:
[root@VM_133_11_centos ~]# pstack 5629 Thread 1 (process 5629): #0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:532 #1 0x0000000000429ddb in runtime.futexsleep (addr=0x207d6a0 <runtime.m0+320>, ns=-1, val=0) at /usr/local/go/src/runtime/os_linux.go:46 #2 0x000000000040a962 in runtime.notesleep (n=0x207d6a0 <runtime.m0+320>) at /usr/local/go/src/runtime/lock_futex.go:151 #3 0x00000000004328ea in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2165 #4 0x0000000000434019 in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2565 #5 0x000000000043419e in runtime.park_m (gp=0xc000255500) at /usr/local/go/src/runtime/proc.go:2676 #6 0x0000000000459f3b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299 #7 0x0000000000459e59 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201 #8 0x0000000000000000 in ?? ()
the output of top `PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5629 nfsnobo+ 20 0 2869436 1.7g 9888 S 106.4 22.3 35:57.19 swift `
The progress: The OOM problem has disappeared after I use version 0.10 instead of 0.11.
So I think this problem arises from upgrading the version of swift?
$ pstack 11008
Thread 17 (Thread 0x7febd9ffb700 (LWP 11874)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf82 in runtime.futexsleep (addr=0x22488a0 <runtime.timers+416>, val=0, ns=4999996575) at /usr/local/go/src/runtime/os_linux.go:63
#2 0x000000000040c562 in runtime.notetsleep_internal (n=0x22488a0 <runtime.timers+416>, ns=4999996575, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3 0x000000000040c6bc in runtime.notetsleepg (n=0x22488a0 <runtime.timers+416>, ns=4999996575, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044d86a in runtime.timerproc (tb=0x2248880 <runtime.timers+384>) at /usr/local/go/src/runtime/time.go:311
#5 0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6 0x0000000002248880 in runtime.timers ()
#7 0x0000000000000000 in ?? ()
Thread 16 (Thread 0x7febda7fc700 (LWP 11031)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0000a39c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0000a39c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc00005b400, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc00056cc00) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c0000a5800 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febda7fc700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 15 (Thread 0x7febdaffd700 (LWP 11026)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0000a2f48, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0000a2f48) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc00056cf00) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c0000a5380 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdaffd700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 14 (Thread 0x7febdb7fe700 (LWP 11025)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0003c84c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0003c84c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc00005d900, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc000245e00) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c000244180 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdb7fe700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 13 (Thread 0x7febdbfff700 (LWP 11020)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0003c92c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0003c92c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000056a00, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc00056cc00) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c0003c0900 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febdbfff700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 12 (Thread 0x7febfcde7700 (LWP 11019)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0002f24c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0002f24c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc00056c780) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c0000fc600 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfcde7700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 11 (Thread 0x7febfd5e8700 (LWP 11018)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0001d5d48, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0001d5d48) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000054500, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc000245800) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c000245680 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfd5e8700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 10 (Thread 0x7febfdde9700 (LWP 11017)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc0001d59c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc0001d59c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000058f00, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc000245b00) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c000245200 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febfdde9700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 9 (Thread 0x7febfe5ea700 (LWP 11016)):
#0 runtime.epollwait () at /usr/local/go/src/runtime/sys_linux_amd64.s:675
#1 0x000000000042cdc6 in runtime.netpoll (block=true, ~r1=...) at /usr/local/go/src/runtime/netpoll_epoll.go:71
#2 0x0000000000435ca5 in runtime.findrunnable (gp=0xc000052000, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2380
#3 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#4 0x0000000000436ba1 in runtime.park_m (gp=0xc00056cf00) at /usr/local/go/src/runtime/proc.go:2605
#5 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#6 0x000000c0000fc300 in ?? ()
#7 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#8 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#9 0x00007fec10dcd5aa in start_thread (arg=0x7febfe5ea700) at pthread_create.c:463
#10 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 8 (Thread 0x7febfedeb700 (LWP 11015)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf82 in runtime.futexsleep (addr=0x22487a0 <runtime.timers+160>, val=0, ns=3215267039472) at /usr/local/go/src/runtime/os_linux.go:63
#2 0x000000000040c562 in runtime.notetsleep_internal (n=0x22487a0 <runtime.timers+160>, ns=3215267039472, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3 0x000000000040c6bc in runtime.notetsleepg (n=0x22487a0 <runtime.timers+160>, ns=3215267039472, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044d86a in runtime.timerproc (tb=0x2248780 <runtime.timers+128>) at /usr/local/go/src/runtime/time.go:311
#5 0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6 0x0000000002248780 in runtime.timers ()
#7 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7febff5ec700 (LWP 11014)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0xc00003f2c8, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0xc00003f2c8) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434bc1 in runtime.stopm () at /usr/local/go/src/runtime/proc.go:1936
#4 0x0000000000435cd0 in runtime.findrunnable (gp=0xc000062500, inheritTime=false) at /usr/local/go/src/runtime/proc.go:2399
#5 0x000000000043688e in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2525
#6 0x0000000000436ba1 in runtime.park_m (gp=0xc0004bc180) at /usr/local/go/src/runtime/proc.go:2605
#7 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#8 0x000000c000001500 in ?? ()
#9 0x0000000000800000 in github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.(*Requirement).DeepCopyInto (in=<optimized out>, out=<optimized out>) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/zz_generated.deepcopy.go:29
#10 github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels.internalSelector.DeepCopy (s=<error reading variable: access outside bounds of object referenced via synthetic pointer>, ~r0=...) at /home/xk/go_pro/src/github.com/appscode/swift/vendor/k8s.io/apimachinery/pkg/labels/selector.go:91
#11 0x00007fec10dcd5aa in start_thread (arg=0x7febff5ec700) at pthread_create.c:463
#12 0x00007fec10b02cbf in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
Thread 6 (Thread 0x7febfffff700 (LWP 11013)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0x22675d8 <runtime.newmHandoff+24>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0x22675d8 <runtime.newmHandoff+24>) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x0000000000434ae3 in runtime.templateThread () at /usr/local/go/src/runtime/proc.go:1914
#4 0x00000000004337d4 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1206
#5 0x00000000004336ef in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1172
#6 0x0000000000401a83 in runtime/cgo(.text) ()
#7 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fec0cff7700 (LWP 11012)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf82 in runtime.futexsleep (addr=0x2248820 <runtime.timers+288>, val=0, ns=29999939229) at /usr/local/go/src/runtime/os_linux.go:63
#2 0x000000000040c562 in runtime.notetsleep_internal (n=0x2248820 <runtime.timers+288>, ns=29999939229, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3 0x000000000040c6bc in runtime.notetsleepg (n=0x2248820 <runtime.timers+288>, ns=29999939229, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044d86a in runtime.timerproc (tb=0x2248800 <runtime.timers+256>) at /usr/local/go/src/runtime/time.go:311
#5 0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6 0x0000000002248800 in runtime.timers ()
#7 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fec0d7f8700 (LWP 11011)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0x2267740 <runtime.sig>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c4b8 in runtime.notetsleep_internal (n=0x2267740 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:174
#3 0x000000000040c6bc in runtime.notetsleepg (n=0x2267740 <runtime.sig>, ns=-1, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044529c in os/signal.signal_recv (~r0=<optimized out>) at /usr/local/go/src/runtime/sigqueue.go:139
#5 0x00000000005b57b2 in os/signal.loop () at /usr/local/go/src/os/signal/signal_unix.go:23
#6 0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#7 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fec0dff9700 (LWP 11010)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf82 in runtime.futexsleep (addr=0x2248720 <runtime.timers+32>, val=0, ns=14867423858) at /usr/local/go/src/runtime/os_linux.go:63
#2 0x000000000040c562 in runtime.notetsleep_internal (n=0x2248720 <runtime.timers+32>, ns=14867423858, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:193
#3 0x000000000040c6bc in runtime.notetsleepg (n=0x2248720 <runtime.timers+32>, ns=14867423858, ~r2=<optimized out>) at /usr/local/go/src/runtime/lock_futex.go:228
#4 0x000000000044d86a in runtime.timerproc (tb=0x2248700 <runtime.timers>) at /usr/local/go/src/runtime/time.go:311
#5 0x000000000045e011 in runtime.goexit () at /usr/local/go/src/runtime/asm_amd64.s:1337
#6 0x0000000002248700 in ?? ()
#7 0x0000000000000000 in ?? ()
Thread 2 (Thread 0x7fec0e7fa700 (LWP 11009)):
#0 runtime.usleep () at /usr/local/go/src/runtime/sys_linux_amd64.s:131
#1 0x000000000043af9b in runtime.sysmon () at /usr/local/go/src/runtime/proc.go:4282
#2 0x00000000004337d4 in runtime.mstart1 () at /usr/local/go/src/runtime/proc.go:1206
#3 0x00000000004336ef in runtime.mstart () at /usr/local/go/src/runtime/proc.go:1172
#4 0x0000000000401a83 in runtime/cgo(.text) ()
#5 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fec111e1740 (LWP 11008)):
#0 runtime.futex () at /usr/local/go/src/runtime/sys_linux_amd64.s:536
#1 0x000000000042cf0b in runtime.futexsleep (addr=0x2244b08 <runtime.m0+328>, val=0, ns=-1) at /usr/local/go/src/runtime/os_linux.go:46
#2 0x000000000040c3d1 in runtime.notesleep (n=0x2244b08 <runtime.m0+328>) at /usr/local/go/src/runtime/lock_futex.go:151
#3 0x000000000043521c in runtime.stoplockedm () at /usr/local/go/src/runtime/proc.go:2076
#4 0x0000000000436a2c in runtime.schedule () at /usr/local/go/src/runtime/proc.go:2477
#5 0x0000000000436ba1 in runtime.park_m (gp=0xc0003c0480) at /usr/local/go/src/runtime/proc.go:2605
#6 0x000000000045bf2b in runtime.mcall () at /usr/local/go/src/runtime/asm_amd64.s:299
#7 0x000000000045be49 in runtime.rt0_go () at /usr/local/go/src/runtime/asm_amd64.s:201
#8 0x0000000000000000 in ?? ()
I had tried to use 0.11.1 version to debug on my local PC. and it looks work very well.
These are about heap-top output :
flat flat% sum% cum cum%
513kB 50.04% 50.04% 513kB 50.04%
github.com/appscode/swift/vendor/golang.org/x/net/http2/hpack.newInternalNode
512.25kB 49.96% 100% 512.25kB 49.96%
github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseProperties
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/pkg/apis/swift/v2.init.2
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parse
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseReference
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*Schema).parseSchema
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.(*SchemaLoader).Compile
0 0% 100% 512.25kB 49.96% github.com/appscode/swift/vendor/github.com/xeipuuv/gojsonschema.NewSchema
0 0% 100% 513kB 50.04% github.com/appscode/swift/vendor/golang.org/x/net/http2/hpack.init.ializers
0 0% 100% 1025.25kB 100% runtime.main
If I switch to 0.11.0 again, the output display that there is a high utilization:
Showing top 10 nodes out of 25
flat flat% sum% cum cum%
37376kB 97.31% 97.31% 37376kB 97.31% bytes.makeSlice
520.04kB 1.35% 98.67% 520.04kB 1.35% net/http.(*http2ClientConn).frameScratchBuffer
512.06kB 1.33% 100% 512.06kB 1.33% net/url.parse
0 0% 100% 37376kB 97.31% bytes.(*Buffer).Write
0 0% 100% 37376kB 97.31% bytes.(*Buffer).grow
0 0% 100% 512.06kB 1.33% github.com/appscode/swift/pkg/apis/swift/v2.init.2
0 0% 100% 37376kB 97.31% github.com/appscode/swift/vendor/github.com/soheilhy/cmux.(*bufferedReader).Read
0 0% 100% 37376kB 97.31% github.com/appscode/swift/vendor/github.com/soheilhy/cmux.(*cMux).serve
0 0% 100% 37376kB 97.31% github.com/appscode/swift/vendor/github.com/soheilhy/cmux.HTTP2MatchHeaderFieldSendSettings.func1
0 0% 100% 37376kB 97.31% github.com/appscode/swift/vendor/github.com/soheilhy/cmux.matchHTTP2Field
So I think the issue has been fixed for a while, and I will observe for a while.