gohai
gohai copied to clipboard
DataDog
High CPU usage on MacOS Sierra
Hi, gohai processes use ~65% of CPU all the time.
Computer: MacBook Pro Retina Processor: 2.2 GHz Intel Core i7 RAM: 16Gb 1600MHz Graphics: Intel Iris pro 1536Mb
Hi Got the same problem exactly on the same macbook laptop. For now i just quit the datadog agent and forced quit those "gohai" processes.
I am also having this issue with similar hardware. Datadog Agent spins up 2+ gohai processes and they spin forever at 65%+ CPU utilization.
Example sample dump:
Sampling process 65830 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling gohai (pid 65830) every 1 millisecond
Process: gohai [65830]
Path: /opt/datadog-agent/bin/gohai
Load Address: 0x4000000
Identifier: gohai
Version: 0
Code Type: X86-64
Parent Process: ??? [1]
Date/Time: 2017-01-25 16:03:02.005 -0500
Launch Time: 2017-01-25 15:17:51.190 -0500
OS Version: Mac OS X 10.12.2 (16C67)
Report Version: 7
Analysis Tool: /usr/bin/sample
----
Call graph:
2787 Thread_4973621 DispatchQueue_1: com.apple.main-thread (serial)
+ 2787 runtime.semasleep1 (in gohai) + 229 [0x402bbe5] os1_darwin.go:385
+ 2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
2787 Thread_4973622
+ 2777 runtime.usleep (in gohai) + 44 [0x405f57c] sys_darwin_amd64.s:285
+ 6 runtime.sysmon (in gohai) + 285,258 [0x403986d,0x4039852] proc1.go:2995
+ 2 ??? (in <unknown binary>) [0x7469622d343620]
+ ! 2 nanotime (in gohai) + 18 [0x405f332] sys_darwin_amd64.s:126
+ 1 runtime.retake (in gohai) + 623 [0x4039edf] proc1.go:3118
+ 1 runtime.sysmon (in gohai) + 665 [0x40399e9] proc1.go:3038
+ 1 runtime.atomicload64 (in gohai) + 13 [0x4006ded] atomic_amd64x.go:30
2787 Thread_4973623
+ 2787 runtime.semasleep1 (in gohai) + 129 [0x402bb81] os1_darwin.go:374
+ 2787 runtime.mach_semaphore_timedwait (in gohai) + 19 [0x405f6c3] sys_darwin_amd64.s:417
2787 Thread_4973624
+ 2787 ??? (in gohai) load address 0x4000000 + 0xfffffffffc001103 [0x1103]
+ 2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
2787 Thread_4974270
+ 2787 ??? (in gohai) load address 0x4000000 + 0xfffffffffc002203 [0x2203]
+ 2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
2787 Thread_4974271
+ 2787 runtime.sigtrampgo (in gohai) + 134 [0x4041f36] signal_darwin.go:67
2787 Thread_4974272
+ 2787 ??? (in gohai) load address 0x4000000 + 0xfffffffffc002503 [0x2503]
+ 2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
2787 Thread_4974275
+ 2787 ??? (in gohai) load address 0x4000000 + 0xfffffffffc002803 [0x2803]
+ 2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
2787 Thread_4974276
2787 ??? (in gohai) load address 0x4000000 + 0xfffffffffc002b03 [0x2b03]
2787 runtime.mach_semaphore_wait (in gohai) + 11 [0x405f6ab] sys_darwin_amd64.s:407
Total number in stack (recursive counted multiple, when >=5):
6 runtime.mach_semaphore_wait (in gohai) + 0 [0x405f6a0] sys_darwin_amd64.s:407
Sort by top of stack, same collapsed (when >= 5):
runtime.mach_semaphore_wait (in gohai) 16722
runtime.mach_semaphore_timedwait (in gohai) 2787
runtime.sigtrampgo (in gohai) 2787
runtime.usleep (in gohai) 2777
runtime.sysmon (in gohai) 6
Binary Images:
0x4000000 - 0x4560fff +gohai (0) <43891F83-234C-3265-800C-FEFC0D4E6E23> /opt/datadog-agent/bin/gohai
0x955e000 - 0x959b267 dyld (0.0 - ???) <947FC440-80F9-32F7-A773-6FC418FE1AB7> /usr/lib/dyld
0x7fffa3029000 - 0x7fffa34c3fff com.apple.CoreFoundation (6.9 - 1348.28) <A40AA224-7A50-3989-95D0-5A228A0E2FAF> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
0x7fffa500b000 - 0x7fffa50a0ff7 com.apple.framework.IOKit (2.0.2 - 1324.30.13) <163BE7FA-B29A-348F-8B5F-E301F2E8C964> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
0x7fffa952f000 - 0x7fffa984efff com.apple.security (7.0 - 57740.31.2) <A47D7BAE-0591-3184-8E44-FB2EB08A19C2> /System/Library/Frameworks/Security.framework/Versions/A/Security
0x7fffb6e27000 - 0x7fffb6e28ff3 libDiagnosticMessagesClient.dylib (102) <422911A4-E273-3E88-BFC4-DF6470E48242> /usr/lib/libDiagnosticMessagesClient.dylib
0x7fffb7060000 - 0x7fffb7060fff libOpenScriptingUtil.dylib (172) <D025E180-BB3B-3FFA-98FC-B6835354D723> /usr/lib/libOpenScriptingUtil.dylib
0x7fffb7066000 - 0x7fffb7067ff3 libSystem.B.dylib (1238) <FF55E335-37C3-3567-9415-A3A1473775F5> /usr/lib/libSystem.B.dylib
0x7fffb717f000 - 0x7fffb717fff3 libauto.dylib (187) <5BBF6A00-CC76-389D-84E7-CA88EDADE683> /usr/lib/libauto.dylib
0x7fffb7180000 - 0x7fffb7190ff3 libbsm.0.dylib (34) <20084796-B04D-3B35-A003-EA11459557A9> /usr/lib/libbsm.0.dylib
0x7fffb7191000 - 0x7fffb719fff7 libbz2.1.0.dylib (38) <6FD3B63F-0F86-3A25-BD5B-E243F58792C9> /usr/lib/libbz2.1.0.dylib
0x7fffb71a0000 - 0x7fffb71f6ff7 libc++.1.dylib (307.4) <BEE86868-F831-384C-919E-2B286ACFE87C> /usr/lib/libc++.1.dylib
0x7fffb71f7000 - 0x7fffb7221fff libc++abi.dylib (307.2) <1CEF8ABB-7E6D-3C2F-8E0A-E7884478DD23> /usr/lib/libc++abi.dylib
0x7fffb7249000 - 0x7fffb7249ff7 libcoretls.dylib (121.31.1) <BCC32537-4831-3E9F-876E-8C9F4CF52FD3> /usr/lib/libcoretls.dylib
0x7fffb724a000 - 0x7fffb724bff3 libcoretls_cfhelpers.dylib (121.31.1) <6F37C5AD-7999-3D31-A52F-7AEED935F32D> /usr/lib/libcoretls_cfhelpers.dylib
0x7fffb7656000 - 0x7fffb7656fff libenergytrace.dylib (15) <A1B040A2-7977-3097-9ADF-34FF181EB970> /usr/lib/libenergytrace.dylib
0x7fffb775f000 - 0x7fffb7984ffb libicucore.A.dylib (57149.0.1) <6B5FDA93-AA88-318F-9608-C2A33D602EC7> /usr/lib/libicucore.A.dylib
0x7fffb798c000 - 0x7fffb79a5ffb liblzma.5.dylib (10) <44BD0279-99DD-36B5-8A6E-C11432E2098D> /usr/lib/liblzma.5.dylib
0x7fffb7d11000 - 0x7fffb80e1d97 libobjc.A.dylib (706) <F9AFE665-A3A2-3285-9495-19803A565861> /usr/lib/libobjc.A.dylib
0x7fffb80e4000 - 0x7fffb80e8fff libpam.2.dylib (21.30.1) <71EB0D88-DE84-3C8D-A2C5-58AA282BC5BC> /usr/lib/libpam.2.dylib
0x7fffb81a3000 - 0x7fffb82ebfe3 libsqlite3.dylib (253) <B5BA5C96-AB13-34A0-8237-DD52A0181DFE> /usr/lib/libsqlite3.dylib
0x7fffb83e0000 - 0x7fffb83edfff libxar.1.dylib (357) <58BFB84B-66FE-3299-AA3D-BBA178ADEE39> /usr/lib/libxar.1.dylib
0x7fffb83f1000 - 0x7fffb84e0ffb libxml2.2.dylib (30.11) <E12AF929-0FA5-3214-840F-C81E6AC9F36E> /usr/lib/libxml2.2.dylib
0x7fffb850b000 - 0x7fffb851cff3 libz.1.dylib (67) <46E3FFA2-4328-327A-8D34-A03E20BFFB8E> /usr/lib/libz.1.dylib
0x7fffb852b000 - 0x7fffb852fff7 libcache.dylib (79) <0C8092D3-600F-3ADD-A036-F225B6CDCA43> /usr/lib/system/libcache.dylib
0x7fffb8530000 - 0x7fffb853bff7 libcommonCrypto.dylib (60092.30.2) <B16E29B6-EC8D-3A8F-9A89-DD9CF35F7C4B> /usr/lib/system/libcommonCrypto.dylib
0x7fffb853c000 - 0x7fffb8543fff libcompiler_rt.dylib (62) <E992E8D9-037C-3454-A366-A25E4D31D6BB> /usr/lib/system/libcompiler_rt.dylib
0x7fffb8544000 - 0x7fffb854cfff libcopyfile.dylib (138) <64E285D9-5485-333B-AEE7-8B0C8FB9275F> /usr/lib/system/libcopyfile.dylib
0x7fffb854d000 - 0x7fffb85d0fdf libcorecrypto.dylib (442.30.20) <2074B932-FD79-30A9-8E90-AF25C49F2AF1> /usr/lib/system/libcorecrypto.dylib
0x7fffb85d1000 - 0x7fffb8603fff libdispatch.dylib (703.30.5) <EA0CC14E-D559-3802-B4B2-0E8C7579AAC4> /usr/lib/system/libdispatch.dylib
0x7fffb8604000 - 0x7fffb8609ff3 libdyld.dylib (421.2) <E5A2414F-2480-3BB2-BE94-176119A80876> /usr/lib/system/libdyld.dylib
0x7fffb860a000 - 0x7fffb860affb libkeymgr.dylib (28) <1A318923-1200-3B06-B432-5007D82F195D> /usr/lib/system/libkeymgr.dylib
0x7fffb860b000 - 0x7fffb8617ffb libkxld.dylib (3789.31.2) <524D7C3D-73EB-3F62-ADB1-9DFBACB95924> /usr/lib/system/libkxld.dylib
0x7fffb8618000 - 0x7fffb8618fff liblaunch.dylib (972.30.7) <15FACC21-079A-3BDF-9AFB-4253EFDEB587> /usr/lib/system/liblaunch.dylib
0x7fffb8619000 - 0x7fffb861efff libmacho.dylib (894) <A2F38EC1-C37C-3B93-B0E4-36B07C177F8C> /usr/lib/system/libmacho.dylib
0x7fffb861f000 - 0x7fffb8621ff3 libquarantine.dylib (85) <C1D7749F-5F5F-3BB9-BEFC-1F0B9DA941FD> /usr/lib/system/libquarantine.dylib
0x7fffb8622000 - 0x7fffb8623ffb libremovefile.dylib (45) <CD42974E-BE0B-39FC-9BFC-8A7540A04DC6> /usr/lib/system/libremovefile.dylib
0x7fffb8624000 - 0x7fffb863cff7 libsystem_asl.dylib (349.30.2) <EFAC72D7-CB13-3DF7-ADF3-EC6635C6F1EA> /usr/lib/system/libsystem_asl.dylib
0x7fffb863d000 - 0x7fffb863dff7 libsystem_blocks.dylib (67) <B8C3701D-5A91-3D35-999D-2DC8D5393525> /usr/lib/system/libsystem_blocks.dylib
0x7fffb863e000 - 0x7fffb86cbfef libsystem_c.dylib (1158.30.7) <2F881962-03CB-3B9D-A782-D98C1BBA4E3D> /usr/lib/system/libsystem_c.dylib
0x7fffb86cc000 - 0x7fffb86cfffb libsystem_configuration.dylib (888.30.2) <4FE3983C-E4ED-3939-A578-03AD29C99788> /usr/lib/system/libsystem_configuration.dylib
0x7fffb86d0000 - 0x7fffb86d3fff libsystem_coreservices.dylib (41.4) <1A572B9E-0C47-320F-8C64-7990D0A5FB5A> /usr/lib/system/libsystem_coreservices.dylib
0x7fffb86d4000 - 0x7fffb86ecff3 libsystem_coretls.dylib (121.31.1) <4676F06D-274D-31BE-B61C-4D7A4AEF4858> /usr/lib/system/libsystem_coretls.dylib
0x7fffb86ed000 - 0x7fffb86f3fff libsystem_dnssd.dylib (765.30.11) <DC708D84-ED7D-3936-B996-A67C66B8DDAA> /usr/lib/system/libsystem_dnssd.dylib
0x7fffb86f4000 - 0x7fffb871dff7 libsystem_info.dylib (503.30.1) <9ED9121C-F111-3FAD-BC2F-C95DEE1C9362> /usr/lib/system/libsystem_info.dylib
0x7fffb871e000 - 0x7fffb8740ff7 libsystem_kernel.dylib (3789.31.2) <FCEEBBD4-DF66-34D8-A32F-4F8C8E814E40> /usr/lib/system/libsystem_kernel.dylib
0x7fffb8741000 - 0x7fffb8788fe7 libsystem_m.dylib (3121.4) <266DB92B-A86F-3691-80FB-1B26AD73CFF3> /usr/lib/system/libsystem_m.dylib
0x7fffb8789000 - 0x7fffb87a7ff7 libsystem_malloc.dylib (116.30.3) <F40DEE3B-386A-3529-A3F7-98117ED55BF4> /usr/lib/system/libsystem_malloc.dylib
0x7fffb87a8000 - 0x7fffb87ffffb libsystem_network.dylib (856.30.16) <4AE368E9-605D-379D-B04C-2AC7455B8250> /usr/lib/system/libsystem_network.dylib
0x7fffb8800000 - 0x7fffb8809ff3 libsystem_networkextension.dylib (563.30.15) <EB020B0C-7DF0-3EEF-8E3C-15DA3C01D687> /usr/lib/system/libsystem_networkextension.dylib
0x7fffb880a000 - 0x7fffb8813ff3 libsystem_notify.dylib (165.20.1) <E7FD3A7C-DD07-36E2-9FA4-7561F9F114DA> /usr/lib/system/libsystem_notify.dylib
0x7fffb8814000 - 0x7fffb881cfe7 libsystem_platform.dylib (126.1.2) <3CA06D4E-C00A-36DE-AA65-3A390097D1F6> /usr/lib/system/libsystem_platform.dylib
0x7fffb881d000 - 0x7fffb8827ff7 libsystem_pthread.dylib (218.30.1) <C869ED7C-BE29-3532-8E69-3A8DA1447EDC> /usr/lib/system/libsystem_pthread.dylib
0x7fffb8828000 - 0x7fffb882bff7 libsystem_sandbox.dylib (592.31.1) <7BBFDF96-293F-3DD9-B3A4-7C168280B441> /usr/lib/system/libsystem_sandbox.dylib
0x7fffb882c000 - 0x7fffb882dfff libsystem_secinit.dylib (24) <5C1F1E47-0F7D-3E25-8DEB-D9DB1F902281> /usr/lib/system/libsystem_secinit.dylib
0x7fffb882e000 - 0x7fffb8835fff libsystem_symptoms.dylib (532.30.6) <5D990CF5-B58F-39F7-B375-99B4EC62CFBD> /usr/lib/system/libsystem_symptoms.dylib
0x7fffb8836000 - 0x7fffb8856ff7 libsystem_trace.dylib (518.30.7) <5BD4ECD4-75CA-38EA-AF5C-B481C15955F8> /usr/lib/system/libsystem_trace.dylib
0x7fffb8857000 - 0x7fffb885cffb libunwind.dylib (35.3) <9F7C2AD8-A9A7-3DE4-828D-B0F0F166AAA0> /usr/lib/system/libunwind.dylib
0x7fffb885d000 - 0x7fffb8886ff7 libxpc.dylib (972.30.7) <65E41BB6-EBD5-3D93-B0BE-B190CEE4DD93> /usr/lib/system/libxpc.dylib
Sample analysis of process 65830 written to file /dev/stdout
Same. I've noticed the usage climbs after a couple lock screen / login cycles.
I have the same issue. I'm also seeing this in /var/log/datadog/collector.log:
2017-02-21 09:35:48 PST | WARNING | dd.collector | checks.collector(collector.py:797) | GOHAI LOG | unexpected fault address 0xb01dfacedebac1e
fatal error: fault
[signal 0xb code=0x1 addr=0xb01dfacedebac1e pc=0x409cda6]
goroutine 7 [running]:
runtime.throw(0x43bb538, 0x5)
/usr/local/go/src/runtime/panic.go:527 +0x90 fp=0xc82002ed90 sp=0xc82002ed78
runtime.sigpanic()
/usr/local/go/src/runtime/sigpanic_unix.go:27 +0x2ba fp=0xc82002ede0 sp=0xc82002ed90
github.com/cihub/seelog.(*logConfig).IsAllowed(0x18807bddf25d1, 0x4457302, 0x73401a0, 0xc8202ca620, 0xc82002ef40)
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/cfg_logconfig.go:129 +0x26 fp=0xc82002ee58 sp=0xc82002ede0
github.com/cihub/seelog.(*commonLogger).processLogMsg(0xc8200aa5a0, 0x435be02, 0x7340178, 0xc8202fb860, 0x73401a0, 0xc8202ca620)
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/logger.go:311 +0x6c fp=0xc82002eeb8 sp=0xc82002ee58
github.com/cihub/seelog.(*asyncLogger).processQueueElement(0xc8200aa5a0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/behavior_asynclogger.go:115 +0x156 fp=0xc82002ef70 sp=0xc82002eeb8
github.com/cihub/seelog.(*asyncLoopLogger).processItem(0xc8200aa5a0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/behavior_asynclooplogger.go:57 +0x140 fp=0xc82002efa0 sp=0xc82002ef70
github.com/cihub/seelog.(*asyncLoopLogger).processQueue(0xc8200aa5a0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/behavior_asynclooplogger.go:63 +0x40 fp=0xc82002efb8 sp=0xc82002efa0
runtime.goexit()
/usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002efc0 sp=0xc82002efb8
created by github.com/cihub/seelog.NewAsyncLoopLogger
/var/cache/omnibus/src/datadog-gohai/src/github.com/cihub/seelog/behavior_asynclooplogger.go:40 +0x91
goroutine 1 [runnable]:
os.newProcess(0x1739c, 0x0, 0xc820371a40)
/usr/local/go/src/os/exec.go:21 +0x27
os.startProcess(0xc82028db60, 0x7, 0xc820371a40, 0x6, 0x6, 0xc8201173f8, 0xc8202fce00, 0x0, 0x0)
/usr/local/go/src/os/exec_posix.go:49 +0x49c
os.StartProcess(0xc82028db60, 0x7, 0xc820371a40, 0x6, 0x6, 0xc8201173f8, 0x0, 0x0, 0x0)
/usr/local/go/src/os/doc.go:24 +0x69
os/exec.(*Cmd).Start(0xc8202c0640, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:328 +0x767
os/exec.(*Cmd).Run(0xc8202c0640, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:255 +0x2d
os/exec.(*Cmd).Output(0xc8202c0640, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:408 +0x1a1
github.com/shirou/gopsutil/internal/common.Invoke.Command(0xc82028db60, 0x7, 0xc820305b80, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/shirou/gopsutil/internal/common/common.go:31 +0x7b
github.com/shirou/gopsutil/internal/common.(*Invoke).Command(0x457e280, 0xc82028db60, 0x7, 0xc820305b80, 0x5, 0x5, 0x0, 0x0, 0x0, 0x0, ...)
<autogenerated>:8 +0xee
github.com/shirou/gopsutil/process.callPs(0x44005c0, 0x10, 0x172e6, 0x0, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/shirou/gopsutil/process/process_darwin.go:430 +0x220
github.com/shirou/gopsutil/process.(*Process).MemoryInfo(0xc820117970, 0x40bb635, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/shirou/gopsutil/process/process_darwin.go:265 +0x5d
github.com/DataDog/gohai/processes/gops.newProcessInfo(0xc820117970, 0x172e6, 0x4200000000000000, 0x2, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/processes/gops/process_info.go:67 +0x3a
github.com/DataDog/gohai/processes/gops.GetProcesses(0x0, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/processes/gops/process_info.go:50 +0x623
github.com/DataDog/gohai/processes/gops.TopRSSProcessGroups(0x14, 0x0, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/processes/gops/gops.go:17 +0x50
github.com/DataDog/gohai/processes.getProcesses(0x14, 0x0, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/processes/processes.go:16 +0x5f
github.com/DataDog/gohai/processes.(*Processes).Collect(0x457e280, 0x0, 0x0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/processes/processes_common.go:22 +0x43
main.Collect(0xc82000b4a0, 0x0, 0x0)
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/gohai.go:59 +0x118
main.main()
/var/cache/omnibus/src/datadog-gohai/src/github.com/DataDog/gohai/gohai.go:158 +0x2b8
Hello, this repository has been archived, the gohai library now lives in https://github.com/DataDog/datadog-agent/pkg/gohai, feel free to re-open your issue in https://github.com/DataDog/datadog-agent if it is still relevant.
