pprof icon indicating copy to clipboard operation
pprof copied to clipboard

OS X symbolization still does not work

Open ezyang opened this issue 6 years ago • 15 comments

What version of pprof are you using?

fe62a0baab87ba3abca12f4a621532bf67c9a7d2

What operating system and processor architecture are you using?

OS X Mojave

What did you do?

$ which addr2line
/usr/local/bin/addr2line
$ cat test.cpp
#include <vector>
#include <iostream>

int main() {
  std::vector<int> x(10000000, 2);
  for (int i = 0; i < 100000; i++) {
    x[i] *= x[(i+200) % 1000];
  }
  x.resize(1000000000);
  std::cerr << x[123] << "\n";
}

$ clang++ test.cpp -g3 -fno-pie -ltcmalloc
$ HEAPPROFILE=/tmp/hh ./a.out
Starting tracking the heap
Dumping heap profile to /tmp/hh.0001.heap (3852 MB allocated cumulatively, 3852 MB currently in use)
4
Dumping heap profile to /tmp/hh.0002.heap (Exiting, 0 bytes in use)
$ pprof a.out /tmp/hh.0001.heap
Using local file a.out.
Using local file /tmp/hh.0001.heap.
Welcome to pprof!  For help, type 'help'.
(pprof) top
Total: 3852.8 MB
  3852.8 100.0% 100.0%   3852.8 100.0% 0x00007fff64e72c05

What did you expect to see?

Expected address to be symbolized

What did you see instead?

Not symbolized

ezyang avatar Jul 18 '19 01:07 ezyang

What is the version of addr2line / binutils? If it is less than 2.21, addr2line doesnt support the "-a" flag.

kalyanac avatar Jul 18 '19 11:07 kalyanac

addr2line 2.32

Excerpts from Kalyana Chadalavada's message of 2019-07-18 04:24:34 -0700:

What is the version of addr2line / binutils? If it is less than 2.21, addr2line doesnt support the "-a" flag.

ezyang avatar Jul 19 '19 20:07 ezyang

We'll do our best to look at this some time in the future, but not sure how soon this is going to be. Your best bet is to debug this a bit yourself, as you have the source code.

aalexand avatar Jul 19 '19 22:07 aalexand

My real plan is to just stop trying to run this on OS X :)

Excerpts from Alexey Alexandrov's message of 2019-07-19 15:48:03 -0700:

We'll do our best to look at this some time in the future, but not sure how soon this is going to be. Your best bet is to debug this a bit yourself, as you have the source code.

ezyang avatar Jul 20 '19 20:07 ezyang

@ezyang Can you try again and let us know if you are still having this issue?

kalyanac avatar Jan 22 '21 03:01 kalyanac

Hi. Notably, I see some symbols whenever I run without addr2line available in PATH in osx. It fails to deal with inlinings and it fails to perform C++ demangings, but it kinda works.

I.e. when running this on gperftools cpu profiler output (which uses legacy, e.g. non-protobuf format) of it's malloc_bench:

alk@hair gperftools % pprof --text pro 
Local symbolization failed for libsystem_pthread.dylib: error reading fat Mach-O file /usr/lib/system/libsystem_pthread.dylib: architecture not found in /usr/lib/system/libsystem_pthread.dylib: arm64
Local symbolization failed for libsystem_kernel.dylib: error reading fat Mach-O file /usr/lib/system/libsystem_kernel.dylib: architecture not found in /usr/lib/system/libsystem_kernel.dylib: arm64
Local symbolization failed for libsystem_malloc.dylib: stat /usr/lib/system/libsystem_malloc.dylib: no such file or directory
Local symbolization failed for libsystem_c.dylib: stat /usr/lib/system/libsystem_c.dylib: no such file or directory
Local symbolization failed for libdyld.dylib: stat /usr/lib/system/libdyld.dylib: no such file or directory
Local symbolization failed for libc++abi.dylib: stat /usr/lib/libc++abi.dylib: no such file or directory
Some binary filenames not available. Symbolization may be incomplete.
Try setting PPROF_BINARY_PATH to the search path for local binaries.
File: liboah.dylib
Type: cpu
Showing nodes accounting for 10.20s, 98.84% of 10.32s total
Dropped 10 nodes (cum <= 0.05s)
      flat  flat%   sum%        cum   cum%
     2.89s 28.00% 28.00%      6.64s 64.34%  [libsystem_malloc.dylib]
     1.86s 18.02% 46.03%      1.86s 18.02%  [libsystem_pthread.dylib]
     0.84s  8.14% 54.17%      1.85s 17.93%  _tc_malloc
     0.70s  6.78% 60.95%      3.55s 34.40%  [libc++abi.dylib]
     0.69s  6.69% 67.64%      1.75s 16.96%  _tc_free
     0.59s  5.72% 73.35%      1.39s 13.47%  __ZN12_GLOBAL__N_17mz_sizeEP14_malloc_zone_tPKv
     0.43s  4.17% 77.52%      0.43s  4.17%  __ZN15MallocExtension8instanceEv
     0.43s  4.17% 81.69%      0.50s  4.84%  __ZNSt3__124uniform_int_distributionIlEclINS_26linear_congruential_engineIjLj48271ELj0ELj2147483647EEEEElRT_RKNS1_10param_typeE
     0.37s  3.59% 85.27%      0.37s  3.59%  __ZN22TCMallocImplementation12GetOwnershipEPKv
     0.25s  2.42% 87.69%      0.25s  2.42%  __ZN8tcmalloc15CentralFreeList8PopulateEv
     0.24s  2.33% 90.02%      0.24s  2.33%  __ZN22TCMallocImplementation16GetAllocatedSizeEPKv
     0.18s  1.74% 91.76%      0.21s  2.03%  __ZN8tcmalloc15CentralFreeList14ReleaseToSpansEPv
     0.17s  1.65% 93.41%      2.07s 20.06%  __Z29perftools_pthread_getspecificm
     0.15s  1.45% 94.86%      1.78s 17.25%  __Z24randomize_one_size_classm
     0.15s  1.45% 96.32%      8.51s 82.46%  __ZL25bench_fastpath_throughputlm
     0.12s  1.16% 97.48%      0.12s  1.16%  _measure_once.cold.2
     0.07s  0.68% 98.16%      0.07s  0.68%  __ZNSt3__125__independent_bits_engineINS_26linear_congruential_engineIjLj48271ELj0ELj2147483647EEEyE6__evalENS_17integral_constantIbLb1EEE
     0.03s  0.29% 98.45%      0.27s  2.62%  __ZN8tcmalloc15CentralFreeList11InsertRangeEPvS1_i
     0.02s  0.19% 98.64%      0.29s  2.81%  __ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEji
     0.01s 0.097% 98.74%      0.27s  2.62%  __ZN8tcmalloc15CentralFreeList11RemoveRangeEPPvS2_i
     0.01s 0.097% 98.84%      0.26s  2.52%  __ZN8tcmalloc15CentralFreeList21FetchFromOneSpansSafeEiPPvS2_
         0     0% 98.84%      0.13s  1.26%  <unknown>
         0     0% 98.84%     10.32s   100%  [libdyld.dylib]
         0     0% 98.84%      1.79s 17.34%  __Z22randomize_size_classesv
         0     0% 98.84%      0.26s  2.52%  __ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEj
         0     0% 98.84%      0.27s  2.62%  __ZN8tcmalloc11ThreadCache21FetchFromCentralCacheEjiPFPvmE
         0     0% 98.84%     10.32s   100%  _main
         0     0% 98.84%      8.51s 82.46%  _measure_once
         0     0% 98.84%      8.53s 82.66%  _report_benchmark

But whenever I give it addr2line (which is whatever brew install gave me; version 2.40) I only see library names. (Just updating ticket; I am not really osex person and don't intend to try fixing the issue)

alk avatar Jun 24 '23 21:06 alk