flamegraph icon indicating copy to clipboard operation
flamegraph copied to clipboard

Incorrect or partial flamegraph output on MacOS

Open mabbamOG opened this issue 4 years ago • 10 comments

I have recently run cargo flamegraph on MacOS and Linux, and realised that the MacOS svg-output variant is missing a lot of important contextual information. Is there a way to solve this? The package seems quite unreliable or even broken on Macs....

mabbamOG avatar Dec 30 '20 23:12 mabbamOG

Are you running version 0.4? Earlier versions had some issues. The information provided is necessarily different on macOS, where we use dtrace compared to Linux, where we use perf. I'm not sure there's much about that we can do inside the flamegraph codebase, although maybe we can pass options to dtrace to make its output more comprehensive.

djc avatar Dec 31 '20 13:12 djc

Are you running version 0.4? Earlier versions had some issues.

$ cargo flamegraph --version
cargo-flamegraph-flamegraph 0.4.0

The information provided is necessarily different on macOS

The MacOS version is definitely missing a lot of information, and sometimes it is even misleading because some of the program's internal functions are replaced with the names of "more internal" (read: deeper into the stack trace) functions; furthermore, the stack trace on MacOS is much more shallow than the one from Linux, it looks more like a "streetgraph" than a "flamegraph" because so many internal calls are missing.

The situation is such that i cannot rely on the MacOS flamegraph, and without the linux version i would not have been able to accurately profile and optimise my program. I suspect the Mac version has optimised a lot of debug info out of the binary, so perhaps a few dtrace options can do the trick here indeed.

I can share the flamegraphs if need be, but first i would need to change all the function names, as well as those of some internal libraries...

mabbamOG avatar Jan 03 '21 08:01 mabbamOG

I understand that part. It's just that this project is merely a convenient abstraction over running perf or dtrace (as appropriate for the host platform) and generating flamegraphs from it. As far as I know we run a pretty minimal perf or dtrace command, so while I'd be okay with extending that if it substantially improves the results, it's not really within my expertise, and it's also somewhat at the scope of what this project is about IMO (but maybe @spacejam thinks differently).

djc avatar Jan 03 '21 19:01 djc

As a concrete example, the same flamegraph generated on

Linux

image

macOS

image

shepmaster avatar Feb 07 '21 19:02 shepmaster

Yup, I've also seen similar results in my own projects. Still not sure what we could do to tune dtrace to do better.

djc avatar Feb 08 '21 08:02 djc

I hacked things up to use profile-4999 instead of profile-997, but that did not measurably change the output.

shepmaster avatar Feb 09 '21 01:02 shepmaster

I suspect the issue is missing resolution of inline frames. I ran into this yesterday, comparing the output of cargo flamegraph with whatever an Instruments profile gives me. It really looked like inlined frames were completely missing from the flamegraph output.

Swatinem avatar Nov 12 '21 11:11 Swatinem

It's definitely inline frames that are missing but it makes me wonder how inline frames are actually working on other platforms. The output from dtrace's ustack() do not appear to produce inline frames for me but depending on what I am targeting I get just hex addresses or symbolized (not not with inlines resolved) frames. I assume that on other platforms dtrace actually resolves inline addresses. The code indicates that -> would be used to express inline frames, yet I can only find them for the java samples in the repo.

I'm guessing it's dtrace itself that is supposed to produce that, so I wonder if what dtrace uses for symbol resolution fails now. I tried to use other values for split-debuginfo but changing that didn't improve anything.

mitsuhiko avatar Feb 15 '22 01:02 mitsuhiko

how is this going? it doesn't help much with such flamegraph:

image

hanayashiki avatar Oct 07 '22 09:10 hanayashiki

@hanayashiki it looks like there has been some progress in https://github.com/jonhoo/inferno/issues/246, but not much.

djc avatar Oct 10 '22 09:10 djc