pyroscope icon indicating copy to clipboard operation
pyroscope copied to clipboard

Have a way to look at small stacks

Open eh-am opened this issue 3 years ago • 1 comments

Sometimes there're stacktraces with small samples, which are not possible to "drill down" in the UI. image

Although for most use cases we want to look at the "bigger" ones, it's still useful to drill down the smaller ones, specially in the context of exploration.

Suggestions of UI approaches are welcome.

eh-am avatar Apr 07 '22 18:04 eh-am

We got same issue. With many independent cli scripts its very hard to work with flamegraph.

Our reglar workflow is:

  1. Find method in table and click on it to make it hightlighted in flamegraph.
  2. Open flamegraph in full width and increase browser width until selected block became visible.

Is it possible to add some option in config to make all blocks selectable / highlightable? Or add some button to focus on selected function on flamegraph?

xtrime-ru avatar Apr 29 '22 22:04 xtrime-ru

Now that we've added #1636 I think we can revisit this with another "viewing" mode that we could call something like "inspector" mode where your cursor turns into a magnifying glass and then you select the area where you would like to zoom: inspector_mode_01

Selection could look something like this: heatmap_vid_01

Then after you have selected an area the canvas will "zoom" so that that area is the only area you are looking at. Later if we want to get fancy we could make it so that when the flamegraph is in a "zoom" state that you can navigate it using a hand cursor and grabbing and moving the flamegraph around

(similar to adobe suite) Screen Recording 2022-11-24 at 5 04 10 PM

wdyt @eh-am ? @xtrime-ru ?

Rperry2174 avatar Nov 24 '22 22:11 Rperry2174

@Rperry2174 Looks awesome!

And what about color highlight for selected items from table in such narrow branches?

Off-top question: How can we get rid of "other" in flamegrapgh? See it quite often in deep traces: Снимок экрана 2022-11-24 в 23 20 26

xtrime-ru avatar Nov 24 '22 22:11 xtrime-ru

@xtrime-ru this typically happens when the trees are too large. Normally the defaults work fine, but in some cases they might lead to this issue with excessive “other” nodes. There are two settings that affect this:

  • max-nodes-serialization
  • max-nodes-render

The first one is used when saving data on disk, the second one is used when rendering data. You can see which limit you’re currently hitting by adding &maxNodes=10000000 to the URL — if you’re hitting max-nodes-render value the “other” nodes will disappear, if you’re hitting max-nodes-serialization they will still be there.

One other thing I should mention is that in version 0.32.0 the serialization algorithm was improved so that even with default settings we do not truncate the trees as much as we used to, preserving the tree structure for as long as possible.

In addition to that, we’ve added support for -1 value for both max-nodes-render and max-nodes-serialization parameters. This allows you to disable the truncation of the trees altogether.

We’ve done tests with an app that has many code paths and pretty diverse trees and the results were good. However, using -1 is definitely a tradeoff, so here’s a few things to keep in mind:

  • Setting max-nodes-serialization to -1 will increase storage requirements. Good news is that it does not significantly increase CPU usage in ingestion path.
  • Setting max-nodes-render to -1 will make rendering slower. Hard to put a number on this one as it depends on your internet connection and your computer

I would not recommend setting both parameters to -1 if you have applications that generate dynamic code paths. In this case I would just set the values higher than defaults (default for serialization is 2048 and render is 8192)

So to summarize, my recommendations are as follows:

  • Definitely upgrade to the latest version. This shouldn’t affect the performance much, but it will improve the accuracy of the flamegraphs.
  • Consider setting max-nodes-serialization and max-nodes-render parameters to -1 or just higher values. See the tradeoffs described above.

Hope that helps, and let me know if you have any other questions!

Rperry2174 avatar Nov 24 '22 22:11 Rperry2174

@Rperry2174 Thanks for such detailed answer. Ive increased max-nodes-serialization and max-nodes-render in 2 times in config and already can see, "other" items share reduced in 2 times.

Also i tried new "sandwich" mode and it helps alot with finding the right path!

xtrime-ru avatar Nov 24 '22 23:11 xtrime-ru