implement heap profiler

[YangKeao]

Fix #44. It seems that the protobuf generation still has a lot of bugs :(.

The flamegraph output works as expected. However, the svg generated by go tool pprof through a protobuf is too simple. The recursive part in the example fails to display in the svg.

The example code snippet is in /examples/heap_profiler.rs. @mineralres Are you willing to take a look and find the problem of protobuf? Feel free to ask any questions about this project. I'm not confident about the code quality (as I have found that it's hard to extend this feature without breaking the compatibility), so if you are anxious about the codes, I'm on your side :sob: .

pprof.zip is the generate flamegraph and protobuf.

Signed-off-by: Yang Keao [email protected]

[mineralres]

@YangKeao I can have a try. Before that , i had to spend some time to be familiar with the whole pprof-rs project.

[YangKeao]

I'm busy with other projects (actually, two projects 😭 ), so this PR may take several weeks to land. There are a lot of things left:

• [ ] set title and units for heap profiler output
• [ ] reorganize the code structure

[mkmik]

Just heads-up: I was playing with a similar approach in https://github.com/mkmik/heappy

I'd like to contribute it upstream here once I figure out whether it works well in practice.

[YangKeao]

Just heads-up: I was playing with a similar approach in https://github.com/mkmik/heappy

I'd like to contribute it upstream here once I figure out whether it works well in practice.

Thanks :+1: . I could provide some thoughts about this PR and I hope it could help you :beers: .

1. One of the most annoying problem is to avoid recursion while writing the collector. The recursion could be caused by the allocation of the same allocator: malloc -> collector::record -> malloc -> collector::record .... Using a thread local flag to store whether this process is inside the malloc could be a solution (here is the pseudo-code):

   fn malloc() {
     in_malloc = true;
     if !in_malloc {
       collector::record
     }
     in_malloc = false;
   }
   

   Another possible solution is to use collections with the inner allocator, which depends on the language feature https://github.com/rust-lang/rfcs/pull/1398. The disadvantages of these solutions are that they fail to record the collector memory footprint.

2. I have to say the methods and structs to generate report (like flamegraph and protobuf) in pprof-rs are not well designed :man_facepalming: . If you need any help to make it work, feel free to contact me and I would try my best to help.

[kwiesmueller]

What is the status of this? Getting heap profiling to work would be great.

[Rperry2174]

We also have gotten a lot of requests for this at Pyroscope :) -- let us know if there's any way we can help @YangKeao !

[mkmik]

@kwiesmueller have you tried out https://github.com/mkmik/heappy ?

[caiobegotti]

Hi there @YangKeao do you still have time or are you still after this, please? That would be much appreciated by many people, given the time since the last update in the PR 🙇 cc: @mkmik too, I guess?

[mkmik]

@caiobegotti have you tried heappy? I would love some feedback