opentelemetry-profiling icon indicating copy to clipboard operation
opentelemetry-profiling copied to clipboard

Initial benchmark setup

Open petethepig opened this issue 3 years ago • 1 comments

This PR moves benchmarking code to opentelemetry repo. It is based on the code from this repo.

I made some additional changes and improvements:

  • added a pprof parser, so now you can provide profiles in collapsed and pprof formats
  • made it so that anonymizer only runs on files marked as sensitive. Unfortunately anonymization affects compression ratio, so I figured we might want to use anonymizer as little as possible.
  • added multiple encoders: json, pprof, pprof-gzip, to showcase that we can compare different encoder implementations. Interesting next step would be to add an otel events encoder (perhaps with gzip compression) and compare it to pprof implementation.
  • added a few new profile examples, goapp and goapp.sensitive, to showcase the difference in encoding data before and after anonymization
  • implemented collection of byte size and CPU time (duration) metrics
  • added some basic formatting for the benchmarking results when printed in the terminal:
Click to expand terminal example
Filename: basic.collapsed.intermediary
+------------+----------+---------+-----------+---------+
| ENCODER    | DURATION | % DIFF  | BYTE SIZE | % DIFF  |
+------------+----------+---------+-----------+---------+
| pprof-gzip | 251.7µs  |         | 114 bytes |         |
| pprof      | 3.5µs    | -98.59% | 89 bytes  | -21.93% |
| json       | 469ns    | -99.81% | 49 bytes  | -57.02% |
+------------+----------+---------+-----------+---------+

Filename: basic.sensitive.collapsed.intermediary
+------------+----------+---------+-----------+---------+
| ENCODER    | DURATION | % DIFF  | BYTE SIZE | % DIFF  |
+------------+----------+---------+-----------+---------+
| pprof-gzip | 151.5µs  |         | 114 bytes |         |
| pprof      | 3µs      | -98.05% | 89 bytes  | -21.93% |
| json       | 484ns    | -99.68% | 49 bytes  | -57.02% |
+------------+----------+---------+-----------+---------+

Filename: goapp.pprof.intermediary
+------------+----------+---------+------------+----------+
| ENCODER    | DURATION | % DIFF  | BYTE SIZE  | % DIFF   |
+------------+----------+---------+------------+----------+
| pprof-gzip | 2.6ms    |         | 16.60 KiB  |          |
| pprof      | 982.1µs  | -61.63% | 40.01 KiB  | +141.06% |
| json       | 438.3µs  | -82.88% | 147.56 KiB | +788.95% |
+------------+----------+---------+------------+----------+

Filename: goapp.sensitive.pprof.intermediary
+------------+----------+---------+------------+----------+
| ENCODER    | DURATION | % DIFF  | BYTE SIZE  | % DIFF   |
+------------+----------+---------+------------+----------+
| pprof-gzip | 3.2ms    |         | 25.31 KiB  |          |
| pprof      | 1.4ms    | -56.08% | 40.01 KiB  | +58.07%  |
| json       | 418.4µs  | -86.94% | 147.56 KiB | +482.92% |
+------------+----------+---------+------------+----------+
  • implemented a markdown rendered as well. For now this helps share results on Github. Maybe one day we'll make it post this on github PRs automatically:

Filename: basic.collapsed.intermediary

Encoder Duration % diff Byte Size % diff
pprof-gzip 251.7µs 114 bytes
pprof 3.5µs -98.59% 89 bytes -21.93%
json 469ns -99.81% 49 bytes -57.02%

Filename: basic.sensitive.collapsed.intermediary

Encoder Duration % diff Byte Size % diff
pprof-gzip 151.5µs 114 bytes
pprof 3µs -98.05% 89 bytes -21.93%
json 484ns -99.68% 49 bytes -57.02%

Filename: goapp.pprof.intermediary

Encoder Duration % diff Byte Size % diff
pprof-gzip 2.6ms 16.60 KiB
pprof 982.1µs -61.63% 40.01 KiB +141.06%
json 438.3µs -82.88% 147.56 KiB +788.95%

Filename: goapp.sensitive.pprof.intermediary

Encoder Duration % diff Byte Size % diff
pprof-gzip 3.2ms 25.31 KiB
pprof 1.4ms -56.08% 40.01 KiB +58.07%
json 418.4µs -86.94% 147.56 KiB +482.92%

petethepig avatar Nov 15 '22 09:11 petethepig