ginkgo icon indicating copy to clipboard operation
ginkgo copied to clipboard

Add some micro-benchmarking support to Ginkgo

Open pratikvn opened this issue 3 years ago • 7 comments

This PR adds some micro-benchmarking support to benchmark all the small functionalities of Ginkgo. For example, you can get information like the following:

  1. Executor creation and run timings and overheads. mbench

  2. Array creation, copy, move timings and differences. array-mbench

  3. More specific backend-specific timings as well are possible, example for arrays with CUDA: cuda-array-mbench

Of course there are a few things that we need to discuss:

  1. Timings for GPUs is from the host side, but that may be okay.
  2. The repetitions for the function calls is not something that can be clearly controlled through google benchmark, which as discussed before is a downside of this.

I think this type of micro-benchmarking information can be really useful in detecting performance degradations and can atleast give us a hint if some commit/change regressed the performance.

pratikvn avatar Jun 11 '21 09:06 pratikvn

Kudos, SonarCloud Quality Gate passed!

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
9.5% 9.5% Duplication

sonarqubecloud[bot] avatar Jun 11 '21 12:06 sonarqubecloud[bot]

Codecov Report

Merging #790 (ff9b336) into develop (6127408) will decrease coverage by 0.00%. The diff coverage is n/a.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #790      +/-   ##
===========================================
- Coverage    94.74%   94.73%   -0.01%     
===========================================
  Files          434      434              
  Lines        35705    35706       +1     
===========================================
  Hits         33827    33827              
- Misses        1878     1879       +1     
Impacted Files Coverage Δ
core/base/extended_float.hpp 91.26% <0.00%> (-0.98%) :arrow_down:
omp/reorder/rcm_kernels.cpp 97.53% <0.00%> (-0.61%) :arrow_down:
include/ginkgo/core/base/range_accessors.hpp 78.94% <0.00%> (+5.26%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6127408...ff9b336. Read the comment docs.

codecov[bot] avatar Jun 11 '21 12:06 codecov[bot]

@upsj, Thank you for the review and the comments. I will try to update it soon.

pratikvn avatar Jun 17 '21 07:06 pratikvn

Thanks for this, I think using google benchmark for these parts can make sense, since that's pretty much what it was designed for. I have a few remarks after a quick glance at the code, although I didn't look into it in depth:

  • I like the name micro_benchmarks for the directory as that explains what should go in there.
  • I added some overhead benchmarks to our base benchmark line, maybe that can go in there instead (or in addition?).
  • We maybe should use the full name googlebenchmark for the targets and some other aspects, at first it wasn't clear to me what GBench and mbench were. I would stick to using micro_benchmarks always for the Ginkgo side, and the full googlebenchmark or whichever is the original name for the third party.
  • A big question is CUDA: by default the timer is CPU only and that's terrible for doing GPU micro-benchmarking. I know it's possible to add custom timers but I'm not sure how extensible they are, back when I had tried it (many years ago) that aspect wasn't satisfactory, maybe it's better now?

tcojean avatar Jun 17 '21 08:06 tcojean

@tcojean, thanks. I will look into GPU timers and see if it extensible from within googlebenchmark. I agree with your other suggestions.

pratikvn avatar Jun 17 '21 09:06 pratikvn

Error: The following files need to be formatted:

core/micro_bench/base/array.cpp
core/micro_bench/base/executor.cpp
cuda/micro_bench/base/array.cu
cuda/micro_bench/base/executor.cu

You can find a formatting patch under Artifacts here or run format! if you have write access to Ginkgo

ginkgo-bot avatar Oct 19 '21 15:10 ginkgo-bot

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 2 Code Smells

No Coverage information No Coverage information
9.5% 9.5% Duplication

sonarqubecloud[bot] avatar Oct 20 '21 00:10 sonarqubecloud[bot]