NVIDIA
Comparison outputs should be colorful when pasted into GitHub
Copying and pasting a markdown comparison report to GitHub will render as tables and look like this:
| T{ct} | OffsetT{ct} | Elements{io} | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status |
|---|---|---|---|---|---|---|---|---|---|
| I8 | I32 | 2^16 | 6.092 us | 7.97% | 6.189 us | 9.24% | 0.097 us | 1.60% | SAME |
| I8 | I32 | 2^20 | 6.703 us | 11.35% | 7.095 us | 12.43% | 0.391 us | 5.84% | SAME |
| I8 | I32 | 2^24 | 14.092 us | 2.18% | 16.161 us | 2.11% | 2.068 us | 14.68% | SLOW |
| I8 | I32 | 2^28 | 115.971 us | 0.72% | 128.828 us | 0.24% | 12.857 us | 11.09% | SLOW |
Sometimes, pasting as preformatted text is denser:
| T{ct} | OffsetT{ct} | Elements{io} | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status |
|---------|---------------|----------------|------------|-------------|------------|-------------|-----------|---------|----------|
| I8 | I32 | 2^16 | 6.092 us | 7.97% | 6.189 us | 9.24% | 0.097 us | 1.60% | SAME |
| I8 | I32 | 2^20 | 6.703 us | 11.35% | 7.095 us | 12.43% | 0.391 us | 5.84% | SAME |
| I8 | I32 | 2^24 | 14.092 us | 2.18% | 16.161 us | 2.11% | 2.068 us | 14.68% | SLOW |
| I8 | I32 | 2^28 | 115.971 us | 0.72% | 128.828 us | 0.24% | 12.857 us | 11.09% | SLOW |
(a bit wider but less long)
Both versions loose color though, which makes reading the benchmarks on GitHub harder. The console output contained colors. We should find a way to make GitHub render the comparison output with colors as well. One way to do this, is to abuse the diff highlighting and prepend lines with +, -, etc.:
| T{ct} | OffsetT{ct} | Elements{io} | Ref Time | Ref Noise | Cmp Time | Cmp Noise | Diff | %Diff | Status |
|---------|---------------|----------------|------------|-------------|------------|-------------|-----------|---------|----------|
| I8 | I32 | 2^16 | 6.092 us | 7.97% | 6.189 us | 9.24% | 0.097 us | 1.60% | SAME |
!| I8 | I32 | 2^20 | 6.703 us | 11.35% | 7.095 us | 12.43% | 0.391 us | 5.84% | SAME |
-| I8 | I32 | 2^24 | 14.092 us | 2.18% | 16.161 us | 2.11% | 2.068 us | 14.68% | SLOW |
+| I8 | I32 | 2^28 | 115.971 us | 0.72% | 128.828 us | 0.24% | 12.857 us | 11.09% | SLOW |
Kudos to @miscco for the idea.
we can do even better https://mermaid.js.org/intro/#xy-chart
xychart-beta
title "Perf"
x-axis ["2^16", "2^20", "2^24", "2^28"]
y-axis "Speedup (in %)" 0 --> 15
bar [1.60, 5.84, 14.68, 11.09]
generates
xychart-beta
title "Perf"
x-axis ["2^16", "2^20", "2^24", "2^28"]
y-axis "Speedup (in %)" 0 --> 15
bar [1.60, 5.84, 14.68, 11.09]
Sure, but I need such plots for benchmarks with 160 numbers, like the one here: https://github.com/NVIDIA/cccl/pull/5122. Could you try to plot that? I am curious how it looks :D
the syntax looks very flexible. What about now 😄?
---
config:
xyChart:
height: 900
---
xychart-beta horizontal
title "Perf"
x-axis [" I8 / I32 / 2^16", " I8 / I32 / 2^20", " I8 / I32 / 2^24", " I8 / I32 / 2^28", " I8 / I64 / 2^16", " I8 / I64 / 2^20", " I8 / I64 / 2^24", " I8 / I64 / 2^28", " I16 / I32 / 2^16", " I16 / I32 / 2^20", " I16 / I32 / 2^24", " I16 / I32 / 2^28", " I16 / I64 / 2^16", " I16 / I64 / 2^20", " I16 / I64 / 2^24", " I16 / I64 / 2^28", " F32 / I32 / 2^16", " F32 / I32 / 2^20", " F32 / I32 / 2^24", " F32 / I32 / 2^28", " F32 / I64 / 2^16", " F32 / I64 / 2^20", " F32 / I64 / 2^24", " F32 / I64 / 2^28", " F64 / I32 / 2^16", " F64 / I32 / 2^20", " F64 / I32 / 2^24", " F64 / I32 / 2^28", " F64 / I64 / 2^16", " F64 / I64 / 2^20", " F64 / I64 / 2^24", " F64 / I64 / 2^28", "I128 / I32 / 2^16", "I128 / I32 / 2^20", "I128 / I32 / 2^24", "I128 / I32 / 2^28", "I128 / I64 / 2^16", "I128 / I64 / 2^20", "I128 / I64 / 2^24", "I128 / I64 / 2^28"]
bar [8.50, 7.12, 11.05, 12.78, 9.71, 8.35, 14.70, 22.16, 2.78, 3.53, 0.75, -0.16, 3.45, 3.86, 0.58, -0.16, 2.47, 0.86, 0.10, -0.04, 1.77, 0.80, -0.00, -0.06, 1.70, 0.46, -0.11, -0.03, 1.00, 0.41, -0.10, -0.04, 0.78, 0.15, -0.09, -0.02, -0.12, -0.91, -0.10, 0.00]
ok, this is awesome! I also didn't realize you could just paste markdown-like code into a GH text box and it would render. Amazing! Now we need a mermaid backend for nvbench!