Add perf.Benchmark(b *testing.B)

[pwaller]

This is a proposed API for making benchmark runners.

For example:

var (
	Result, v, x int
)

func BenchmarkMultiply(b *testing.B) {
	defer perf.Benchmark(b).Stop()

	for i := 0; i < b.N; i++ {
		v += 10*x
	}
	Result = v
}

Results in the following go test -bench=. output:

BenchmarkMultiply-16    	1000000000	         1.04 ns/op	         5.20 cycles/op	         1.35 instrs/cycle	         7.00 instrs/op

Before committing this needs a docstring for the benchmark function at least. I also think it would be nice if Benchmark accepted variadic measures, and added them all per-op.

I note that this functionality is only supported in go 1.13, so for previous versions the benchmarks are skipped with a message indicating this.

[acln0]

I like the idea very much!

I had something similar in mind, but I didn't know the custom labels proposal ended up getting merged for 1.13. Great news.

I wonder if the API could be made more flexible by letting the user pass an Event or even a Group, and somehow automatically reporting metrics based on the event(s), instead of hard-coding cycles / op, IPC, and instructions / op.

[pwaller]

Agreed! I'd love to see this improved. I'd also love for the base case to be a small one-liner if possible, since this is a common thing to want to measure, I believe. One idea I had is that it could be named BenchmarkIPC, and Benchmark() could be more general/configurable.

I defer to your great taste in general.

[thepudds]

This is very nice.

For the not-yet-written doc string, would it make sense to include a couple brief points about how instrs/cycle can be useful?

Perhaps the doc string could also include a link to http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html?

[bmkessler]

This is really cool. One API option would be:

func Benchmark(b *testing.B, cfgs ...Configurator) Stopper

And then you could pass in whatever config to output and treat len(cfgs) == 0 in the Benchmark(b) case as just the default Instructions and CPUCycles as in the current change.

The main disadvantages of that approach is you wouldn't get to output derived metrics like IPC since there would be no way to specify relevant relations. The output names would also come from the Attr associated with each Configurator so would be more verbose Attr.Label/op. The default case would end up looking like:

BenchmarkMultiply-16 1000000000 1.04 ns/op 5.20 cpu-cycles/op 7.00 instructions/op

So you would have to do the IPC math yourself, but then you could easily get things like branch-misses/op and cache-misses/op if you wanted from the same API.

[pwaller]

Anyone interested in picking this up from me?

Alternatively, any objections to merging this as is? I'd love to see this functionality easily to hand in a public library.

[pwaller]

Friendly ping. Every once in a blue moon I want to use this functionality. Would be good to have it in the base library.

[pwaller]

Ping again.

[pwaller]

Friendly ping again. Hope you're doing OK. These open PRs are the last standing in my outbox. I'd love to get to zero.