BenchmarkTools.jl

BenchmarkTools.jl copied to clipboard

It's unlikely I'll get around to doing this in the foreseeable future, but I'm tired of digging through issues to find this comment when I want to link it in...

jrevels

I hope this is isn't a usage question but I think it would be nice if ```julia julia> judge(memory(oldresults["solver"]), memory(results["solver"])) ERROR: MethodError: no method matching judge(::Int64, ::Int64) ``` would work

mschauer

governors advice may be obsolete in linuxtips.md

1

[linuxtips.md](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/linuxtips.md#cpu-frequency-scaling-and-boosting) mentions CPU governors and suggests `performance`. However, after the introduction of `pstate` in [Sandy Bridge](https://wiki.archlinux.org/index.php/CPU_frequency_scaling), the default processor frequency scaling became very responsive. In my own benchmarks, I frequently...

tpapp

change name to Benchmarking.jl?

4

BenchmarkTools is the bomb and everyone uses it – or should! The name is a little awkward though and doesn't have quite the gravitas and officialness that it deserves. What...

StefanKarpinski

Remove vector indexing in favor of foldl

1

BenchmarkGroups currently implements an indexing scheme where [a key vector indicates nested indexing](https://github.com/JuliaCI/BenchmarkTools.jl/blob/master/doc/manual.md#indexing-into-a-benchmarkgroup-using-a-vector). This can and should be removed in favor of using `foldl(getindex, g::BenchmarkGroup, keys)` (ref [here](https://github.com/JuliaLang/julia/issues/23031#issuecomment-320080560)).

jrevels

`setup`/`teardown` doesn't allow multiple assignment syntax inside Tuple

2

```julia julia> using BenchmarkTools # this should work, but doesn't julia> @benchmark (a + b + c) setup=(a,b,c=1,2,3) ERROR: syntax: assignment not allowed inside tuple in generate_benchmark_definition(::Module, ::Array{Symbol,1}, ::Array{Symbol,1}, ::Expr,...

jrevels

define ratio and/or `/` for Trials

1

This would compute the ratio of all the relevant metrics: - memory estimate - allocs estimate - minimum time - median time - mean time - maximum time Would it...

StefanKarpinski

Document profiling with @bprofile

3

It would be fantastic if there was a way to get a profile trace of an `@benchmarkable` object, and exploit some of the nice features (i.e. run it enough times...

simonbyrne

Allocations and GC being ignored?

8

I'm finding different results using BenchmarkTools as opposed to my own loops and `@time`, particularly regarding allocating operations. I'm benchmarking `Base.Array` vs the types in *StaticArrays.jl*. Some of these arrays...

andyferris

A better way to inspect the code of the benchmark loop/kernel

1

Doing benchmark correctly sometimes involving making sure there's no additional overhead (e.g. due to accidental use of global variables) and the operation being benchmarked isn't optimized out (most likely due...

yuyichao