Add a sketch of a runtime benchmark suite

[Kobzol]

This PR is a sketch of a very bare-bones implementation of runtime benchmarks. It's not even a draft at this point (which can also be inferred from the lolperf naming :) ), I just wanted to have a more concrete piece of code that we could discuss rather than the general ideas from https://hackmd.io/j19WFamVQlmQMRigF9wrFw.

This PR is mostly about the rustc-perf infrastructure changes (without considering DB and UI yet) and not about the benchmarks itself, I just created three sample ones.

Currently the design looks like this:

• There is a directory collector/lolperf that contains a separate Cargo workspace. It contains benchlib, a simple profiling library that measures metrics like icounts, cycles, wall-time etc. It currently uses perf(_event_open) to measure a specific piece of code, but it could be changed to use callgrind/cachegrind or something else.
• There is a series of crates under lolperf/benchmarks that use the benchmarking lib to define a set of benchmarks. The benchmarks are not executed right away, the benchlib instead decides what to do once it parses CLI arguments (it could have commands e.g. to list available benchmarks and their core requirements, so that when you actually execute the benchmark, you can select to how many cores you could pin it).
• A new command called bench_runtime_local was added to collector. It takes a path to rustc, uses it to compile the runtime benchmarks and then executes them. Eventually, it might make sense to instead create something like bench_local comptime --scenarios ... and bench_local runtime .... The data benchmark measurement results are currently passed as JSON from the benchmark binary to collector. All of the measurements and JSON output is handled by benchlib.

That's the high-level gist. Pretty much anything of this can be changed or bikeshed, and obviously there's a myriad of things missing (like actually storing the results into the DB and providing a UI for it, but that will probably be in separate PRs).

[anp]

This is awesome! After this last RustConf I've been thinking about spending time on this kind of thing again, let me know if I can help! I'm still waiting on some new Linux hardware to be able to use perf_event_open locally but I'll be able to help with results/UI/regression detection/porting benchmarks/etc soon.

[Kobzol]

I think that we first have to get the basic infrastructure in place (code layout, measurements, DB, etc.), I'm not sure how much that is "parallelizable". But once that's in place, we will need to populate the benchmark suite and possibly build some new UI for that, and then it will definitely be good to cooperate :) It might still be a way off though, the infrastructure will probably go through a lot of iterations.

[Kobzol]

I also want to make the structure similar to comptime benchmarks, I just put it into a separate directory for now to better separate it from the rest of the code for now. There's also one additional thing instead of just a directory of benchmarks, namely the benchmarking library. I'm fine with changing the file/directory layout in any way we find better.

[Kobzol]

Why do the benchmarks appear twice? E.g. collector/lolperf/benchmarks/nbody/src/nbody.rs and runtime/benchmarks/nbody/src/nbody.rs. Same story with other source files like measure.rs.

Ooos, sorry, I accidentally committed another directory where I have stored it previously. And I was wondering why the diff is so large.. :sweat_smile:

[Kobzol]

I got some nasty diffs there because of a botched directory move, now it should be fine.

[Kobzol]

@rylev Can you take a look please? :)

[Kobzol]

We can also name it comptime-benchmarks? But I think that compile vs runtime is obvious enough.

[nnethercote]

Ok, compile-benchmarks is good enough.

[Kobzol]

I'm going to go ahead and merge it so that we can start experimenting with it.