use criterion for the benchmarks

[stlankes]

like @jbreitbart suggestion in #50, we should use criterion for the benchmarks

[garrettgu10]

Hello, it's that team from UT Austin again. ;-)

We hit a bit of a wall after implementing the RISC-V bootloader, can we work on this?

Also, is this intended to completely replace the current python-based matplotlib visualizer, or should we aim to maintain support for that?

Thanks!

[jbreitbart]

I guess I have never seen the matplot visualizer, but I found the criterion visualization sufficient and would not need anything beyond that. IIRC criterion also provides its measurements in a format that should be easy to parse and import into another visualizer. I am not sure if we will be able to replicate all benchmarks with criterion, so I guess that is more of an issue with retiring the existing visualizer 😄

Btw: There are some criterion benchmarks in uhyve just in case you have not seen them.

[garrettgu10]

@jbreitbart @stlankes One issue we ran into is that criterion is supposed to run on the host OS (since it needs to write to files and stuff) but the benchmarks here (unlike the benchmarks under uhyve) are supposed to run with HermitCore on the hypervisor.

Our first attempt to port these benchmarks over was to compile Criterion in with the benchmarked code but that means that Criterion has to run within HermitCore, causing issues when it attempts to write results to a file. We believe this is why the original system does not use Criterion.

We were envisioning a system where the benchmarked functions (such as the micro one that performs 1,000,000 syscalls) would be compiled separately from Criterion, and Criterion would then run on the host machine and use Command::new() to execute uhyve, using the kernel arguments to specify which benchmarked function to run. Some tradeoffs is that we have way more benchmark overhead and no possibility of warming up processor caches prior to starting the benchmarked functions.

Ultimately we're not convinced that the second solution is worth the tradeoffs, and we would like to hear your thoughts on the matter.

Thanks!

[jbreitbart]

Can you provide details on the error you get when writing files? That should be possible...

[garrettgu10]

Oh yeah, that was my bad. Just 2 AM things. Here's the error:

Criterion.rs ERROR: error: Failed to access file "target/criterion/1000_syscalls/new": operation not supported on HermitCore yet

This corresponds to an AccessError within Criterion, which seems to only be created from here. We suspect that HermitCore may not be happy with the relative path. (?)

[jbreitbart]

Oh, sorry, I didn't want to be rude. I was referring to the fact that file I/O should be possible. :)

Can you try to change the output path to a absolute path as described here: https://bheisler.github.io/criterion.rs/book/cargo_criterion/configuring_cargo_criterion.html

[garrettgu10]

@jbreitbart No worries! Those instructions are for cargo-criterion, which is criterion's new experimental custom Cargo extension. The issue with this one is that cargo-criterion wants to run on the host. (attempting to compile cargo-criterion for x86_64-unknown-hermit fails with "can't find crate for core".) Because of this, cargo-criterion cannot detect the criterion_main of the bench target, (error is "Non-Criterion.rs benchmark target micro exited with error code None") since the bench target is compiled for hermit while cargo-criterion is compiled for linux. I also couldn't find any way to get criterion to use a custom file path, it really wants to use target/criterion to store its files.

Is it safe to say that porting benchmarks to criterion is blocked on supporting at least some notion of relative paths?

Thanks!

[jbreitbart]

I just gave it a try and relative paths work fine at least for open. My guess is that support for creating directories is not yet implemented and criterion is trying to create some. Is target/criterion/ created? Are there any subdirectories?

@stlankes Any idea if directory creation should be working with uhyve?

[stlankes]

Ah, I think that you are right. Their is a system call missing. I will check it.

[stlankes]

@garrettgu10 Relativ paths should now work. But I think that you also need the possibility to create a directory. This is currently missing. But I will add it soon.