Benchmark within crate source code isn't running

[vlovich]

I'm trying out Divan on Linux. My project is a workspace layout. Within that workspace I have a crate foo. Within foo/benches I create a foo.rs file that has:

fn main() { eprintln!("Benchmark main"); divan::main() }

and register it within foo/Cargo.toml:

[[benches]]
name = "foo"
harness = false

Then within foo/src/lib.rs I put:

#[divan::bench]
pub fn bench() {
  eprintln!("Running registered benchmark");
}

I see "Benchmark main" when I run cargo bench -p foo but not "Running registered benchmark". If I move the bench function to foo/benches/foo.rs then everything works. Not sure what I need to do to make this work.

Semi-related, does putting benchmark code within the crate mean that downstream dependencies compile the benchmarks? I'm hoping it at least is dead-code stripped but would be good to call this out in the docs.

[nvzqz]

Might be that you're not actually using anything from the lib and thus it gets completely removed from your dependency tree. Can you please check if either of the following fixes this?

• Use an item from the library within your benchmarks
• Emit #[used(linker)] in the proc macro

Semi-related, does putting benchmark code within the crate mean that downstream dependencies compile the benchmarks?

Yes. That is why Divan has an internal_benches feature flag. See https://github.com/nvzqz/divan/issues/2#issuecomment-1785991600 for details.

I'm hoping it at least is dead-code stripped but would be good to call this out in the docs.

Dead code elimination doesn't apply here because the approaches used (pre-main and linkme) each have the symbols be required to exist in the final binary regardless of whether they're used.

[vlovich]

Ah, yes. I'm probably not using anything from the lib. Will try again.

FWIW the workaround I'm using re feature flag is to have the library have a dev-dependency on itself with the benchmark feature enabled. It does mean that benchmarks get compiled into tests but that's generally a small price to pay considering that it does eliminate the benchmark support code for external users. Probably the best that can be done at this time until Cargo/Rust improve the benchmarking ecosystem.

[vlovich]

Here's all I needed to change to make it work:

Cargo.toml:

[dependencies]
divan = { version = "0.1", optional = true }

[dev-dependencies]
my-crate = { path = ".", features = ["internal-benchmarks"] }

[features]
internal-benchmarks = ["dep:divan"]

lib.rs:

#[cfg(feature = "internal-benchmarks")]
pub const CRATE_USED: bool = true;

#[cfg(feature = "internal-benchmarks")]
mod benchmarks {
  #[divan::bench]
  fn my_benchmark() {}
}

in benchmark main:

fn main() {
    let _ = memcache::CRATE_USED;
    divan::main();
}

Not the best setup because it forces dependencies needed for running the benchmark itself to be in the [dependencies] section (I mitigate that impact by making them optional & only imported by the internal-benchmarks feature.

[Easyoakland]

@vlovich In my tests it seems that you don't need to create the CRATE_USED const. It's enough to use my_crate; or extern crate my-crate; at the top of the file with the fn main() {divan::main()}

[thomaseizinger]

Here is my workaround:

• Make divan an optional dependency
• Put your benchmarks within a benches module that uses cfg(features = "divan")
• Declare the feature in your Cargo.toml:

  [features]
  divan = ["dep:divan", "dep:rand"] # I also needed rand for my benchmarks
  

• Create benches/divan.rs:

  extern crate my_crate; // Ensure benchmarkes aren't optimised away
  
  fn main() {
    divan::main()
  }
  

• Declare benchmark in your Cargo.toml:

  [[bench]]
  name = "divan"
  harness = false
  required-features = ["divan"]
  

Now, running cargo bench --bench divan will alert you that you need to pass the --features divan feature flag which will compile the benchmarks in.

[nvzqz]

@thomaseizinger I believe https://github.com/rust-lang/cargo/issues/1982 would remove the need to pass an explicit feature flag

[thomaseizinger]

@thomaseizinger I believe rust-lang/cargo#1982 would remove the need to pass an explicit feature flag

The feature flag is required because cargo will always compile the library part of the crate without any cfgs like cfg(test). That is why tests in tests/ can't see stuff that is annotated with cfg(test). The same applies to benchmarks.

[nvzqz]

This seems to be same issue as https://github.com/nvzqz/divan/issues/61. See https://github.com/rust-lang/rust/issues/133491 for upstream discussion.

[ryanseipp]

The following was enough to fix it for me on Linux:

# internal_benchmarks/Cargo.toml
# ...

[dependencies]
divan = "0.1.17"
mycrate = {path = '../mycrate', features = ["internal_benches"]}

[[bench]]
name = "internals"
harness = false

// internal_benchmarks/benches/internals.rs
extern crate mycrate;

fn main() {
    divan::main();
}

I'm happy with this, but appreciate your effort in the upstream discussion.

[madsmtm]

You can also do use mycrate as _; if you prefer to avoid extern crate.

The reason why this is needed is because the compiler chooses to not inspect crates passed on the command line if they're unused. This is documented here.

I found https://github.com/rust-lang/rust/issues/64402 discussing this upstream.