A benchmarking framework for user-defined workloads.

[d0cd]

Motivation

As we roll out on-chain execution via finalize, we need to benchmark its performance on "real" workloads. While the definition of a "real" workload is somewhat arbitrary, we do know that we want to bench a variety of programs (deployments and executions). With that in mind, we desire a benchmarking framework with the following characterisitcs:

1. Easy/ergonomic to define benchmarks
2. Minimal overhead in setting up and running benchmarks
3. Reproducibility

Design

This framework takes an opinionated stance in defining and running benchmarks.

A Benchmark

Users can specify new benchmarks by instantiating objects that implement the Benchmark trait.

pub enum Operation<N: Network> {
    Deploy(Box<Program<N>>),
    Execute(String, String, Vec<Value<N>>),
}

pub trait Benchmark<N: Network> {

    fn name(&self) -> String;

    fn setup_operations(&mut self) -> Vec<Vec<Operation<N>>>;

    fn benchmark_operations(&mut self) -> Vec<Operation<N>>;
}

In creating a new Benchmark, the user specifies:

• A unique name.
• A sequence of "setup" operations that are run before actually running the benchmark. The setup operations are specified in batches, where each batch is included in a single block.
• A sequnece of "benchmark" operations that are to be timed.

See synthesizer/benches/benchmarks for some examples.

A Workload

A Workload is a sequence of benchmarks that are run on a VM.

pub struct Workload {
    name: String,
    benchmarks: Vec<Box<dyn Benchmark<Testnet3>>>,
    object_store: ObjectStore,
}

impl Workload {
    pub fn new(name: String, benchmarks: Vec<Box<dyn Benchmark<Testnet3>>>) -> Result<Self> {...}

    pub fn add(&mut self, benchmark: Box<dyn Benchmark<Testnet3>>) {...}

    pub fn name(&self) -> &String {...}

    pub fn setup<C: ConsensusStorage<Testnet3>>(&mut self) -> (VM<Testnet3, C>, PrivateKey<Testnet3>, BenchmarkBatch, TestRng) {...}
}

Users construct a Workload with a unique name with the relevant benchmarks using new.

A workload is run by first invoking the setup function. setup will check the object_store whether or not precomputed workload objects (blocks containing setup transactions and the benchmark transactions) are stored.

If they are not stored, they are initialized NOTE: initialization may take a long time depending on the size of the workload.

Once the workload objects are initialized, an RNG is instantiated from a stored seed, a VM is set up with the blocks containing the setup transactions, and a triple of the VM, benchmark transactions, and RNG are returned for further use.

The ObjectStore

The ObjectStore is a key component that enables caching workload objects for efficient setup and reproducibility.

pub struct ObjectStore {
    root: PathBuf,
}

impl ObjectStore {
    pub fn new(root: PathBuf) -> Result<Self> {...}

    pub fn get<O: FromBytes, P: AsRef<Path>>(&mut self, path: P) -> Result<O> {...}

    pub fn insert<O: ToBytes + FromBytes, P: AsRef<Path>>(&mut self, path: P, object: &O) -> Result<()> {...}

    pub fn contains<P: AsRef<Path>>(&self, path: P) -> bool {...}

    pub fn clear(&mut self) -> Result<()> {...}
}

The ObjectStore acts as a key-value store using the filesystem. NOTE: the ObjectStore can arbitrarily change the contents of the root directory.

Considerations

Although this PR is designed with benchmarking in mind, a Workload and and ObjectStore can be used to construct sets of blocks and transactions for other forms of testing. For example, one can define a large number of transactions to fire at the network for load testing. Another is to pre-construct a number of blocks to initialize the VM into a complex state.

Usage

This PR introduces benchmarks for VM::finalize and VM::add_next_block. To run these benchmarks, run

cargo bench --features=testing

To run these benchmarks with the RocksDB backend, run

cargo bench --features=testing,rocks

If you'd like to avoid the time it takes to initialize the workloads (~6hrs), feel free to contact me directly for .zips of the workload. Though this method isn't fully tested, I've been able to avoid setup on my secondary machine by copying the files over.

Results

The results of benchmarking finalize can be found here. Note the benchmark names indicate what is tested, but can be hard to parse. Please feel free to ask for clarification.

Future Work

• Reduce time it takes to produce fee records by increasing split width (use external call to credits.aleo).
• Add "accounts". Each operation is associated with a different private key. Allows specifying more complex interactions in a benchmark.
• Support dependent program interactions in a benchmark.
• Improve ObjectStore interface for other uses.