casper-node icon indicating copy to clipboard operation
casper-node copied to clipboard

Add instrumentation to the test suite

Open mpapierski opened this issue 1 year ago • 3 comments

For each exec() call in the test suite, this PR attaches instrumentation data that will uniquely identify a given run, together with duration, gas_cost, and error and save this data into a .csv file that can be used for data analysis. This way, after modifying opcode cost table there's a way to measure real impact across all the exec runs.

Usage:

builder.exec_instrumented(exec_request, instrument!());

instrument! macro accepts extra (key, value) pairs that will be also appended to the CSV i.e. in case there's a for loop with builder.exec calls:

for i in 0..100 {
  let exec_request = make_request(i);
  builder.exec_instrumented(exec_request, instrumented!(i));
}

This should append additional fields to the CSV row: i, 0 etc.

Sample output in execution_results.txt:

module_name,file,line,function,duration_ns,gas,error
casper_engine_tests::test::deploy::stored_contracts,execution_engine_testing/tests/src/test/deploy/stored_contracts.rs,161,should_fail_if_calling_non_existent_entry_point,3363,13663671535,
casper_engine_tests::test::deploy::stored_contracts,execution_engine_testing/tests/src/test/deploy/stored_contracts.rs,196,should_fail_if_calling_non_existent_entry_point,283,2500000000,No such method: electric-boogaloo
casper_engine_tests::test::deploy::stored_contracts,execution_engine_testing/tests/src/test/deploy/stored_contracts.rs,645,should_fail_payment_stored_at_hash_with_incompatible_major_version,3076,13663671535,
casper_engine_tests::test::deploy::stored_contracts,execution_engine_testing/tests/src/test/deploy/stored_contracts.rs,696,should_fail_payment_stored_at_hash_with_incompatible_major_version,317,2500000000,Incompatible protocol major version. Expected version 2 but actual version is 1

mpapierski avatar Sep 08 '23 17:09 mpapierski