Measuring test vector coverage

[Stebalien]

It would be really nice to measure what parts of actors are covered by our current test vectors.

[dtynn]

so basically speaking:

1. tracing & collecting profile data from inside the wasm runtime and make the data human-readable
2. find some way to enable the tracing only in dev mode (tests & benchmarks)

for step 1,

• WebAssembly Code Coverage
• Code Coverage in wasm-bindgen-test?
• minicov may be helpful.

is that right?

[dtynn]

hi @Stebalien would like to know if wasm-coverage-demo is helpful for this issue ?

[dtynn]

completed a simple demo,

and captured a coverage report for executing test-vectors/corpus/extracted/0005-chocolate-01/fil_6_storagemarket/WithdrawBalance/Ok/ext-0005-fil_6_storagemarket-WithdrawBalance-Ok-1.json.

[Stebalien]

tracing & collecting profile data from inside the wasm runtime and make the data human-readable

Yes and yes.

find some way to enable the tracing only in dev mode (tests & benchmarks)

Also yes. We'll have to have some way to compile actors with tracing enabled.

[Stebalien]

completed a simple demo,

That's awesome!

Note: We probably don't need the human readable side of things. As long as the format is compatible with codecov.

@raulk is currently in the process of factoring out the actors into a separate repo (with a single wasm build script so this should be a bit simpler). Once that lands, are you up for making a PR?

[dtynn]

Note: We probably don't need the human readable side of things. As long as the format is compatible with codecov.

sure, the reults could be the compatible format, or at least they should be easily converted to, since all the things are generated by llvm-cov

@raulk is currently in the process of factoring out the actors into a separate repo (with a single wasm build script so this should be a bit simpler). Once that lands, are you up for making a PR?

of course :)