Keep running all tests in clarinet, even if one fails

[gregorycoppola]

trafficstars

Current State

Right now, when running clarinet test, it stops after the first test failure (at least in some cases).

E.g., in the hirosystems/stacks-hyperchains repo, with many tests, it will stop after the first test if it has an error, and will not run the rest of the tests.

~/subnet2/core-contracts$ clarinet test
Running file:///home/greg/subnet2/core-contracts/tests/hyperchains/hyperchains_test.ts
* Ensure that block can be committed by subnet miner ... FAILED (20ms)

failures:

Ensure that block can be committed by subnet miner
Error: Expected 3, got 2
    at consume (https://deno.land/x/[email protected]/index.ts:467:11)
    at String.expectInt (https://deno.land/x/[email protected]/index.ts:532:5)
    at Object.fn (file:///home/greg/subnet2/core-contracts/tests/hyperchains/hyperchains_test.ts:53:14)
    at fn (https://deno.land/x/[email protected]/index.ts:271:23)
    at asyncOpSanitizer (deno:runtime/js/40_testing.js:21:15)
    at resourceSanitizer (deno:runtime/js/40_testing.js:58:13)
    at exitSanitizer (deno:runtime/js/40_testing.js:85:15)
    at runTest (deno:runtime/js/40_testing.js:199:13)
    at Object.runTests (deno:runtime/js/40_testing.js:244:13)
    at file:///home/greg/subnet2/core-contracts/$deno$test.js:1:27

failures:

    Ensure that block can be committed by subnet miner

Request

Consider to keep running all the tests that initially matched the filter, even after the first failure.

This way, we can know how many of the tests are broken without doing things like manually deleting tests.

[moodmosaic]

👍

Clarinet tests are essentially Deno tests. Perhaps that's also how deno test works? (I'm not in front of my laptop to verify this theory, just a thought.)

[lgalabru]

To be addressed in https://github.com/hirosystems/clarinet/pull/511.