🚀 Feature: Re-run a test multiple times: `this.repeat`

[guyb7]

this.retries() allows to run a test multiple times, but the test will pass on the first successful run.

I'm suggesting to add this.rerun() for tests that might randomly succeed (false positives). It will run the test n times and will pass only if all of the reruns have passed successfully. It's useful when testing behavior that is probabilistic.

At the moment I'm running the test in a loop:

for (var i = 1; i <= 100; i++) {
  it('tests something #' + i, function(done) {
    //
  });
}

The problem with this approach is that it outputs 100 different tests and messes up the test counting.

[boneskull]

Hmm. What's the problem with putting the loop inside your test?

[guyb7]

It's just not as convenient - I have to manually call beforeEach() on each iteration and call the next iteration (and call done() on the last one).

Compare this: (can be improved)

describe('#multiple-runs', function() {
  var beforeEachFunction = function() {
    // Code
  };

  beforeEach(beforeEachFunction);

  it('runs 15 times', function(done) {
    var recursive_test = function(n) {
      if (n > 15) {
        done();
        return;
      }
      beforeEachFunction();
      // Code
      recursive_test(n + 1);
    }
    recursive_test(1);
  });
});

With this:

describe('#multiple-runs', function() {
  beforeEach(function(){
    // Code
  });

  it('runs 15 times', function(done) {
    this.rerun(15);
    // Code
  });
});

[ghost]

i think a better sintax could be something like this

it.rerun(15, 'runs 15 times', function(done) {
   //Code
});

[fantasyRqg]

did this function realize ?

[Bamieh]

@fantasyRqg no.

[plroebuck]

No. But in case someone wants to mess with it, here's some code that does testing manually. Tests if a random integer is divisible by subset of prime numbers over a series of 5 runs.

Save the following as "multirun.spec.js".

var debuglog = require('util').debuglog('multirun');

describe('#multiple-runs', function () {
  var NRUNS = 5;

  // Returns vector of prime numbers (up to max) [Sieve of Eratosthenes]
  function getPrimes(max) {
    var primes = [], sieve = [];

    for (var i = 2; i <= max; ++i) {
      if (!sieve[i]) {
        // i has not been marked -- it is prime
        primes.push(i);
        for (var j = i << 1; j <= max; j += i) {
          sieve[j] = true;
        }
      }
    }
    return primes;
  }

  // Returns random integer (up to max)
  function getRandomInt(max) {
    return Math.floor(Math.random() * Math.floor(max));
  }

  // Randomize array element order in-place. [Durstenfeld]
  function shuffle(array) {
    for (var i = array.length - 1; i > 0; i--) {
      // Pick a remaining element
      var j = Math.floor(Math.random() * (i + 1));

      // Swap it with current element
      var temp = array[i];
      array[i] = array[j];
      array[j] = temp;
    }
    return;
  }

  //=========================================================
  it('should succeed ' + NRUNS + ' times', function (done) {
    (function _suite(nruns) {
      var candidate, primes, somePrimes;

      function _suiteSetup() {
        var MAXPRIME = 31;
        primes = getPrimes(MAXPRIME);
        debuglog('primes[' + primes.length + '] =', primes, '(orig)');

        // :NOTE: Remove '2' from list to improve odds of test success
        primes = primes.slice(1);
      }

      function _setup() {
        var MAX = 100;
        candidate = getRandomInt(MAX);

        // (Re)shuffle primes
        shuffle(primes);
        debuglog('primes[' + primes.length + '] =', primes, '\n');

        // Use first NKEEP prime #s for test
        var NKEEP = 3;
        somePrimes = primes.slice(0, NKEEP);
        debuglog('somePrimes[' + somePrimes.length + '] =', somePrimes);
      }

      // Code being tested [throw to indicate failure]
      function _test(run) {
        somePrimes.forEach(function (prime) {
          debuglog('candidate:', candidate + ', prime:', prime);
          if (candidate % prime === 0) {
            throw new Error('' + candidate + ' is divisible by ' + prime);
          }
        });
      }

      function _teardown() {
        candidate = undefined;
        somePrimes = undefined;
      }

      try {
        _suiteSetup();
        for (var n = 0; n < nruns; n++) {
          debuglog('==========');
          _setup();
          _test(n);
          _teardown();
        }
      }
      catch (err) {
        return done(new Error('run ' + (n+1) + ' failed: ' + err.message));
      }

      return done();
    })(NRUNS);
  });
});

Execute it, expecting both failure/success:

$ mocha multirun.spec.js

Same as above, with verbose output.

$ NODE_DEBUG=multirun mocha multirun.spec.js

[AidanWelch]

This is also useful for testing that the first run didn't break any future usages of whatever your testing without requiring a restart.

[sriharikapu]

I am facing the similar issues with tests as well cant run the same test multiple times

[shunjizhan]

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

[TierK]

Any date prognosis of realization?

[kberg]

one scenario this can be useful is that when I suspect a test to be flaky, instead of modifying the code to re-run it multiple times, I can just use this command to rerun a test many times to test it's flakiness

This is what I'm facing. I'll just wrap it in a shell script and run it a couple hundred times.

[mmomtchev]

Does anyone know if there is any reason why this hasn't been implemented besides no one actually doing it?

My use case is testing for leaks in C++ modules.

[mmomtchev]

npm i mocha@github:mmomtchev/mocha#repeats

We will see if we will be able to get this into mocha.

[JoshuaKGoldberg]

I like this.repeats as implemented by #5011, personally. It has a nice symmetry with the existing this.retries. 🙂

cc @mochajs/maintenance-crew - would like to hear from another maintainer before deciding anything.

[mmomtchev]

@JoshuaKGoldberg In case you are wondering, judicial corruption come first before any Github PRs. That's how the pyramid of Maslow goes.

[kberg]

@JoshuaKGoldberg In case you are wondering, judicial corruption come first before any Github PRs. That's how the pyramid of Maslow goes.

What does judicial corporation have to do with this?

[mmomtchev]

I have been living in a total isolation for the last 4 years since an employer covered up a huge judicial scandal involving sex and corruption. I have been working on lots of opensource projects during this time while living on social welfare. The situation has been worsening ever since, with many of my PRs initially being blocked - after which I am signaled by simultaneous comments that if I was to shut up about the affair, they will be willing to merge them. I am also offered a job - which includes a condition of a sexual nature - if I accept to not talk about the affair anymore. @JoshuaKGoldberg's employer is one of the companies involved and he posted this simultaneously with a few other pings.

PS. Just as your comment, too.

[voxpelli]

Very sorry to hear about that @mmomtchev. @JoshuaKGoldberg himself is an independent freelancer and has no employer. And we're all here to make Mocha the best possible project. Judicial corruption and similar is out of scope of this issue and while that does not at all diminish the importance of such issues, they are better dealt with in other channels 🙏

[mmomtchev]

@voxpelli it is not, since I was clearly shown that these two subjects were in fact related. It is a condition that I have been asked in a significant number of open-source projects on Github during the last few years and my only answer can be no. Should anyone have any interest in pushing this PR forward, I am willing to contribute. Should there by any other conditions, my answer is negative.

[kberg]

I like this.repeats as implemented by #5011, personally. It has a nice symmetry with the existing this.retries. 🙂

cc @mochajs/maintenance-crew - would like to hear from another maintainer before deciding anything.

I'm not a maintainer but this.repeats looks good.

[JoshuaKGoldberg]

Talked with @voxpelli: we think this is a good idea and worth having in. Marking as accepting PRs (i.e. #5011). ✅

Note that per #5027 our main focus is on maintenance rather than features. This particular feature goes a bit against our maintenance goals... but we think it's a straightforward + useful enough edition to justify it.

...and it doesn't hurt that #5011 is a wonderfully written PR (docs! tests! 😍!).

[kberg]

That's great, thank you.

[JoshuaKGoldberg]

Marking as semver-major because this changes output formats and generally touches a lot of areas of code. Even if it might technically be a semver-minor change in some interpretations, we're trying to be careful.