Triggering failed benchmarks

[mitermayer]

I was wondering if there would be a way to use this in the form of

• compare foo with bar, and if foo is 10% slower then bar fail the suite.

anothe useful way would be to fail a suite if performed less then x operations per second.

[JamieMason]

Thanks @mitermayer. It's been a long while since I've worked on karma-benchmark, but once I've finished what I'm doing on shrinkpack and ImageOptim-CLI I plan to pick this back up again in a couple of months.

[JamieMason]

Should be able to pick this up in the next couple of weeks @mitermayer and have a couple of questions.

compare foo with bar, and if foo is 10% slower then bar fail the suite.

Could you add some more detail on what this might look like? eg. what karma.conf.js and/or your benchmarks might look like to use.

In this example:

suite('Array iteration', function() {
  benchmark('_.each', function() {
    _.each([1, 2, 3], function(el) {
      return el;
    });
  });

  benchmark('native forEach', function() {
    [1, 2, 3].forEach(function(el) {
      return el;
    });
  });
});

Are you suggesting that if eg native forEach is 10% slower than _.each it should fail? Could you give me some examples of what kinds of things you'd be working on when doing this? I'm a bit stuck on this one.

anothe useful way would be to fail a suite if performed less then x operations per second.

Are you thinking here of an option in karma.conf.js which applies to all benchmarks, if any single benchmark doesn't pass that value, mark it as a failed test?

Thanks.

[JamieMason]

Thinking about it some more (and correct me if I'm wrong, but) I think triggering failures is normally handled by reporters – @rawrmonstar is this something you might consider adding as an option to FormidableLabs/karma-benchmarkjs-reporter?

Thanks all.

[rawrmonstar]

@JamieMason I'll add an issue, but I'm not sure when I can get around to it.