Add ability to merge multiple reports into one

[mattflix]

Given a Mocha test suite, using some custom Mocha "ui" code and supporting infrastructure that I have written, I have the ability to "shard" the constituent tests of the suite out to N computers, which each computer running only a distinct group of the tests.

Basically, this sharding allows me to a test suite of, say, 100 tests on 1 computer (running all 100 tests, generating a single report) or on, say, 10 computers (each running a distinct group of 10 of the tests, generating 10 reports). The benefit, of course, being that the overall run time should be much shorter when running with multiple computers (with 10 computers, maybe 10% of the time vs. with 1 computer).

However, the downside is that, if each of these computers is running mochawesome, I get 10 different awesome but distinct reports, for each distinct group, none of which gives a complete picture of my overall tests results. If all of the reports could be joined somehow (merged back into one report), then I would once again have a complete picture.

Given this constraint (multiple mochawesome reports, each for a distinct group of tests from the same suite), it would be great if mochawesome offered the ability to merge the results back into a single report.

Even if the constraint was not followed (i.e., some of the computers run some or all of the same tests), the reconciliation behavior for merging the reports could be one of following options:

• Keep only the chronologically first (or last) result (success or failure) result for a given test.
• Keep only the first (or last) failure (or success) for a given test, if any.
• Keep all results for a given test (but rename the runs, perhaps by simply adding "1", "2", etc. to the test name).
• "Fail" the test due to the conflict (i.e., don't allow more than one computer to run the same test, regardless of whether the tests itself succeeds or fails).

Perhaps the behavior could be configurable. There are probably also other reconciliation behaviors I haven't thought of.

[adamgruber]

This is definitely an interesting idea but one with many challenges. To start I think you would need some way to differentiate between reports and test runs and know whether a report belonged to a larger test run or not. Have you given thought to how something like this would be implemented? I'd be happy to offer feedback on a PR but it's not something I have time to tackle myself right now.

[mattflix]

I assume any supporting infrastructure that somehow runs multiple invocations of Mocha, somehow getting them to each run their own "portion" of the overall test suite, would be able to track and collect all the individual reports in order to merge them as part of a single "test run". On the other hand, a feature (in mochawesome) to allow such an infrastructure to supply some sort of "test run" identifier (to be included as part of the generated reports) could be helpful.

Other thoughts...

If you are going to rework the mochawesome report format anyway, then perhaps give some thought to the following issues (even if you don't intend to implement merging yourself):

Make the report structure a bit of more conducive to merging

Objects (maps) are much easier to merge than arrays of "keyed" objects. For example, instead of filling the report with arrays of objects that contain UUID keys, why not use maps? (Where the map keys are UUIDs or other values; see next issue below.) Of course, then you'll have to add some sort of "ordinal" to the mapped objects to establish an order (implicit when using an array), but perhaps timestamps would be better for that (in the face of possible merging)?

Make the content more "stand-alone"

Instead of using (freshly-generated) UUID keys to unique identify tests and suites, why not use stable hashes of the tests and suites? The identifier for a test simply could be a hash of the test's fullTitle and source code text (and so will not change so long as the test suite source remains the same). The hash for a suite could be the hash of all the child test's hashes, etc.. This would make it easier to merge the output of multiple Mocha invocations (which would otherwise have generated unrelated and un-joinable UUIDs in their individual reports). Using hashes would make it possible to uniquely identify suites and tests, even when run in different Mocha instances.

Compute stats while rendering instead of pre-computing them in the report

The current report format seems to contain lots of pre-computed stats and counts, which significantly complicates the job of a merger, which would basically need to duplicate the stat-computing logic of mochawesome. Instead, if the report only contained the basic "raw" information (the suites, tests, hooks and the individual results for each), and the mochawesome report UI renderer computed the stats "just-in-time", then merging could seemingly be a much simpler affair.

[adamgruber]

Thanks for the detailed comment/explanations. Some thoughts...

Make the report structure a bit of more conducive to merging

Can you show me an example what this might look like?

Make the content more "stand-alone"

The idea of using unique hashes is pretty interesting. So I'm clear, this hash would take the place of the UUID correct? Also, what is the risk of collision? Or if for some reason there are identical tests would that produce identical hashes? Again, some examples here would be appreciated.

Compute stats while rendering instead of pre-computing them in the report

I have begun work to slim down the JSON output. This is the PR where I made changes to eliminate data. It includes removing stats that are computable during render. Would you suggest any other changes to trim the data?

[adokania]

I am running into the same issue, Is it easy to add an 'append' option where instead of overwriting and replacing the existing json, it adds to the json file. I can see this being difficult cause of the parallel nature and all updating the same file at once, But just a thought I had so putting it in here

[ethangodt]

I'm working on this right now as well. Currently I'm just going to rely on a cli tool I'm writing myself that will merge the reports. It's not trivial 😦. Once I make some more progress I'd be happy to share the things I had to consider for my use case.

[adamgruber]

It's not trivial

@ethangodt Yeah, this is kind of why I've been focused on other improvements. I understand the value in this feature but when you start to break it down there's a lot of issues that come up. I'd definitely be interested in what you come up with.

[mattflix]

(I initially filed this issue.)

I have already gone ahead a written a simple merger for my purposes. It doesn't do nearly all the things I suggested in my original post. Yes, it was not trivial, and could have been much simpler, but it wasn't too bad. It's just that the current report format isn't really designed to help with merging.

So, I stand by my original suggestion that, if the report generated by mochawesome was much simpler and more basic (literally, nothing other than test/hook results), and the "evaluation" of the results was done perhaps only by the Javascript baked into the page, it would be much easier to merge multiple reports.

As the report format currently stands, you must traverse many objects of various shapes, with similarly-named, but sometimes differently-used, fields, and apply various algorithms to either sum values together, take the minimum or maximum value, logically OR-ing and AND-ing flags, etc. -- all of which surely duplicates some of the "evaluation" logic that should just be left to the final presentation (UI).

[adamgruber]

@mattflix I had hoped to get some more feedback from you on this so it could inform the direction of the 3.0 branch. In any case I'd be interested to see what your final solution looks like. Would be good to share so others might benefit.

[callumlocke]

@mattflix I am also interested to see what you came up with

[phenomnomnominal]

I too, would love to see more info about this

[fsmaia]

Could we accept a [file] wildcard in reporterFilename, to have at least predictable filenames?

[mrbar42]

this would indeed be a great feature.

meanwhile i ended up merging reports with some naive monkey patching. although it doesn't merge all of the possible cases - it generates one big report from all of my partial reports which in turn renders correctly in the html page. might do the trick for you as well and or serve as a seed for such feature

monkey patch merging

Written in node v8+ flavor

'use strict';

const fs = require('fs-extra');

/**
 *
 * @param {object} options
 * @param {string} options.source - path to a source json file that contains partial results
 * @param {string} options.target - path to target json file to combine all reports to
 *
 * @example
 * const {mergeMochaAwesomeReports} = require('./mocha-awesome-merger');
 *
 * mergeMochaAwesomeReports({
 *   source: './output/mocha-awesome.json',
 *   target: './output/mocha-awesome-combined.json'
 * });
 *
 */
exports.mergeMochaAwesomeReports = function mergeMochaAwesomeReports({target, source}) {
  if (!fs.existsSync(target)) {
    fs.copySync(source, target);
    return;
  }

  const sourceJson = fs.readJsonSync(source);
  const targetJson = fs.readJsonSync(target);

  targetJson.suites.suites.push(...sourceJson.suites.suites);
  mergeStats(targetJson.stats, sourceJson.stats);
  mergeArrays(targetJson, sourceJson);

  fs.writeJsonSync(target, targetJson);
};

function mergeArrays(target, source) {
  Object.keys(source).forEach(key => {
    const value = source[key];
    if (!Array.isArray(value)) return;

    target[key].push(...value);
  });
}

function mergeStats(target, source) {
  Object.keys(source).forEach(key => {
    const value = source[key];
    if (key === 'start') {
      // do nothing
    }
    else if (key === 'end') {
      target[key] = value;
    }
    else if (typeof target[key] === 'number') {
      target[key] += value;
    }
    else {
      target[key] = value;
    }
  });

  target.hasSkipped = Boolean(target.skipped);
  target.hasOther = Boolean(target.other);
}

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.