Sending custom data to reporter

[oreporan]

What is your Test Scenario?

We are using testcafe to measure performance regressions. Our test case does some setup, tests something, and makes sure its under some X time. I would like to send the time it took (currently just a console.log) to our custom reporter so we can track this in a graph and see how performance increases/decreases over time.

Currently this is not possible, the closest solution i saw was to use meta like here but I need custom information after the test has passed, not before.

What are you suggesting?

Some way to pass data, either by log or by the test object, into my reporter. The time of the test is not good enough because it includes the setup/teardown of the test.

What alternatives have you considered?

Right now I need to log the results to a file, and pick them up in another step of my CI disconnected completely from testcafe.

[sijosyn]

I would also like to send custom data to reporter. My use-case scenario is as follows:

I grouped bunch of test-scenarios together in one single test function (this is done in this way for test execution time saving purpose). Each test-scenario is considered to be a single test case and has associated testcase ID in TestRail test management system (https://www.gurock.com/testrail) and I use TestRail API in reporter plugin (https://www.npmjs.com/package/testcafe-reporter-html-testrail) to update the status of the test cases. Since I grouped the tests, I want to be able to capture the status during course of the test execution and make that info available in test reporter. For e.g

test("My Test", async (t) => {
    // Test scenario for caseID_123
    await t.click(<DOM element found>);
    await t.report({caseID_123: "PASS"},/*  set the status of  caseID_123 as PASS*/);
  
    // Test scenario for caseID_234  
    await t.click(<DOM element found>);
    await t.report({caseID_234: "PASS"},/*  set the status of  caseID_234 as PASS*/);
   
    // Test scenario for caseID_456
     await t.click(<DOM element not found>);
     await t.report({caseID_456: "FAIL"},/*  set the status of  caseID_456 as FAIL or NOT TESTED */)
});

I don't think meta helps. So how can we send custom data to reporter?

[AlexKamaev]

Currently, TestCafe does not have such functionality. However, I think it'll be useful. I discussed the issue with the team and we decided that since we already have meta in reporters, we can add some API for modifying meta at runtime. I marked this issue as enhancement, however, I cannot give you estimates as to when it will be implemented. Of course, your PR will be welcomed.

[sivanisentinel]

Was this enhancement implemented? I really need this feature

[Dmitry-Ostashev]

Currently, this feature is still not implemented, but we keep it in mind. You are welcome to submit your PR and contribute to our project.

[danlomantoSamsungNext]

I'd love to try and create a PR for this. Is there any direction or tips I should know before starting? Digging into a completely new codebase can get confusing sometimes. :)

[AndreyBelym]

@danlomantoSamsungNext, I think we need to discuss API extensions first. From what it seems to me, simply adding t.log(...args) that will handle its arguments similarly to console.log and append log lines to test reports will be enough. To implement such function, we need to introduce changes to TestController and Reporter subsystems, plus implement support for this feature for in all reporter packages. I'm afraid this feature is too big to get started with contributing to the TestCafe codebase.

[danlomantoSamsungNext]

@AndreyBelym I think people are looking for more than just a t.log(...args) ability that's similar to console.log(). I believe there were discussions for using t.meta(key, value) so that people could save different kinds of test information during a test run that would be accessible via a reporter. There are a couple of referenced tickets that demonstrate this.

I believe the thinking was to use the meta() functionality because it's already accessible when building a custom reporter as well.

[AndreyBelym]

@danlomantoSamsungNext yes, I've got your point. I discussed it in private with some other team members, and we agreed that it is easier to add metadata modification functions to the test controller. What do you think about such API for this feature:

type Meta = { [name: string]: string };

interface TestControllerMetaExtensions {
    /**
     * Returns the test metadata as an key-value object.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     console.log(t.meta);
     * });
     * // { 'foo': 'bar' }
     */
    readonly meta: Meta;

    /**
     * Adds an additional metadata entry to the test metadata.
     * If an entry with the specified name already exists, overrides its value.
     * @param name -  The name of the metadata entry to be added.
     * @param value - The value of the metadata entry to be added.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     await t.addMeta('ans', '42');
     *     console.log(t.meta);
     *     await t.addMeta('foo', 'foobar');
     *     console.log(t.meta);
     * });
     * // { 'foo': 'bar', 'ans': '42' }
     * // { 'foo': 'foobar', 'ans': '42' }
     */
    addMeta(name: string, value: string): TestController;

    /**
     * Adds an additional metadata entries to the test metadata.
     * If an entry with the specified name already exists, overrides its value.
     * @param meta -  A set of metadata entries to be added.
     * @example
     * test.meta('foo', 'bar')('test', async t => {
     *     await t.addMeta({ 'foo': 'foobar', 'ans': '42' });
     *     console.log(t.meta);
     * });
     * // { 'foo': 'foobar', 'ans': '42' }
     */
    addMeta(meta: Meta): TestController;

    /**
     * Removes metadata entries with the specified names from the test metadata;
     * @param names - A list of metadata names to be removed
     * @example
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta('foo');
     *     console.log(t.meta);
     * });
     * // { 'ans': '42' }
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta('foo', 'ans');
     *     console.log(t.meta);
     * });
     * // { }
     * test.meta({ 'foo': 'bar', 'ans': '42' })('test', async t => {
     *     await t.removeMeta(['foo', 'ans']);
     *     console.log(t.meta);
     * });
     * // { }
     */
    removeMeta(...names: (string | string[])[]): TestController;
}

[danlomantoSamsungNext]

@AndreyBelym That looks absolutely fantastic to me!!!! Thank you so much for digging into that!

[sivanisentinel]

It seems to me like this solution is “abusing” meta data. It is strange that this is the only way to send data to the report. I feel like we need dedicated api for that. I also thinks that we should have the option to send general data to the test and not only fixture or test related one. For example if I want to send environment details or username who triggered the run, do we really have to send it in all fixtures again and again?

On 31 Jul 2019, at 15:51, Dan Lomanto [email protected] wrote:

@AndreyBelym https://github.com/AndreyBelym That looks absolutely fantastic to me!!!! Thank you so much for digging into that!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/DevExpress/testcafe/issues/3584?email_source=notifications&email_token=AENHQ2HBHMNMIMG42KZRLDDQCGDDJA5CNFSM4G7QQYGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3HELHI#issuecomment-516834717, or mute the thread https://github.com/notifications/unsubscribe-auth/AENHQ2FW2YQPHM7X5Y3UXM3QCGDDJANCNFSM4G7QQYGA.

[danlomantoSamsungNext]

@sivanisentinel I think I see your point, but I think your request is a bit different than the original intention of this ticket.

I think it would be good to have some general way to associate data/info to an entire test run, so that you don't have to repeat yourself in all your beforeEach() and afterEach() methods (ie. Test Environment, User Info, etc.). I think this would be a different enhancement though.

The reason why I think using the meta() capability is valid here is because the data/info I'd like to set it truly test-specific data/info that I'd like to have access to in a custom reporter.

[AndreyBelym]

@danlomantoSamsungNext thank you for warming words :)

@sivanisentinel I also thought about this way first, as you can see in my comment. I would like to see it implemented someday, but we need to invest a lot more time and powers into it.

[danlomantoSamsungNext]

I see that this is marked as "Planned". Does that mean, by any chance, that this has a potential release date? Or an estimate on when this might be worked on? Thanks!!!!

[miherlosev]

The "Planned" milestone means that the TestCafe team has plans to implement this feature someday. At present, there is no precise time estimation for that. If the feature is very important for you, the pull request would be appreciated.

[Evilweed]

@AlexKamaev this feature is in Your best interest as having this functionality allows creating much powerful Test Reporters like the one that prints Error Stack, URL of the website during error, and console errors from the browser, which is currently impossible to do. Debugging errors that occured on CI on complicated websites, without this information is much harder.

[AndreyBelym]

@Evilweed error stacks should be printed with all built-in reporters for both server-side and client-side errors automatically by TestCafe. If there is no error stack in a report, likely it means that your browser or Node.js failed to capture it.

To properly integrate reporting current URLs and browser console logs in reports, we should also solve a problem with delivering this data consistently from a browser to the TestCafe. Otherwise, it will be not a big difference from using console.log and TestCafe APIs like ClientFunction and getBrowserConsoleMessages that can work unreliably in cases of fatal errors on a tested page.

[Evilweed]

My current solution which is rarely used by any developer because it is not presented in Allure report and it's not presented as a part of console error.

Forgot to wrap getBrowserConsoleMessages in try/catch.

// Try catch was added as a precaution in order not to break Test Runner
// TODO(m) Create out of this proper Test Reporter plugin to Test Cafe
export async function printInformationFromBrowsersConsole(t: TestController): Promise<any> {
    const { log, warn, info, error } = await t.getBrowserConsoleMessages();
    let url = "";
    let testName = "";
    let fixtureName = "";
    try {
        url = await getUrl();
    } catch(e) {}

    try {
        // Do not remove - those objects are not defined in typings,
        // and can potentially break in future Test Cafe versions.
        // @ts-ignore
        testName = t.testRun.test.name;
        // @ts-ignore
        fixtureName = t.testRun.test.fixture.name;
    } catch(e) {}

    if ((log && log.length > 0) ||
        (warn && warn.length > 0) ||
        // (info && info.length > 0) ||
        (error && error.length > 0)) {
        console.log("\n");
        console.log("--------------------------------");
        console.log(`Test information:`);
        console.log("--------------------------------\n");
        console.log(`Fixture: ${fixtureName}`);
        console.log(`Test: ${testName}`);
        console.log(`🌍 Page URL: ${url}`);

        printEachIfExists(LogType.LOG, log);
        printEachIfExists(LogType.WARNING, warn);
        //printEachIfExists(LogType.INFO, info);
        printEachIfExists(LogType.ERROR, error);

        console.log("--------------------------------");
    }

}

[Farfurix]

One more request: https://github.com/DevExpress/testcafe/issues/4370.

[ghost]

Hello ! I am also very interrested in adding personnalized info in my reporter. The addMeta method would suit my needs I think.

So do you have any news on this functionnality ? Any (rought) idea on when it will be implement ? (So I can choose whether to wait for it, or spend time to find a suitable workaround)

[Ogurecher]

@KouraiBee, We haven't yet started implementing this functionality, and we cannot give you any estimates as to when we start working on this. In the meantime, I suggest that you use one of the workarounds outlined in this thread.

[wentwrong]

https://stackoverflow.com/questions/65364252/testcafe-report-the-page-html-on-any-assertion-failure One more feature request. Use case: provide the HTML code of a page when the assertion is failed.

[miherlosev]

https://stackoverflow.com/questions/65191797/testcafe-can-you-pass-ctx-context-variables-to-reporter

[AlexGrisin]

Hello! Do you have any updates on this issue ?

[github-actions[bot]]

No updates yet. Once we get any results, we will post them in this thread.

[kPepis]

+1 on this. This would be really helpful.

[Jgrabenbauer]

This would be very helpful with passing different errors to the reporter (without failing the tests) so that they can be reviewed locally and in CircleCI.

[miherlosev]

Hi @Jgrabenbauer,

This would be very helpful with passing different errors to the reporter

Could you please clarify what kind of errors you want to pass on to the reporter? We have plans to collect browser console errors and request errors and notify customers about them. So, if you want to pass generic errors, we will consider adding them to the report.

[Jgrabenbauer]

Hey @miherlosev what we are mainly looking for is to enhance Testcafe’s reporter options to allow the user to implement an EventEmitter that will print a message of the failed request(Promise). This will allow TestCafe to finish running the tests while letting the users see other failures that happened during the run (400s and 500s). It would be nice if this was an option in the .testcaferc file as well as the fixture and test level.

[miherlosev]

@Jgrabenbauer

Thank you for sharing the detailed information about your use case.

[JESii]

Given that this issue has been open since 2019 with many supporting comments, I have to assume that there really is no intent to implement it. In the hope that I am wrong, I'll add my use case: this is for an investment reporting app.

1. Clients are offered a view of their portfolios, with assets grouped into various categories.
2. The app offers a "current month" view, with the ability to switch to previous month's data -- called AsOfDates.
3. Within each monthly view, the data is organized into various periods; e.g., CYTD, FYTD, 1Year, 3Years... etc each of which offers a view of the portfolio over the respective time period.
4. There are numerous graphs throughout the app, with different display specs for the graph type (line, bar, ...): for example how many x-axis points there are for each period and how they are labelled.

I have a working TC regression test that: loops thru multiple clients; loops through the AsOfDates; loops through the available Periods; and examines the various graphs to ensure that the x-axis data is presented according to spec.

Obviously, there are many possible points of failure in this process, but in the event of a failure I want to simply write a message documenting the failure and continue to the end of the test.

When the test completes, I want to collect any error information and report it to the TC reporter. There was a suggestion https://github.com/DevExpress/testcafe/issues/5563 made some time ago to provide a mechanism to explicity cancel a test and provide some information but that was closed last year.

I just have to say that, from our perspective, this is a huge omission and one that will count heavily against TestCafe when we are considering a testing framework for future client implementations.