Take screenshot when a test fails

[gregberge]

As suggested in #43, we would like to have a screenshot when a test fail, it is the same idea as #130.

Technical detail

• The screenshot would be created in a .jest-puppeteer/screenshots folder.
• The name of the screenshot would be ${fileName}-${describe}-${it}-${timestamp}.png

We could redefine it and test but I think this is not the cleanest method. Any help from Jest team to do it properly is welcome @SimenB.

[SimenB]

This is exactly the same as #130 (from Jest's perspective), right? Whether you take a screenshot or not shouldn't matter, as long as you can intercept the error reporting. Unless I'm misunderstanding?

[testerez]

Here is how I did it:

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  jasmine.getEnv().addReporter({
    specDone: async result => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

[elmolicious]

+1 for a much needed feature

[kevlened]

The jest-screenshot-reporter handles screenshots the way @testerez shows. That reporter currently assumes browser uses the webdriver api, but the logic is simple, so you could write something similar for puppeteer or add a contribution there.

[jacobweber]

@testerez Can you tell me where you're calling that code from? I was trying to do it the way jest-screenshot-reporter does, but it seems like the page gets closed before I can take the screenshot.

[testerez]

@jacobweber I just call registerScreenshotReporter() in setupTestFrameworkScriptFile (I'm not using jest-puppeteer anymore)

[jacobweber]

@testerez Thanks! Got it working.

[ricwal-richa]

@testerez On using your custom reporter, I get an error that takeScreenshot is not a valid method. I am using jest-puppeteer so I searched for this method in jasmine2 (as jest-jasmine2 used in jest-puppeteer) but could not find a reference. Can you please tell me if jasmine reporter would also need browser/page handle explicitly for this method to work ?

eg. browser.takeScreeshot() or page.takeScreenshot

I am able to make this work with puppeteer screenshot method (using that in custom defined method); but would prefer to use the in-built support for screenshot available with jasmine reporter, if any.

[testerez]

@ricwal-richa yes, takeScreenshot was not part of my code sample so it's normal you got this error. Here is the full code I use (this is typescript):

import path from 'path';
import mkdirp from 'mkdirp';

const screenshotsPath = path.resolve(__dirname, '../testReports/screenshots');

const toFilename = (s: string) => s.replace(/[^a-z0-9.-]+/gi, '_');

export const takeScreenshot = (testName: string, pageInstance = page) => {
  mkdirp.sync(screenshotsPath);
  const filePath = path.join(
    screenshotsPath,
    toFilename(`${new Date().toISOString()}_${testName}.png`),
  );
  return pageInstance.screenshot({
    path: filePath,
  });
};

export const registerScreenshotReporter = () => {
  /**
   * jasmine reporter does not support async.
   * So we store the screenshot promise and wait for it before each test
   */
  let screenshotPromise: Promise<any> = Promise.resolve();
  beforeEach(() => screenshotPromise);
  afterAll(() => screenshotPromise);

  /**
   * Take a screenshot on Failed test.
   * Jest standard reporters run in a separate process so they don't have
   * access to the page instance. Using jasmine reporter allows us to
   * have access to the test result, test name and page instance at the same time.
   */
  (jasmine as any).getEnv().addReporter({
    specDone: async (result: any) => {
      if (result.status === 'failed') {
        screenshotPromise = screenshotPromise
          .catch()
          .then(() => takeScreenshot(result.fullName));
      }
    },
  });
};

[petrogad]

Did you ever see blank screenshots and get around that issue?

Thank you for sharing!

@testerez ^

[gregberge]

It looks like a lot of people are close to the solution. Could someone try to submit a PR to add screenshot? 🙏

[helloguille]

@testerez Thanks for the code! However, I cannot I get undefined error for the pageInstance variable.

I just put all the code (In javascript) in my setupTestFrameworkScriptFile file + a call to registerScreenshotReporter().

I modified the declaration for Vainilla as such:

export const takeScreenshot = (testName, pageInstance = page) => { ... } And page is indeed the name of the page instance in my tests.

[RoiEXLab]

To come back at this issue, before I found this issue I had several attempts at this myself. My goal had higher ambitions though: I wanted jest to automatically generate a screenshot before and after each individual test. Disclaimer: I haven't really used the jest API before, so this is probably the best thing I came up with:

const path = require('path');
const fs = require('fs').promises;

const hash = (str) => {
    if (str.length === 0) {
        return 0;
    }
    let hash = 0;
    for (let i = 0; i < str.length; i++) {
        hash  = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0; // Convert to 32bit integer
    }
    return hash;
};

const origIt = global.it;
global.it = (description, test, timeout) => {
    const result = origIt(description, async () => {
        const basePath = path.join(result.result.testPath, '..', '__screenshots__');
        try {
            await fs.mkdir(basePath, { recursive: true });
            await page.screenshot({ path: path.join(basePath, `before-${hash(result.result.fullName)}.png`) });
        } finally {
            try {
                return await test();
            } finally {
                await page.screenshot({ path: path.join(basePath, `after-${hash(result.result.fullName)}.png`) });
            }
        }
    }, timeout);
    return result;
};

Re-defining it is kinda ugly but it works better than all my other versions. Also I ignored test for now, because in my case I don't need it. Now with Jest 24 the file can be specified in setupFilesAfterEnv as part of an array. I'd really like to see such a feature soon in this project, and I would create a PR as well, it's just that I'm not confident enough in my code. So if someone could nudge me into the right direction that would be great.

[xiaoyuhen]

@RoiEXLab

Great solution！welcome a PR to solve this problem.

Now, jest-puppeteer has a bug with jest@24 and we will fix it this week as soon as possible. see https://github.com/smooth-code/jest-puppeteer/issues/193

[celador]

// jest.setup.js

global.it = async function(name, func) {
  return await test(name, async () => {
    try {
      await func();
    } catch (e) {
      await fs.ensureDir('e2e/screenshots');
      await page.screenshot({ path: `e2e/screenshots/${name}.png` });
      throw e;
    }
  });
};

[redrockzee]

Use this package: https://github.com/jeeyah/jestscreenshot

[gregberge]

It is amazing to see that everyone has a solution and nobody wants to make a PR, welcome in Wordpress world 🤗

[lukaszfiszer]

@petrogad check if you're not calling jestPuppeteer.resetPage() or doing something else with global page object in afterEach/afterAll. This was the reason I was getting blank screenshots.

@neoziro I should be able to submit a PR with adapted solution of @testerez in following days

[ajs139]

Incase it's helpful, It seemed for me like the page was closed by the time jasmine reporter was called when using one of the above solutions, and overriding it prevents things like it.only from working. Ended up with this riff on @testerez's code, which takes a screenshot after every test (on a CI env), but only saves the screenshot if the test fails: https://gist.github.com/ajs139/3ddc10e807ee9b94b581c80a762de587

[rostgoat]

@testerez where would u even call registerScreenshotReporter ?

[testerez]

@rostgoat you can call it in your setupFilesAfterEnv jest script.

[rostgoat]

is this how your jest.config.js file looks @testerez?

setupFilesAfterEnv: [
    "<rootDir>/test/e2e/methods/common/screenshot.js"
  ]

how would this call the registerScreenshotReporter though?

[testerez]

registerScreenshotReporter should be called in the file you set as setupFilesAfterEnv

[AlexMarchini]

Thought I'd share my solution to this, using jest-circus and extending the jest-puppeteer environment. It's a little messy with the sleeps, I want to find a better way to handle that, but it works for now. Maybe jest-puppeteer can integrate with jest-circus out of the box once circus has more adoption and make this better.

const fs = require('fs');
const PuppeteerEnvironment = require('jest-environment-puppeteer');
require('jest-circus');

const retryAttempts = process.env.RETRY_ATTEMPTS || 1;

class CustomEnvironment extends PuppeteerEnvironment {

  async setup() {
    await super.setup();
  }

  async teardown() {
    // Wait a few seconds before tearing down the page so we
    // have time to take screenshots and handle other events
    await this.global.page.waitFor(2000);
    await super.teardown()
  }

  async handleTestEvent(event, state) {
    if (event.name == 'test_fn_failure') {
      if (state.currentlyRunningTest.invocations > retryAttempts) {
        const testName = state.currentlyRunningTest.name;
        // Take a screenshot at the point of failure
        await this.global.page.screenshot({ path: `test-report/screenshots/${testName}.png` });
      }
    }
  }

}

module.exports = CustomEnvironment

[allimuu]

@AlexMarchini thanks for that! Perfect, I wanted to use jest-circus as well. If anyone else is wondering (or anyone has suggestions) like I was how to actually use this this is what my jest-config.js looks like:

{
    ...
    globalSetup: 'jest-environment-puppeteer/setup',
    globalTeardown: 'jest-environment-puppeteer/teardown',
    testEnvironment: './jest-environment.js',
    testRunner: 'jest-circus/runner',
    setupFilesAfterEnv: ['./jest-setup.js'],
    ...
}

So I removed preset and jest-environment.js is the above snippet. And jest-setup.js has nothing but require('expect-puppeteer'); in it.

And that seems to be working.. with jest-circus as the runner. (Note just run yarn add --dev jest-circus)

Hope this helps someone.

[favna]

Previously I commented here that we had a solution for work. The short version is that it was based on @testerez work. Sadly for whatever reason that was a moment of glory yesterday evening as when I started to get it into our CI/CD pipeline this morning it all crashed and burned and even refused to work locally so I deleted that comment.

That said, I ended up completely rewriting the library I made now relying on a solution very similar to @AlexMarchini / @allimuu thus relying on Jest Circus and at least it seems to work properly now.

For those interested you can find it here (bumped to v2.0.0 as it broke previous releases). Instructions and example on how to use the library are included in the README you can read on the yarnpkg / npm site. This includes a maybe more detailed (due to including code sample) version of @allimuu 's contribution.

And as before here is the actual source code behind it

[dennismphil]

This was not as straightforward as I thought.

I found in the above methods it.only stops working. What worked for me was the below solution.

Monkey patching the Jest.it method.

// jest.config.js

module.exports = {
    ...
    setupFilesAfterEnv: [
        '<rootDir>/setupFilesAfterEnv.js',
    ]
};

// setupFilesAfterEnv.js

// Wrapper function that wraps the test function
// to take a screenshot on failure
const wrappedTest = (test, description) => {
    return Promise.resolve()
        .then(test)
        .catch(async (err) => {
            // Assuming you have a method to take a screenshot
            await takeScreenshot('some file name');

            throw err;
        });
};

// Make a copy of the original function
const originalIt = global.it;

// Modify `it` to use the wrapped test method
global.it = function it(description, test, timeout) {
    // Pass on the context by using `call` instead of directly invoking the method.
    return originalIt.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

// Copy other function properties like `skip`, `only`...
for (const prop in originalIt) {
    if (Object.prototype.hasOwnProperty.call(originalIt, prop)) {
        global.it[prop] = originalIt[prop];
    }
}

// Monkey patch the `only` method also to use the wrapper method
global.it.only = function only(description, test, timeout) {
    return originalIt.only.call(this, description, wrappedTest.bind(this, test, description), timeout);
};

[noomorph]

I think this pull request https://github.com/facebook/jest/pull/9397 may come to the rescue if you are struggling with asynchronicity.

[tihuan]

For future readers, if you wanna get https://github.com/smooth-code/jest-puppeteer/issues/131#issuecomment-493267937 to work, please make sure BOTH of your jest and jest-circus are on versions that include https://github.com/facebook/jest/pull/9397, which means both jest and jest-circus need to be v25.3.0+

Our project updated jest to v25.5.0, but jest-circus stayed at v25.2.7, resulting in Circus runner not waiting for test_fn_failure test event handler:

https://github.com/facebook/jest/pull/9397/files#diff-e77e110fa517918f09bcd38e70996fafR164

Note to self that jest and jest-circus have version parity, so always update both libraries together!

[stefanteixeira]

In case someone wants to know how to get the test describe name when using jest-circus (https://github.com/smooth-code/jest-puppeteer/issues/131#issuecomment-493267937 saves a screenshot only with the test method name), you can get it like:

const testDescription = state.currentlyRunningTest.parent.name 

I had to spend some time searching in the source code how State is defined inside jest-circus to find that information, hope it helps someone :smile: