[Feature] Should not skip test cases if one test case failed when running sequentially

[nikhilgupta1789]

Is there a way to prevent playwright from closing the browser context when one test fails inside a test.describe block? I have multiple tests inside the test.describe and I want to continue running the rest of the tests after some test fails in the same browser context.

I want to run the test cases sequentially but it shouldn't skip if one test case failed. What to do for this?

Every time invoking fresh state of browser consuming time to re-login and navigate to the desired screen where last test case executed. Maintaining same page instance will saves a lot of time a). reinvoking browser b). login again c). revisiting to the screen page

[dgozman]

Playwright strongly believes in test isolation, so we always start afresh after a failure. Are you using the reuse a single page between tests mode? If you could share more details, perhaps we can help with the time consuming re-login issue?

[nikhilgupta1789]

Sure @dgozman I have a scenario please have a look into this.

import { test, expect, Page } from '@playwright/test';

test.describe("Orange HRM -> Admin search functionality test", ()=>{

    let page:Page;
    //Login and navigating to Admin screen
    test.beforeAll(async({browser})=>{
      page = await browser.newPage();
      await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login');
      await page.locator("input[name='username']").type("Admin");
      await page.locator("input[name='password']").type("admin123");
      await page.locator("button[type='submit']").click();
      await expect(page.locator("h6[class*='oxd']")).toHaveText('PIM');
      await page.locator("//a/span[text()='Admin']").click();
    })

    //Test Case #1 : Search record(s) by Admin role
    test('Orange HRM -> Search records by Role as Admin', async() => {
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='Admin']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dabtabase query result with result search on UI
    });

    //Test Case #2 : Search record(s) by ESS role
    test('Orange HRM -> Search records by Role as ESS', async() => {
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='ESS']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dabtabase query result with result search on UI
    });

});

Configuration in playwright.config.ts file as fullyParallel as false. If you look into above scenario I want to login only one time in every test script in beforeAll hook, then I wanted to carry out all subsequent test cases with the same page instance in case of any failure in the test case so in that way we are saving a lot of time in re-login and navigate to the expected page to continue with the test execution flow. Here my above test cases is not the case like if previous test case failed then don't move ahead and stop the execution because subsequent has different scenario functionality.

There are two ways to execute test cases in sequentially using (a). serial mode [issue facing : if any test case failed it stops entire execution] (b). fullyParallel: false [issue facing : if any test case failed it is again invoking new browser with fresh state and losing the page state]

We have many teams and modules and we're planning to consider Playwright automation tool but missing this feature of not maintaining page state across the test cases puts us in dilemma. Could you please consider this feature to be introduce like in serial mode don't stop execution and continue with the page state introducing flag whenever failure count encounters defined by user at the runtime then stop the execution this kind of flexibility makes a difference in Playwright next generation tool. Thank you!! :)

[dgozman]

@nikhilgupta1789 I'd suggest to look into reusing signed in state. This way, you can log in once, and then start each test with the logged-in state pretty fast. You'll achieve test isolation, so that errors in one test do not affect other tests.

If you insist on using a single page, even in the case of failures, which we strongly discourage, you'll have to create a single test. Take a look into test steps to help you structure your single long test into logical parts.

[nikhilgupta1789]

Hi @dgozman for your first approach using reusing of signed in state and running test cases independently I tried running above example test cases with 2 workers making flag true for fullyParallel you will be surprised by seeing the difference in all the time invoking new browser instance or using signed in state with using same page instance across the test cases and log in only one time. By running parallel independent test cases took 51 secs of time to execute whereas running them sequentially but login only one time as in above example it is taking just 33 secs because it saves time in making worker processor up and then launching new browser. I am sharing the code below you may try at your end and notice the difference.

Case 1: Please try running below code with fullyParallel as true, it completes in 51 secs of time

import { test, expect, Page } from '@playwright/test';

test.describe("Orange HRM -> Admin search functionality test", ()=>{

    //Test Case #1 : Search record(s) by Admin role
    test('Orange HRM -> Search records by Role as Admin', async ({page}) => {
        await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login');
        await page.locator("input[name='username']").type("Admin");
        await page.locator("input[name='password']").type("admin123");
        await page.locator("button[type='submit']").click();
        await expect(page.locator("h6[class*='oxd']")).toHaveText('PIM');
        await page.locator("//a/span[text()='Admin']").click();

        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='Admin']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dbtabase query result with result search on UI
    });

    //Test Case #2 : Search record(s) by ESS role
    test('Orange HRM -> Search records by Role as ESS', async ({page}) => {
        await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login');
        await page.locator("input[name='username']").type("Admin");
        await page.locator("input[name='password']").type("admin123");
        await page.locator("button[type='submit']").click();
        await expect(page.locator("h6[class*='oxd']")).toHaveText('PIM');
        await page.locator("//a/span[text()='Admin']").click();
        
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='ESS']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dbtabase query result with result search on UI
    });

});

Case 2: Please try running below code with fullyParallel as false, it completes in 33 secs of time

import { test, expect, Page } from '@playwright/test';

test.describe("Orange HRM -> Admin search functionality test", ()=>{

    let page:Page;
    //Login and navigating to Admin screen
    test.beforeAll(async({browser})=>{
      page = await browser.newPage();
      await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login');
      await page.locator("input[name='username']").type("Admin");
      await page.locator("input[name='password']").type("admin123");
      await page.locator("button[type='submit']").click();
      await expect(page.locator("h6[class*='oxd']")).toHaveText('PIM');
      await page.locator("//a/span[text()='Admin']").click();
    })

    //Test Case #1 : Search record(s) by Admin role
    test('Orange HRM -> Search records by Role as Admin', async() => {
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='Admin']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dabtabase query result with result search on UI
    });

    //Test Case #2 : Search record(s) by ESS role
    test('Orange HRM -> Search records by Role as ESS', async() => {
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='ESS']").click();
        await page.locator("//button[text()=' Search ']").click();
        await page.waitForTimeout(1000);
        //compare dabtabase query result with result search on UI
    });

});

And sometime it is not feasible to run test cases in parallel mode because there are high chances of data mess-up for an example test case 1 is trying to add one record, and test case 2 is trying to export total records on UI and comparing result with exported excel and with records displayed on UI, lets say these two test cases are concurrently executing at the time of export it exports total 10 records and second 2nd test case has added one record now on UI total 11 records present and 1st test case got impacted because of one additional record added on UI and exported contains only 10 records because at the time of reading excel at the same time another test case has added the record. There are many more scenarios which is not compatible to run parallel all the time. We wanted to reuse page state and maintain across the test cases without quitting the browser instance.

Now let's talk about second approach which you have suggested of using test.step but again when something failure is encountering in test.step then again for the next subsequent test case it is invoking fresh browser instance which we don't want.

And trying to wrap in between the try/catch block to not forcefully fail the test cases and maintain the page instance which is also not feasible because we have to report the failure test case instead of handling in try/catch block and marking it pass instead of fail. I tried my possible best to explain you all my scenarios, please let me know other approaches. Simply giving control to user in serial mode introducing failure count until failure count meets then stop execution and maintain that page state.

[dgozman]

@nikhilgupta1789 I followed the reuse state documentation, created a global setup and start tests with page.goto(). Performance results:

• Running your first snippet with fullyParallel is 8 seconds each test, total 8 seconds. Extrapolating this to 40 tests: 320 seconds cpu time, 160 seconds real time with 2 cpus, 40 seconds real time with 8 cpus.
• Running your second snippet is 8 seconds login, 500ms each test, total 9 seconds. Extrapolating this to 40 tests: 28 seconds cpu time, 28 seconds real time with 2 or 8 cpus.
• Running with reused state, global setup is 8 seconds, each test is 4 seconds. Extrapolating this to 40 tests: 168 seconds cpu time, 88 seconds real time with 2 cpus, 28 seconds real time with 8 cpus.

So, around ~40 tests, running them in parallel with reused state is not slower than running them all in a single page. But you get test isolation and avoid any issues of failing tests affecting healthy ones.

---

Here is my global setup and test:

// global-setup.ts
import { chromium, expect } from '@playwright/test';

async function globalSetup() {
  const browser = await chromium.launch();
  const page = await browser.newPage();
  await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/auth/login');
  await page.locator("input[name='username']").type("Admin");
  await page.locator("input[name='password']").type("admin123");
  await page.locator("button[type='submit']").click();
  await expect(page.locator("h6[class*='oxd']")).toHaveText('PIM');
  await page.locator("//a/span[text()='Admin']").click();
  await page.context().storageState({ path: 'storage.json' });
  await browser.close();
}

export default globalSetup;

// example.spec.ts
import { test } from '@playwright/test';

test.describe("Orange HRM -> Admin search functionality test", ()=>{

    //Test Case #1 : Search record(s) by Admin role
    test('Orange HRM -> Search records by Role as Admin', async ({page}) => {
        await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/admin/viewSystemUsers');
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='Admin']").click();
        await page.locator("//button[text()=' Search ']").click();
    });

    //Test Case #2 : Search record(s) by ESS role
    test('Orange HRM -> Search records by Role as ESS', async ({page}) => {
        await page.goto('https://opensource-demo.orangehrmlive.com/web/index.php/admin/viewSystemUsers');
        await page.locator("//button[text()=' Reset ']").click();
        await page.waitForSelector("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]");
        await page.locator("//label[text()='User Role']/../following-sibling::div//div[contains(@class,'after')]").click();
        await page.locator("//label[text()='User Role']/../following-sibling::div//*[text()='ESS']").click();
        await page.locator("//button[text()=' Search ']").click();
    });
});

---

Overall, we believe in test isolation and prioritize features according to that. This feature request is explicitly asking to break test isolation. Instead, we would like to help you with your real issue, which seems to be performance? I'd still encourage you to look into reusing authenticated state, we have a dedicated guide for that.

[dgozman]

Note from an internal discussion on this issue: we might look into reusing browser's http cache in addition to cookies/storage so that loading from the stored state is much faster.

[nikhilgupta1789]

Thank you @dgozman this might really helpful in reusing browsers http cache. We're looking forward for this to be released in next playwright release version.

[nikhilgupta1789]

Actually there are many scenarios in our application where we can't always run test cases in parallel just because of data mess-up let me share example suppose there are two test cases running parallel first test is adding record in grid where 40 records already exists and second is exporting that record into excel which contains 40 at the time of exporting and comparing it with record displayed on UI now lets says there is a high chance when comparing record count first test case added one record which makes total 41 records whereas second test case comparing record with exported and record presents on UI so there is a mismatch because at the time of comparing first test case has added one record in the grid. Overall like running test cases makes data mess-up and impact other test cases scenarios. Our application has many features to get cover them we need to maintain the page state irrespective of failures so we can continue from that state.

Another example let's say one test case is adding a record and another is for editing a record, in this scenario edit record need additional record to be added because it can't rely on first test case where it is adding a record because by the time second test case will start and how would second test case know that first test case has added a record and then edit it so in that case unnecessary for second case we have first add again one record and then edit it. Now may be your question raised where if you want to run it sequentially then if first test case failed then why to run second test case because it has failed to add a record then what record in second test case it will edit so my point is give a control to user with introducing new flag such as --bail or --max-failures in case of fullyParallel as false and maintain the page state after each test run.

[nikhilgupta1789]

Hi @dgozman I have gone through the all playwright issues and many people needed the same requirement of maintaining the state and execution go on for subsequent test cases and not invoking fresh instance of browser to avoid repetition of steps and code. In serial mode subsequent are skipping, and in fullyParallel false not maintaining the page instance for subsequent test cases and we're getting new fresh instance of browser which requires code repetition and time consuming.

Here are all the list of defects raised in playwright: https://github.com/microsoft/playwright/issues/7050 https://github.com/microsoft/playwright/issues/7475 https://github.com/microsoft/playwright/issues/7458 https://github.com/microsoft/playwright/issues/15752 https://github.com/microsoft/playwright/issues/17266 https://github.com/microsoft/playwright/issues/17994 https://github.com/microsoft/playwright/issues/16199 https://github.com/microsoft/playwright/issues/16119 https://github.com/microsoft/playwright/issues/18329

I hope you may think on this feature we have many projects in the organization and migrating from protractor end-to-end test cases.

[nikhilgupta1789]

Hi @dgozman I hope you're doing well. Any update on this feature. We're eagerly waiting for this feature to be introduced so start working on migrating task from Protractor to Playwright. Thank you!! :)

[sree030289]

Good have this feature

[rahulgupta07]

Much needed feature indeed

[sagar-goldcast]

@dgozman I hope you are fine. But really this feature is becoming bottleneck for larger flows and e2e requirements. Can we introduce this feature please?

[sagar441989]

Yes, me too facing the same issue with few test. Due to application issue, I can not execute test in parallel as they interfere with the test data. So, I used test.describe.configure({mode: 'serial'}); However the problem with this approach if the first test fails the other tests get skipped.

[Girish-Deshmukh]

Hi all, i am also facing the same issue in my Application we really need this feature to execute tests even if one of the test cases fails since there are multiple applications that require this feature

[sagar-goldcast]

There is a work around like this. Please use carefully if any things fails except step it will occurs with error.

test.describe.serial('@P0 @all', () => {
//First test case
test('Test case 1', async ({}, testInfo) => {
    await CommonPage.goto(event_url); // this method call should be always correct
    await test.step('Verification', async () => {
      await CommonPage.verifyDisplay(); // use method call with soft assert.
    });
     await CommonPage.clickLogin();  // this method call should be always correct
        await test.step('Verification Step1', async () => {
          await CommonPage.verifySecondDisplay(); // use method call with soft assert.
        });
     await test.step('Verification Step2', async () => {
          await CommonPage.verifySecondDisplay(); // use method call with soft assert.
        });
  });

// second test case
test('Test case 2', async ({}, testInfo) => {
    await test.step('Verification', async () => {
      await CommonPage.verifySecondDisplay(); // use method call with soft assert.
    });
  });
});

[mrrinatino]

Hello! I also wanted to express a wish regarding this function. In my project, we currently have around two thousand tests, and to optimize and speed up testing, we run tests in parallel. However, there are still many complex scenarios that need to be run sequentially. It's very inconvenient that in the function test.describe.configure({ mode: 'serial' }), there's a nuance where subsequent tests are skipped if one fails. We are eagerly awaiting a solution that allows not skipping the remaining tests.

[nikhilgupta1789]

Fortunately I have moved to webdriverio framework which is more robust, frequent release and good community who works on raised issues. I got what I actually wanted from webdriverio framework in sequential mode and as well as in parallel mode. I was also looking for good framework for migrating our Protractor framework to another javascript/typescript framework. Playwright no doubt good framework but it lacks in core and basic features that's why I planned to move to webdriverio framework.

[AdamOakman]

I think the issue is about the importance of understanding by the playwright framework maintainers that both serial mode with skipping and without skipping subsequent tests within one describe block are valid use cases.

@dgozman Ona elegant way of distinguishing it would be something like this (it should override whatever is set in global config in fullyParallel):

test.describe.configure({ mode: 'serialDependent' }); // would skip all subsequent tests after first failure
test.describe.configure({ mode: 'serialIndependent' }); // would go on with test execution regardless of results of preceeding tests

Looks like both are achievable at the moment but very unintuitive to understand the logic behind it - first one by setting mode test.describe.configure({ mode: 'serial' }); and the second one by not setting anything in spec file (or having implicitly test.describe.configure({ mode: 'default' }); ) AND having fullyParallel: false in global config.

But I understand some people would like to utilize fullParallel: true mode with having an option to run some spec files in serial mode without skipping subsequent tests after failure - which is not possible from my observation at the moment.

[MaksimMedvedev]

upvoting, that would really be useful to run a tests inside a certain file sequentially withour failing the whole test suit on a single test failure

wouldn't the

test.describe.configure({ mode: 'default' });

help though? We have a fullyParallel option set to true and running tests using this default mode for a single file runs all tests in a single worker, but without default mode runs them using 8 workers

Seems like the default mode works here or is it?

[siurbele420]

I think the issue is about the importance of understanding by the playwright framework maintainers that both serial mode with skipping and without skipping subsequent tests within one describe block are valid use cases.

@dgozman Ona elegant way of distinguishing it would be something like this (it should override whatever is set in global config in fullyParallel):

test.describe.configure({ mode: 'serialDependent' }); // would skip all subsequent tests after first failure
test.describe.configure({ mode: 'serialIndependent' }); // would go on with test execution regardless of results of preceeding tests

Looks like both are achievable at the moment but very unintuitive to understand the logic behind it - first one by setting mode test.describe.configure({ mode: 'serial' }); and the second one by not setting anything in spec file (or having implicitly test.describe.configure({ mode: 'default' }); ) AND having fullyParallel: false in global config.

But I understand some people would like to utilize fullParallel: true mode with having an option to run some spec files in serial mode without skipping subsequent tests after failure - which is not possible from my observation at the moment.

I totally agree on this. We are responsible for implementing the features correctly and we should be aware of any unisolated env dependencies as well. So by having an option to run tests in serial and allow further tests in the same group to be executed even if preceding ones are failing is perfectly valid situation. I can structure my tests in a way that it doesn't matter if one of the same group test fails by using fixtures or hooks.

[Omarabdkhaleq]

Hi, Is there an ETA when this feature is gonna be released? I'm in the middle of implementing a poc to migrate to Playwright. but this issue is a blocker for our situation

[ritapure-tibco]

Do we have nay update on this from playwright team. We are eagerly waiting for this feature. We are also blocked when run tests serial mode and one test case failed, all further test cases are failed.

[suyamamadushan]

• vote Thank you

[afreen07]

I am running API tests and one failure causes all other tests to skip, please provide this feature for playwright. Thank you