[Feature] --max-failures for test.describe.serial mode

[nikhilgupta1789]

Context:

• Playwright Version: 8.7.0
• Operating System: Windows
• Node.js version: 14.5.1
• Browser: Chromium
• Extra: N/A

Code Snippet

Help us help you! Put down a short code snippet that illustrates your bug and that we can run and debug locally. For example:

import { chromium, expect } from "@playwright/test";
const {test, config} = require('@playwright/test');

test.describe.serial("Failure test scenario", ()=>{

    test("explicitly trying to fail test case - 1", async()=>{
            await expect(false).toBe(true);
    });

    test("explicitly trying to fail test case - 2", async()=>{
        await expect(false).toBe(true);
    });

    test("explicitly trying to fail test case - 3", async()=>{
        await expect(false).toBe(true);
    });

    test("explicitly trying to fail test case - 4", async()=>{
        await expect(false).toBe(true);
    });

    test("explicitly trying to fail test case - 5", async()=>{
        await expect(false).toBe(true);
    });

});

Describe the bug test stop executing as soon as it detects 1 failure even though used flag as --max-failures=10 with test.describe.serial, without serial it is working fine but I want this to be for serial also, because wanted to continue test execution flow and page instance should be shared between the test blocks.

Without using serial I have to again login and use the repetitive steps for continue test as my test cases are not independent.

Add any other details about the problem here. command: npx playwright sample-test.spec.ts --max-failures=10

[mxschmitt]

if you want to continue test execution, you should remove serial, then tests will continue to run.

Does this work for you?

[nikhilgupta1789]

@mxschmitt Hi Max, yes this will work. BUT my test cases are dependent to each other and I want continuation of test block running serially one after another. Because independent test cases can't be design all the time for large application for end to end scenarios. And in case of independent test block there is a code repetition which I don't want. Could you please add this feature to decide migration of our framework from protractor to playwright. This is only feature which is a blocker for us.

[nikhilgupta1789]

Hi @mxschmitt any update on this feature? we're still waiting for this feature to be introduce in the new release so we can decide the migration from Protractor to Playwright.

[nikhilgupta1789]

Hi @mxschmitt, I have an example of Playwright site after launching Playwright site then clicking on Get Started link and then clicking on every links e.g. Writing Tests, Running Tests, Test Generator links and verifying page header title. I have explicitly failed 3rd test block and running with fullyParallel flag as false. Here in the below code example I would like to run all test blocks in sequentially and it should maintain the page state after failing any of the earlier test block so that the subsequent test case can takeover from that page state and continue with the test case instead of launching new worker which in results calling beforeAll once again open new page but subsequent test block expecting the state which was at the stage of previous failure test.

import { test, expect, Page } from '@playwright/test';
test.describe("Playwright visiting on every page and match page title", async()=>{

  let page:Page;
  test.beforeAll(async({browser})=>{
      page = await browser.newPage();
      await page.goto('https://playwright.dev/');
  });

  test('Click on Get Started and verify page header title', async () => {
    // Expect a title "to contain" a substring.
    await expect(page).toHaveTitle(/Playwright/);
  
    // create a locator
    const getStarted = page.locator('text=Get Started');
  
    // Expect an attribute "to be strictly equal" to the value.
    await expect(getStarted).toHaveAttribute('href', '/docs/intro');
  
    // Click the get started link.
    await getStarted.click();
  
    // Expects the URL to contain intro.
    await expect(page).toHaveURL(/.*intro/);

    //Verify page header title
    let pageHeader = page.locator("header h1");
    await expect(pageHeader).toContainText('Installation');

  });

  test('Click on Writing Test link and verify page header title', async () => {
    // create a locator for Writing Test link
    const writingTestLink = page.locator("a[href='/docs/writing-tests'][class='menu__link']");
  
    // Click the Writing Test link.
    await writingTestLink.click();
  
    // Expects the URL to contain intro.
    await expect(page).toHaveURL(/.*writing-tests/);

    //Verify page header title
    let pageHeader = page.locator("header h1");
    await expect(pageHeader).toContainText('Writing Tests');
  });

  test('Click on Running Test link and verify page header title', async () => {
    // create a locator for Running Test link
    const runningTestLink = page.locator("a[href='/docs/running-tests'][class='menu__link']");
  
    // Click the Writing Test link.
    await runningTestLink.click();
  
    // Expects the URL to contain intro.
    await expect(page).toHaveURL(/.*running-tests/);

    //Verify page header title
    let pageHeader = page.locator("header h1");
    await expect(pageHeader).toHaveValue('Running Tests');
  });

  test('Click on Test Generator link and verify page header title', async () => {
    // create a locator for Test Generator link
    const testGeneratorLink = page.locator("a[href='/docs/codegen-intro'][class='menu__link']");
  
    // Click the Writing Test link.
    await testGeneratorLink.click();
  
    // Expects the URL to contain intro.
    await expect(page).toHaveURL(/.*codegen-intro/);

    //Verify page header title
    let pageHeader = page.locator("header h1");
    await expect(pageHeader).toContainText('Test Generator');
  });


});

Could you please please add this feature of maintaining the state of page if any earlier test fails and subsequent test case can take control from that state and include this feature with --max-failures or --bail flag if someone wanted to stop the execution if number of failures encountered as defined in max-failures and bail flags.

We have major End-to-End test cases where we are not worried on previous test block failure but subsequent test block can take the page state form where it left and do not invoke new browser as worker in specially case of fullyParallel as false.

There are millions of web applications where it is not always the case first 'Add Item To Cart' and then 'Pay For Item', applications behaves differently having different domains. For an example If I talk about my scenarios I am clicking on 'Get Started' button on Playwright site and then clicking on Writing Tests, Running Tests, Test Generator, Trace Viewer etc links and verifying title of each landed page, if anyone of the page title is misspell doesn't mean other subsequent test cases has not to be tested or skipped. Why not this feature should be added while in case of test.describe.serial where any earlier failure should not skip subsequent test blocks and give control on failures with flags as --max-failure or --bail counts encountered and then stop execution when expected failure counts meet. I am eagerly waiting for this feature to be introduce so we can freely write our some of the end to end test scenarios in dependent manner which we had written in Protractor where page state maintains and shared between test blocks.

Also many people raised this queries for dependent test cases atleast it has to be given a control to user in test.describe.serial. I request to you @mxschmitt please think on this. This is a very valid scenarios used in many applications where earlier of test failure should continue with the subsequent test blocks because applications has different scenarios. Thank you very much!! :)

[nikhilgupta1789]

@mxschmitt any update on this feature to maintain the state in subsequent test cases and to not start with fresh state of browser in subsequent, persist same instance across the test cases if running in serial mode. else we have to reach to that state in subsequent test cases with fresh state.

[pavelfeldman]

Why was this issue closed?

Thank you for your involvement. This issue was closed due to limited engagement (upvotes/activity), lack of recent activity, and insufficient actionability. To maintain a manageable database, we prioritize issues based on these factors.

If you disagree with this closure, please open a new issue and reference this one. More support or clarity on its necessity may prompt a review. Your understanding and cooperation are appreciated.