[Feature] Add support for test.each / describe.each

[mxschmitt]

Note from maintainers:

You can already do each via running test in a loop:

for (const data of [1, 2,3, 4]) {
  test('testing ' + data, async ({ page }) => {
    // ...
  });
}

If the above does not work for you, please tell us why in the comments.

[partparsh]

Hi, I am currently using the this approach (loop) for data driven tests and was wondering, is there a way we can make these tests run in parallel? Because right now they are being executed sequentially and for "slow" tests the execution time is increased.

[suchlike]

+1 to @partparsh 's suggestion of parallelization. We have a suite of a single test with 20+ parameters that unfortunately adds a lot of time to our build.

[pheeel]

For of does work, but Jest's test.each() is more convenient for us, here's an example: We have a data-driven test that checks URL redirecting. The test works with an array that represents 3 cases with 2 parameters in each case, e.g:

const testCases = [
  ['singlePath', `${ROUTES.HOME}singlePath`],
  ['nestedPath', `${ROUTES.HOME}nested/path`],
  ['section', `${ROUTES.COMPLICATED_PATH}section`],
];

So with Jest, I can write my test like this:

test.each(testCases)('%s url', async (urlType, url)) => {
  await page.goto(url);
  expect(page.url()).toBe(errorPage);
}

With PW test I have to write like this:

  for (const data of testCases) {
    test(`${data[0]} url`, async ({ page }) => {
      await page.goto(data[1]);
      expect(page.url()).toBe(errorPage);
    });
  }

These data[0], data[1] don't look good, I can reassign them...

  for (const data of testCases) {
    const urlType = data[0];
    const url = data[1];
    ...

...or use destructuring assignment...

 for (const [urlType, url] of testCases) {
    ...

...but with data-driven tests with a lot of parameters, it's not very convenient and tests look slightly messy.

[dgozman]

I don't see any real different between the following two. If anything, the latter even gives me more control - for example, I can skip a test when urlType has some specific value, and more.

test.each(testCases)('%s url', async (urlType, url)) => {
});

for (const [urlType, url] of testCases) {
  test(`${url} url`, async ({ page }) => {
  });
}

[hungdao-testing]

In my case below, the hook beforeEach, afterEach runs 5 times per 1 test.

How could we achieve the goal that 1 hook 1 test ? Note that, the beforeEach/afterEach could not move out of the for loop because I need to get data from array arr to initialize (e.g. log-in with admin, log-in with supervisor...).

test.describe.only('Looping', () => {
    let arr = [1, 2, 3, 4, 5]
    for (const tour of arr) {

        test.beforeEach(async () => {
            console.log("before each: ", tour)

        })

        test.afterEach(() => {
            console.log("after each", tour)
            console.log("======")

        })


        test(`TEST ${tour}`, async () => {
            console.log("test: ", tour)
            expect(1).toBe(1)
        })
    }
})

[dgozman]

@hungdao-testing Just make a few describes in a loop.

let arr = [1, 2, 3, 4, 5];
for (const tour of arr) {
   test.describe.only(`Looping ${tour}`, () => {
        test.beforeEach(async () => {
            console.log("before each: ", tour)
        })

        test.afterEach(() => {
            console.log("after each", tour)
            console.log("======")
        })


        test(`TEST ${tour}`, async () => {
            console.log("test: ", tour)
            expect(1).toBe(1)
        })
    })
}

[SuwiDeity]

I don't see any real different between the following two. If anything, the latter even gives me more control - for example, I can skip a test when urlType has some specific value, and more.

Playwright already allows us to do conditional test.skip inside test method, so i don't think "more control" argument is valid.

[4ekki]

+1 for test.each notation. Also, Jest has amazing way of populating data via table, which is very convenient when test case has 3+ parameters

[w9ahmed]

That would be nice to have IMO. My argument is more psychological :smile: I do agree there is no difference between having a for loop but test.each keeps you more in a "testing mindset". Also for consistency.

[gegoncalves]

I understand that there is no difference between them but it makes everything more focused on the test side aspect. Including the fact that @4ekki brought.

Moreover, that would help to run the tests in parallel by data group.

This is just my opinion

[gegoncalves]

@pavelfeldman and @mxschmitt Do you think that is a possible new feature?

[agusmakmun]

@mxschmitt my tests didn't work for deep loop tests, somehow it stopped at the first iteration, i.e:

test.describe('Test all exercises', () => {
    for (let courseIndex = 0; courseIndex < settings.COURSES.length; courseIndex++) {
        let courseSlug: string = settings.COURSES[courseIndex]
        test(`testing ${courseSlug}`, async ({page}) => {
            await page.goto(`${settings.BASE_URL}/courses/${courseSlug}/`)
            await expect(...)
            
            let chapters = ....
            for (let chapterIndex = 0; chapterIndex < chapters.length; chapterIndex++) {
                await expect(....).toBeVisible()
                ...
                
                let exercises = ....
                
                for (let exerciseIndex = 0; exerciseIndex < exercises.length; exerciseIndex++) {
                    await page.goto(...)
                    await startLab(page)
                    await expect(...)
                    
                    let steps = ....
                    for (let ...) {
                        await expect(...)
                        ... <== stoped here
                    }
                }
            }
       })
    }
})

Update ^

It's works by adding test.setTimeout(0) inside the test, https://stackoverflow.com/a/69468830/6396981 for example,

test.describe('Test all exercises', () => {
    for (const courseSlug of settings.COURSES) {
        test(`testing ${courseSlug}`, async ({page}) => {
            test.setTimeout(0)
            ...
        }
    }
})

[alexkrolick]

Another limitation of for ... of is that if one test throws an error and fails, the rest of the loop fails to execute. That means you get varying numbers of tests reported per run, as well as hiding errors in tests later on in the loop until you fix the first one.

[nathaner]

Hello

Thanks for considering the option to add support for this feature.

I think it would help to easily create tests for multiple users. To demonstrate this, we can take a todo app as an example with the following 2 properties:

• The app is dedicated for teams sharing todos, 3 different roles exist: owner, admin and member.
• There is a free plan and a paid plan. Paid plan allows the user to assign a specific todo. In paid plan, only the owner and the admin can assign a todo.

Now, some test cases for assigning a todo can be:

• can assign a todo (owner and admin in paid plan).
• cannot assign a todo (member in paid plan, owner, admin and member in free plan).

Using for loops

In this example, a parameterized test in Playwright might look like:

import { test, expect } from '@playwright/test'

const authorizedRoles = [
    { role: 'owner', plan: 'business' },
    { role: 'admin', plan: 'business' },
]

const unauthorizedRoles = [
    { role: 'member', plan: 'business' },
    { role: 'owner', plan: 'free' },
    { role: 'admin', plan: 'free' },
    { role: 'member', plan: 'free' },
]

for (const { role, plan } of authorizedRoles) {
    test.describe('', () => {
        test.use({ storageState: getStorageStatePath(role, plan) })
        test(`a ${role} can assign a todo to a user in ${plan} plan @${role} @${plan}`, async ({ page }) => {
            await page.goto('todo/123')
            const assignee = '[email protected]'
            await page.getByLabel('Assignee').fill(assignee)
            await expect(page.getByText(`"Assigned task to ${assignee}`)).toBeVisible()
        })
    })
}

for (const { role, plan } of unauthorizedRoles) {
    test.describe('', () => {
        test.use({ storageState: getStorageStatePath(role, plan) })
        test(`a ${role} cannot assign a todo to a user in ${plan} plan @${role} @${plan}`, async ({ page }) => {
            await page.goto('todo/123')
            await expect(page.getByLabel('Assignee')).not.toBeVisible()
            await expect(page.locator('button >> "Assign"')).not.toBeVisible()
        })
    })
}

We can argue that tests verifying everything concerning the authorization should not be tested using Playwright. Nevertheless, this implementation helps to easily reuse the test case and apply it to other roles and plans.

The for loop is not a big deal, but it might become one if we want to do something like this for many tests. Here are the 2 ways I found to achieve something sort of reusable.

Leverage Playwright projects

We can create multiple projects corresponding to the different plans and roles we have:

const plans = ['free', 'paid']
const roles = ['owner', 'admin', 'member']

const projects = []
for (const plan of plans) {
    for (const role of roles) {
            projects.push({
            name: `chromium-${plan}-${role}`,
            use: {
                ...devices['Desktop Chrome'],
                plan,
                role,
            },
            })
    }
}

const config: PlaywrightTestConfig = {
    // ...
    projects: [
        ...projects,
        {
        name: 'chromium',
        use: {
            ...devices['Desktop Chrome'],
        },
        },
        {
        name: 'firefox',
        use: {
            ...devices['Desktop Firefox'],
        },
        },
        {
        name: 'webkit',
        use: {
            ...devices['Desktop Safari'],
        },
        },
    ],
    // ...
}

Now we can skip a test if it should not be run for the current project:

const authorizedProjects = ['chromium-paid-owner', 'chromium-paid-admin']

test('a user can assign a todo to a user', async ({ page }) => {
    const projectName = test.info().project.name
    test.skip(!authorizedProjects.includes(projectName))
    await page.goto('todo/123')
    const assignee = '[email protected]'
    await page.getByLabel('Assignee').fill(assignee)
    await expect(page.getByText(`"Assigned task to ${assignee}`)).toBeVisible()
})

However:

• I lose the tags
• Multiple test cases will have the same name
• The output will have lots of skipped test cases
• The --project command line option only works with 1 project at a time, I cannot start multiple projects at once without starting all the projects

Custom implementation

To keep the tags and have distinct names for all the test cases we can implement a custom function.

export function parametrizedTest(
    plans: Plan[],
    roles: Role[]
    name: string,
    testFunction: (args: PlaywrightTestArgs & PlaywrightTestOptions & PlaywrightWorkerArgs & PlaywrightWorkerOptions, testInfo: TestInfo) => void
): Promise<void> | void {
    for (const plan of plans) {
        for (const role of roles) {
            const tags = [plan, role].map(tag => `@${tag}`)
            const title = `a ${role} ${name} in ${plan} plan ${tags}`
            test.describe('', () => {
                test.use({storageState: getStorageStatePathForRole(role)})
                test(title, testFunction)
            })
        }
    }k
}

We can then use this function to create parametrized tests:

parametrizedTest(['paid'], ['owner', 'admin'], 'can assign a todo to a user', async ({ page }) => {
    await page.goto('todo/123')
    const assignee = '[email protected]'
    await page.getByLabel('Assignee').fill(assignee)
    await expect(page.getByText(`"Assigned task to ${assignee}`)).toBeVisible()
})

Here again we have a big issue:

• as the callback function is actually the one running the test, all tests are shown as being run in the file containing the parametrizedTest function.

Conclusion

I haven't found any clean way to achieve that for now. Maybe this shouldn't be covered by Playwright and maybe a test.each feature wouldn't help. What are your thoughts? Is this use case out of scope or is it a good example for why a test.each might help?

[gegoncalves]

I have the same problem as @alexkrolick Is there a way to avoid that?

Another limitation of for ... of is that if one test throws an error and fails, the rest of the loop fails to execute. That means you get varying numbers of tests reported per run, as well as hiding errors in tests later on in the loop until you fix the first one.

[dgozman]

I have the same problem as @alexkrolick Is there a way to avoid that?

Another limitation of for ... of is that if one test throws an error and fails, the rest of the loop fails to execute. That means you get varying numbers of tests reported per run, as well as hiding errors in tests later on in the loop until you fix the first one.

This should not be the case. Calling test() function does not actually run the test immediately, and it never throws. Unless you explicitly throw somewhere between calling test(), all the tests are collected and run. If you believe otherwise, please file an issue with a repro, and we'll definitely fix the bug.

[NikkTod]

We tent to use multiple params and I was wondering if it is possible to have more elegant and easy to read representation like we have for example in other test runners:

NUnit:

[TestCase(10,10,10,90)] [TestCase(10,10,0,100)] public void testCalculate(double price,int quantity,double discount,double expectedFinalAmount) {...}

Cucumber:

Scenario Outline: Test Name Examples: | start | eat | left | | 12 | 5 | 7 | | 20 | 5 | 15 |

Jest

test.eacha | b | expected ${1} | ${1} | ${2} ${1} | ${2} | ${3} ${2} | ${1} | ${3}('returns $expected when $a is added to $b', ({a, b, expected}) => { expect(a + b).toBe(expected); });

[sergtimosh]

Here there is another good reason why test.each approach might be a good option https://github.com/microsoft/playwright/issues/26431

There are two other ways to achieve what topic starter wants:

1. With global setup

projects: [
        {
            name: 'get test parameters',
            testDir: './',
            testMatch: 'global.setup.ts',
        },
        {
            name: 'chromium',
            use: {
                ...devices['Desktop Chrome'],
                viewport: { width: 1900, height: 1200 },
            },
            dependencies: ['get test parameters'],
        },

        
//global.setup.ts
import fs from 'fs';

async function runGlobalSetup() {
    try {
       const parameters = await fetchTestCaseParameters()
       fs.writeFileSync('params.json', parameters);
    } catch (error) {
        console.error('Error retrieving test parameters:', error);
    }
}
runGlobalSetup();

//test.spec.ts
import fs from 'fs';

let rawdata = fs.readFileSync('params.json');
let parameters = JSON.parse(rawdata).feature1

test.describe('Test suite', async () => {

for(parameter of parameters)
    test(`${parameter}`, async ({ page }) => {
        //test
    });
});

2. With synchronous lib

import { test } from '@playwright/test';
import request from 'sync-request';

const parameters = Object.keys(request('GET', 'https://api.publicapis.org/entries').headers);

test.describe('Test suite', async () => {
    parameters.forEach(parameter => {
        test(`${parameter}`, async ({ page }) => {
            console.log(parameter);
        });
    });
});

But it feels like it's better to have some native pw approach focused on parametrisation

[maxsudik]

+1 for test.each

[giuliagutierrez]

I don't know if there's something wrong with my code, but the setup doesn't work with the for aproach.

If others have the same problem, it would be another valid point to add the test.each feature.

[jamieomaguire]

I'd also prefer using test.each syntax to the for loop, fwiw. I am finding the loop approach to be rather clunky.

[vekunz]

test.each and describe.each would have an advantage for IDE integrations also. These integrations cannot handle tests inside a regular for-loop, but the *.each syntax could be handled well.

[Morgy93]

test.describe "Something" {
    test.describe.configure "serial mode"
    
    links = [];

    test "get some links from a page" {
        populate $links
    }

    foreach $links {
       test "check link"
    }
}

This does not work. I need to put the loop inside the "check link" test. MAYBE or HOPEFULLY, this is approach is possible with test.each.

[iamfj]

Could we have an update on the progress of the ticket regarding the each syntax implementation? Other testing frameworks have already adopted this feature, and it would be beneficial for us to understand the timeline for its integration into our processes.

[PowerCreek]

disclaimer: this is just a gist of how to shape the code setup for what your goals are. disclaimer2: I quickly whipped this up together, so pick it apart if you want, ya'll can take it and run with it

testProxy.js

export const testFor = (testInput, instanceOfThis) => {
    const thisReference = instanceOfThis??testInput;

    const self = {};

    self.cases = (function(arrayReference){
        return ({run: this});
    }).bind(thisReference);
    
    self.casesAsync = (async function(arrayReference){
        return {run: this};
    })
    
    const getSelf = (property) => self[property].bind(thisReference)

    return new Proxy(testInput,
        {
            get:function(target,property,receiver){
                if(property in self)
                    return getSelf(property);
                return target[property];
            },
        }
    );
}

sample.spec.js

import {test as t} from 'your test source';
const test = testFor(t);

test.cases([1,2,3]).run('testname', () => expect(1).toEqual(1));

it's basically cheating.

[PowerCreek]

scenario.js

const withName = (obj) => typeof obj === 'object' && 'name' in v && 'name' || JSON.stringify(obj);

export const getScenario = (test, op) => ({
  scenario: ({ name, testCases: testCases = [undefined], run }) => {
    const items = testCases.map((testCase) => {
      const testFunction = ({ pageInstance }) => run(pageInstance, { testCase });
      return [`${name} ${(testCase === undefined && ' ' || 'case' && withName(testCase))}`, testFunction];
    });

    for (let a of items) (async () => op(a))();
  },
  test: test,
});

your.spec.js

import { test as t } from '../customExtendedTest';
import { getScenario, testFor } from '../scenario.js';
import { expect } from '@playwright/test';

const { test, scenario } = getScenario(t, (e) => t(...e));

test.describe('the test', async () => {
  scenario({
    name: 'my scenario',
    testCases: [1, 2, 3],
    run: async ({ pageInstance }, {testCase}) => {
      expect(1).toEqual(1);
    },
  });
});

[davidenke]

For the time being we're using a helper tagging function:

export function fromTable<
    S extends string,
    A extends ReadonlyArray<S>,
    P extends any[]
>([header]: A, ...values: P): { [K in S]: P[number] }[] {
    // read header from table
    const headers = header.split('|').map(h => h.trim());
    const rows: Array<Record<S, P>> = [];

    // gather data from table
    for (let i = 0; i < values.length; i += headers.length) {
        const row = values.slice(i, i + headers.length);
        const params = {} as Record<S, P>;
        for (let j = 0; j < headers.length; j++) {
            params[headers[j] as S] = row[j];
        }
        rows.push(params);
    }

    return rows;
}

This can be used in a for loop with a template string:

for (const { a, b, c } of fromTable`
        a    | b      | c
        ${1} | ${'a'} | ${true}
        ${2} | ${'b'} | ${false}
    `) {
    test(`Each with fromTable ${a}`, async () => {
        expect(typeof a).toBe('number');
        expect(typeof b).toBe('string');
        expect(typeof c).toBe('boolean');
    });
}

[curiosity26]

+1 to test.each and describe.each with parallelization. The ergonomics of playwright already align with things like testing library and jest, so why not complete the catalog? As an engineer, if I can flow between unit tests and e2e tests without having to reach for different tooling, it makes my life easier.

[AsemK]

+1 for test.each

[jaronaearle]

Another +1 for test.each/describe.each