[BUG] Authentication only works for the first tests

[rufi0h]

System info

• Playwright Version: [v1.33]
• Operating System: [Windows 10 Enterprise]
• Browser: [Chromium]
• Other info: using ping for auth

 

Auth.setup.ts

import { expect, test as setup } from '@playwright/test';
import envConfig from './config';
const ENV: string = process.env.ENV;
const password: string = process.env.PASSWORD;
let username: string;

 


const authFile = 'tests/utils/playwright/.auth/user.json';

 

setup('authenticate', async ({ page }) => {
  await page.goto(`url`);
  //login
  await page.locator('id=username').fill(username);
  await page.locator('id=password').fill(password);
  await page.locator('button', { hasText: 'Sign On' }).click();

 

  await Promise.all([
    page.waitForResponse(
      (res) =>
        res.status() === 200 &&
        res.url() === 'url'
    )
  ]);
  // End of authentication steps.

 

  await page.context().storageState({ path: authFile  as string  });

 

});

 

Config file

 

import { defineConfig, devices } from '@playwright/test';

 


export default defineConfig({
  reporter: [
    ['line'],
    ['junit', { outputFile: 'tests/test-results/playwright-test-results.xml' }]
  ],
  outputDir: 'tests/test-results/',
  workers: 1,
  timeout: 60 * 1000,
  expect: {
    timeout: 15000
  },
  use: {
    actionTimeout: 0,
    trace: 'on-first-retry',
    video: 'on-first-retry',
    screenshot: 'only-on-failure',

 

  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        viewport: { width: 1920, height: 1080 },
        storageState: 'tests/utils/playwright/.auth/user.json'
      },
      dependencies: ['setup']
    }
  ],
});

 

Test file

 

import { expect, test } from '@playwright/test';
const globalTimeout = 60000;

 


test.beforeEach(async ({ page }) => {
  test.setTimeout(120000);
  await page.goto(`URL`);
});

 

test('test 1', async ({ page }) => {
  test.setTimeout(globalTimeout);

 

  //navigating to Analytics ->page 1 and wait for APIs to return 200
  await page.locator('text=Analytics').first().click();
  await Promise.all([
    page.waitForResponse(resp => resp.url().includes('API PATH') && resp.status() === 200),
    await page.locator('text=Hospital Compare >> nth=0').click()
  ]);

 

  await expect(page.locator('[data-test-id="global-filters-system"]')).toHaveText('test 2 ');

 

});

 

test('test 2', async ({ page }) => {
  test.setTimeout(globalTimeout);

 

  //navigating to Analytics -> page 2
  await page.locator('text=Analytics').first().click();
  await Promise.all([
    page.waitForResponse(resp => resp.url().includes('API PATH') && resp.status() === 200),
    await page.locator('text=equalizerKey Indicators').click()
  ]);

 

  //Verify the global filters
  await expect(page.locator('[data-test-id="global-filters-system"]')).toHaveText('test 2 ');
});

 

Steps

• [Run the test]

 

Expected  I expect that all tests will use the storage state and bypass the authentication after the auth.setup.ts runs successfully.

 

Actual The first test that is run after the auth.setup correctly bypasses the auth, but the next tests will no longer use the stored auth. I will see my login page on every test after the first test successfully uses the auth.setup

[Exoow]

You should try printing the auth.json file contents at the start of every test and compare them. Maybe accessing the app invalidates your token somehow, or, I've had it that the json was missing "origins" and thus subsequent test runs couldn't use it.

[rufi0h]

I spent the morning testing this and found no difference between the auth.json when I printed it out before the tests.

[a16su]

same to me

[mxschmitt]

@a16su or @rufi0h would it be possible to share a reproduction with us which we can run locally?

[a16su]

@a16su or @rufi0h would it be possible to share a reproduction with us which we can run locally?

try this https://github.com/a16su/playwright_demo

[mxschmitt]

I tried to reproduce on mac and Windows but was not able to:

Repro steps:

• clone the repo
• npm ci
• npx playwright test

[a16su]

@mxschmitt try use 1 worker npx playwright test --workers=1

[mxschmitt]

~~I can reproduce!~~

I was talking with my peers, and this is actually intended. If you want to have storage state for multiple files, you have two options:

a) set a global storageState file inside your playwright.config.ts under the "use section". b) have test.use in every test file. Having test.use inside the fixtures file is not recommended.

[rufi0h]

I have added a the following code to the beginning of each on of my test files under the import section and I still have the same results as before. test.use({ storageState: 'tests/utils/playwright/.auth/user.json' }),

I have also added it to the use section in config and have the same results as before where it only works on the first test ran after auth.

export default defineConfig({`
  reporter: [
    ['line'],
    ['junit', { outputFile: 'tests/test-results/playwright-test-results.xml' }]
  ],
  outputDir: 'tests/test-results/',
  workers: 1,
  timeout: 60 * 1000,
  expect: {
    timeout: 15000
  },
  use: {
    actionTimeout: 0,
    trace: 'on-first-retry',
    video: 'on-first-retry',
    screenshot: 'only-on-failure',
    storageState: 'tests/utils/playwright/.auth/user.json'

  },
  projects: [
    { name: 'setup', testMatch: /.*\.setup\.ts/ },
    {
      name: 'chromium',
      use: {
        ...devices['Desktop Chrome'],
        viewport: { width: 1920, height: 1080 },
        storageState: 'tests/utils/playwright/.auth/user.json'
      },
      dependencies: ['setup']
    }
  ],
});

[mxschmitt]

Since we don't have a repro for your scenario, its hard to triage it. Maybe the authentication provider you are using does not like if you re-use the sessions? you could add a console.log(await context.cookies()) at the beginning of test1 and test2 to see if any cookies are loaded. This would help to understand if the problem is that the cookies are not getting loaded or that he cookies are not getting accepted by your web-application.

[rufi0h]

Here are the two different cookies that I console logged a the beginning of the tests. the only difference is the value and expires for the ST, and ST-NO-SS cookies. these cookies are related to ping identity test 1

{
    name: 'ai_user',
    value: 'hhaflw80XfuEX24mpwFYjR|2023-06-05T18:19:21.892Z',
    domain: 'DOMAIN.com',
    path: '/',
    expires: 1717525161.893532,
    httpOnly: false,
    secure: true,
    sameSite: 'None'
  },
  {
    name: 'JSESSIONID',
    value: '2fb4656a3e3dbaa2',
    domain: '.nr-data.net',
    path: '/',
    expires: -1,
    httpOnly: false,
    secure: true,
    sameSite: 'None'
  },
  {
    name: 'ai_session',
    value: 'o4fmijq9HLzOxWayRg8DbZ|1685989163612|1685989186111',
    domain: 'DOMAIN.com',
    path: '/',
    expires: 1685990986.287656,
    httpOnly: false,
    secure: true,
    sameSite: 'None'
  },
  {
    name: 'ST',
    value: '1012149f-3778-4c74-b4de-a84e0db47ae5',
    domain: '.auth.DOMAIN.com',
    path: '/',
    expires: 1688581210.199321,
    httpOnly: true,
    secure: true,
    sameSite: 'None'
  },
  {
    name: 'ST-NO-SS',
    value: '107ad229f-3778-4c74-b4de-a84e0db47ae5',
    domain: '.auth.DOMAIN.com',
    path: '/',
    expires: 1688581210.199353,
    httpOnly: true,
    secure: true,
    sameSite: 'Lax'
  }

test 2

{
  name: 'ai_user',
  value: 'hhaflw80XfuEX24mpwFYjR|2023-06-05T18:19:21.892Z',
  domain: 'DOMAIN.com',
  path: '/',
  expires: 1717525161.893532,
  httpOnly: false,
  secure: true,
  sameSite: 'None'
},
{
  name: 'JSESSIONID',
  value: '2fb4656a3e3dbaa2',
  domain: '.nr-data.net',
  path: '/',
  expires: -1,
  httpOnly: false,
  secure: true,
  sameSite: 'None'
},
{
  name: 'ai_session',
  value: 'o4fmijq9HLzOxWayRg8DbZ|1685989163612|1685989186111',
  domain: 'DOMAIN.com',
  path: '/',
  expires: 1685990986.287656,
  httpOnly: false,
  secure: true,
  sameSite: 'None'
},
{
  name: 'ST',
  value: '5dde89e6-a534-41fe-aafc-05cd4c9209d6',
  domain: '.auth.DOMAIN.com',
  path: '/',
  expires: 1688581168.359412,
  httpOnly: true,
  secure: true,
  sameSite: 'None'
},
{
  name: 'ST-NO-SS',
  value: '5Edf29e6-a534-41fe-aafc-05cd4c9209d6',
  domain: '.auth.DOMAIN.com',
  path: '/',
  expires: 1688581168.359524,
  httpOnly: true,
  secure: true,
  sameSite: 'Lax'
}

and a screenshot comparison of the two Secure information has been redacted and guid info was slightly modified, same format

[mxschmitt]

Maybe they have some smartness in place, that does not allow to re-use it multiple times at the same time. So for the next run it gets invalidated! But just guesses, since we don't have a repro or don't know much about ping auth provider.

You could try running it with --workers=1

[rufi0h]

We got this working by adding this snippet every one of the test files

test.afterEach(async ({ page }) => {
  await page.context().storageState({ path: authFile as string });
});

The theory we came up with is that the local storage isn’t getting saved at the end of the test into the session storage and ping identity is modifying in the background to constantly refresh the security token.

[mxschmitt]

Is this your authentication provider? https://www.pingidentity.com

Yeah this means then that they invalidate the old sessions and this is not really compatible with with the concept of re-using auth for tests. Maybe you can turn this off somewhere, most likely they immediately know whats going on and have a better solution for you.

[rufi0h]

You are correct, we are using PingIdentity One as our authorization server. Because this works for now we'll continue to use this going forward, thank you for your assistance.

[mxschmitt]

Sounds good, just wanted it to verify so for future users who run into it, they will find it via search engines.