Axe Plugin - Authentication Support

[hanna-skryl]

User story

As a Code PushUp user, I want to test accessibility of authenticated pages using axe-core so that I can ensure my login-protected web applications meet WCAG compliance requirements.

Acceptance criteria

Setup script configuration

• [ ] The plugin configuration accepts an optional setupScript field that specifies a path to a setup script file
• [ ] The path can be relative (resolved from process.cwd()) or absolute
• [ ] The main plugin config schema is extended to include an optional setupScript field:

const axePluginConfigSchema = z.union([
  z.url(),
  z.array(z.url()),
  z.record(z.url(), z.number()),
]);

const axePluginOptionsSchema = z.object({
  preset: z
    .enum(['wcag21aa', 'wcag22aa', 'best-practice', 'all'])
    .default('wcag21aa'),
  setupScript: z.string().min(1).optional(),
  scoreTargets: pluginScoreTargetsSchema.optional(),
});

• [ ] Invalid configurations (empty string, non-string value) are rejected by Zod validation with descriptive error messages

Setup script contract

• [ ] The setup script must export a default async function with signature (page: Page) => Promise<void> where Page is imported from playwright-core
• [ ] The function receives a Playwright Page instance that has been initialized but not yet navigated to the target URL
• [ ] The setup script can perform any Playwright operations, including navigation, form filling, clicking, waiting for elements, cookie manipulation, localStorage access, and multi-tab handling

Script execution flow

• [ ] When the runner executes with a setupScript, it launches a headless Chromium browser using chromium.launch() from playwright-core and creates a new page
• [ ] The runner resolves the script path using join(process.cwd(), setupScript) to get an absolute path
• [ ] The runner dynamically imports the setup script using the resolved path
• [ ] When the setup script file is not found, the import fails and the runner throws an error with the message: Setup script not found: ${setupScript}
• [ ] When the imported module has no default export or the default export is not a function, the runner throws an error with the message: Setup script must export a default function: ${setupScript}
• [ ] The runner calls the setup script's default export with the page instance and awaits completion
• [ ] If the setup script function throws an error, the runner catches it and throws a new error with the message: Setup script execution failed: ${originalErrorMessage}
• [ ] After setup completes successfully, the runner navigates to each configured URL and runs axe analysis using AxeBuilder from @axe-core/playwright
• [ ] The browser session is maintained across all URL analyses to preserve authentication state (cookies, localStorage, etc.)
• [ ] The browser closes only after all URLs have been analyzed, regardless of success or failure

Error handling

• [ ] All runtime errors are thrown as regular Error objects with descriptive messages
• [ ] Zod validation errors are handled automatically by the schema validation layer
• [ ] Error messages include context to help users debug issues:
  • File not found: includes the path that was attempted
  • Missing export: includes the path to the problematic file
  • Execution failure: includes the original error message from the setup script
  • Navigation timeout: includes the URL that failed to load

Unit tests

• [ ] Configuration validation tests verify that valid setup script paths are accepted and empty strings are rejected
• [ ] Mock tests verify the setup script is dynamically imported using the resolved path
• [ ] Mock tests verify the setup script's default export is called with the page instance before navigation to target URLs
• [ ] Mock tests verify the browser session is maintained across multiple URL analyses
• [ ] Error handling tests verify appropriate errors are thrown for missing files, missing default export, non-function export, and setup script execution failures

E2E tests

• [ ] A test fixture exists with the following structure:

e2e/plugin-axe-e2e/mocks/fixtures/auth-setup/
├── test-app/
│ ├── login.html           Login form
│ └── protected.html       Protected page (requires auth cookie)
├── axe-setup-script.ts    Playwright script that logs in
└── code-pushup.config.ts  Config using axe plugin with setupScript

• [ ] The protected.html file contains accessibility violations that can only be detected after authentication
• [ ] The axe-setup-script.ts navigates to login.html, fills the form, submits it, and waits for successful authentication
• [ ] The code-pushup.config.ts uses the axe plugin with setupScript: './axe-setup-script.ts' pointing to the authentication script
• [ ] The E2E test starts a local HTTP server that serves the test app and validates authentication via cookies
• [ ] When the collect command runs, it executes the setup script, successfully authenticates, navigates to the protected page, and analyzes it with axe
• [ ] The generated report.json contains audit results from the authenticated protected page
• [ ] The test verifies that without the setup script, the protected page would not be accessible (returns 401 or redirects to login)

Documentation

• [ ] The README.md includes a "Testing authenticated pages" section with setup script examples
• [ ] Code examples are provided for common scenarios:
  • Basic username/password login with environment variables
  • OAuth/SSO flow (e.g., GitHub login)
  • Multi-step navigation to nested authenticated pages
  • Testing multiple authenticated pages with a shared session
• [ ] Documentation includes environment variable setup instructions for local development (.env file), GitHub Actions (secrets configuration), and GitLab CI (variables configuration)
• [ ] The README clarifies that setup scripts use Playwright API for authentication, then @axe-core/playwright runs the axe analysis

Implementation details

Basic file structure

packages/plugin-axe/
├── src/
│ ├── lib/
│ │ ├── axe-plugin.ts
│ │ ├── types.ts
│ │ ├── constants.ts
│ │ ├── config.ts                    Add setupScript to schema
│ │ ├── runner/
│ │ │ ├── runner.ts                  Update to handle setupScript
│ │ │ ├── execute-axe.ts
│ │ │ ├── setup.ts                   NEW: Setup script execution
│ │ │ └── transform.ts
│ │ ├── meta/
│ │ │ ├── rules.ts
│ │ │ └── groups.ts
│ │ └── utils.ts
│ └── index.ts
├── README.md
├── package.json
└── vitest.unit.config.ts

e2e/plugin-axe-e2e/
├── tests/
│ ├── collect.e2e.test.ts
| └── collect-with-auth.e2e.test.ts  NEW: Auth-specific E2E test
├── mocks/
│ └── fixtures/
|     ├── auth-setup/
|     |  ├── test-app/
|     |  |   ├── login.html
|     |  │   └── protected.html
│     |  ├── axe-setup-script.ts
│     |  └── code-pushup.config.ts
│     └── default-setup/
│        ├── code-pushup.config.ts
│        └── test-app.html
└── vitest.e2e.config.ts

Documentation examples

// axe-setup.ts
import type { Page } from 'playwright-core';

export default async function setup(page: Page): Promise<void> {
  // Perform authentication using Playwright API
  await page.goto('https://example.com/login');
  await page.fill('[name="username"]', process.env['TEST_USERNAME']);
  await page.fill('[name="password"]', process.env['TEST_PASSWORD']);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard');
}

// code-pushup.config.ts
import axePlugin from '@code-pushup/axe-plugin';

// basic
export default {
  plugins: [await axePlugin('https://example.com')],
};

// advanced
export default {
  plugins: [
    await axePlugin(
      [
        'https://example.com/dashboard',
        'https://example.com/profile',
        'https://example.com/settings'
      ],
      {
        preset: 'wcag21aa',
        scoreTargets: {
          'no-autoplay-audio': 0
        },
        setupScript: './axe-setup.ts'
      }
    )
  ],
  categories: [
    {
      slug: 'critical-accessibility',
      title: 'Critical Accessibility',
      scoreTarget: 1,
      refs: [
        { plugin: 'axe', slug: 'color-contrast', weight: 1 },
        { plugin: 'axe', slug: 'image-alt', weight: 1 },
        { plugin: 'axe', slug: 'button-name', weight: 1 }
      ]
    }
  ]
}

[matejchalk]

Again, very well documented. 👍

A few suggestions:

1. The main plugin config schema is extended to include an optional setupScript field

   After setup completes successfully, the runner navigates to each configured URL and runs axe analysis

   1 script for all URLs might not be flexible enough for some scenarios. Specific pages may require their own interactions to reach some common application state - e.g., opening a quick search modal on the homepage. Apps may also have different content per user (e.g. based on access level), so may require different logins per URL.

   It would still be useful to have 1 script applied to all URLs, but maybe we could combine that with an option to define a script per URL as well? Perhaps something like:

   axePlugin({
     'https://example.com/': { setupScript: './home-setup.ts', weight: 2 },
     'https://example.com/admin': { setupScript: './admin-setup.ts', weight: 1 },
   }, { setupScript: './base-setup.ts' })
   

2. Documentation examples

   I'm missing a simple Accessibility category configuration in these examples. The only category example is a custom Critical accessibility category, which is fine as an advanced example, but I would always show the simple "where most users should start" example first.

[hanna-skryl]

I'm missing a simple Accessibility category configuration in these examples. The only category example is a custom Critical accessibility category, which is fine as an advanced example, but I would always show the simple "where most users should start" example first.

The easiest approach is to create a mergeAxeCategories() function similar to the one Lighthouse implements and expose it to users. It will automatically generate an "Accessibility" category comprising the audits/groups returned based on the selected preset, and if needed, handle multiple URLs.

Something like this:

const axe = await axePlugin('https://example.com', { preset: 'wcag21aa' });

export default {
  plugins: [axe],
  categories: mergeAxeCategories(axe),
};

[matejchalk]

The easiest approach is to create a mergeAxeCategories() function similar to the one Lighthouse implements and expose it to users. It will automatically generate an "Accessibility" category comprising the audits/groups returned based on the selected preset, and if needed, handle multiple URLs.

A helper function for merging multiple URLs will definitely be needed, yes. I would approach this like we did for the Lighthouse plugin.

One minor difference is that, unlike Lighthouse, Axe results should typically only map to 1 high-level category. So I'd consider adjusting the helper function accordingly. 🤔

// original - looks like it could be multiple categories 😕 
export default {
  plugins: [axe],
  categories: mergeAxeCategories(axe),
};

// 💡 something like this is explicitly singular
export default {
  plugins: [axe],
  categories: [mergeAxeCategory(axe)],
};

// 💡 alternatively (or additionally), we could provide only the refs
// more advanced, but could be useful for a category combining other plugins 🤔 
export default {
  plugins: [axe, lighthouse, eslint],
  categories: [
    {
      slug: 'accessibility',
      title: 'Accessibility',
      refs: [
        ...mergeAxeGroupRefs(axe, 10), // 👈 possible weight multiplier?
        { type: 'group', plugin: 'lighthouse', slug: 'accessibility', weight: 3 },
        { type: 'group', plugin: 'eslint', slug: 'angular-eslint-template-a11y', weight: 1 },
      ],
    }
  ],
};

[hanna-skryl]

@matejchalk Thanks for the suggestion! I completely agree.

I'm currently working on the core Axe plugin functionality (#1134) and have extracted shared multi-URL handling logic to utilities. Between doing that work and reading through this discussion, I realized category merging really deserves its own focused issue. I've created #1139 to track it.

It makes perfect sense that the ideal solution would support aggregating accessibility categories across multiple plugins, not just Axe alone, but that's a more complex problem. I'd implement basic Axe category aggregation first, and then we could discuss how to aggregate a category from multiple plugins (I'm not sure it has been attempted before).