Implement CLI filtering for audits and groups in plugins

[hanna-skryl]

User story

As a CLI user, I want to have more granular control over which audits and groups are executed by plugins so that I can customize my reports without modifying the configuration file for every run.

Currently, CLI supports filtering by categories and plugins through options like --onlyPlugins, --skipPlugins, --onlyCategories, and --skipCategories. By introducing finer-grained filtering for audits and groups, I will be able to run tailored reports that focus on exactly what I need.

While individual plugins are not required to implement custom filtering logic, extending plugin configurations to support these filters is recommended for better user control and consistency.

Acceptance criteria

• [ ] CLI arguments --onlyAudits, --skipAudits, --onlyGroups, and --skipGroups are parsed and passed correctly to the filtering middleware.
• [ ] Middleware identifies and filters out items (audits/groups) marked as skipped based on options passed directly to the configuration object (no regression).
• [ ] Categories with no valid references after filtering are removed from the final report.
• [ ] If an audit is referenced by multiple groups and one group is filtered out, the audit should remain in the report if it is still referenced by another group.
• [ ] The CLI should log all applied filters and list the affected audits, groups, plugins, and categories.
• [ ] Ensure backward compatibility by maintaining existing behavior when no audit or group filters are applied.

Implementation details

Middleware Extension

Extend the existing filterMiddleware to handle new CLI options (--onlyAudits, --skipAudits, --onlyGroups, --skipGroups).

Plugin Modification (optional)

Currently, only the Lighthouse plugin supports options for filtering by audits and groups, which allows early marking of skipped audits and groups. To enable similar filtering behavior in other plugins, their respective configurations may be extended to handle these options.

While this extension is optional, it is recommended as it can improve user control over which audits and groups to include in the report.

Logging and Validation

• Extend existing logging to display removed audits and groups, and plugins and categories that were emptied by audit- and group-level filters.
• Extend final state validation logic to ensure consistency and prevent empty reports.
• If all plugins are filtered out, display a warning and exit gracefully.
• If --onlyAudits and --skipAudits or --onlyGroups and --skipGroups are both provided, throw an error indicating conflicting options.

[hanna-skryl]

I don't think this solution was thoroughly thought through when originally proposed. We're stuck on some fundamental architectural issues:

1. Timing mismatch - Plugins are configured in code-pushup.config.ts before CLI arguments are parsed. By the time CLI runs, plugins are already static objects. There's no clean way to retroactively change what they should execute.

2. Performance concerns - Filtering at the middleware level means executing all audits/groups first, then throwing away results. Much more efficient to filter at the plugin level, but...

3. No standard filtering pattern - Each plugin handles filtering differently (or not at all). Some are wrappers around third-party tools that don't even support filtering at the audit level.

These issues are interconnected - we can't efficiently filter at the CLI level due to issue 2, but we also can't standardize plugin-level filtering due to issue 3. The current approach (plugin-specific config options where available) is probably the most pragmatic solution.

Should we close this as a won't-fix?