Assert against budgets

[BioPhoton]

https://github.com/code-pushup/cli/wiki/Glossary#budget

User story

As a developer I want to lock in my improvements. I want to be able to set budgets for every Category/Plugin/Audit. Example implementation see LHCI

Acceptance criteria

• [ ] the assertions can be defined as separate file or in the core config
• [ ] asserts the conditions in the config and exits with the appropriate status code if there were any failures
• [ ] I can set a budget for category scores
• [ ] I can set a budget for audit scores
• [ ] I get logs in the terminal about what is over budget
• [ ] I get logs in the md report about what is over budget
• [ ] Even if audit fails the report should be created & uploaded

Implementation details

assert command

Asserts the conditions in the code-pushup config and exits with the appropriate status code if there were any failures.

Options:
  --assertions               The assertions to use.
  --includePassedAssertions  Whether to include the results of passed assertions in the output.
                                                                                           [boolean]

assertions

The result of any report in code-pushup CLI can be asserted. Assertions are keyed by the plugin audit slug and follow an eslint-style format of level | [level, options]. When no options are set, the default options of {"aggregationMethod": "optimistic", "minScore": 1} are used.

export default = {
    // ...
    assert: {
        assertions: {
            "lighthouse:first-contentful-paint": "off",
            "eslint": ["warn", { minScore: 1 }],
            "package-json:type": ["warn", { maxLength: 0 }],
            "file-size:file-size-unmodified": ["error", { maxValue: 300 }]
        }
    }
};

Types

export interface CoreConfig {
    // ...
    assert: {
        assertions: AssertionsConfig;
    };
}

export interface AssertionsConfig {
    [key: string]: AssertionLevel | AssertionConfig;
}

type AssertionLevel = 'off' | 'warn' | 'error';

type AssertionConfig = [AssertionLevel, AssertionOptions?];

interface AssertionOptions {
    minScore?: number;
    maxLength?: number;
    maxSize?: number;
}

const AUDIT_TYPE_VALUE_GETTERS = {
  auditRan: result => (result === undefined ? 0 : 1),
  minScore: result => {
    if (typeof result.score === 'number') return result.score;
    return undefined;
  },
  maxLength: result => (result.details && result.details.items && result.details.items.length) || 0,
  maxValue: result => result.value,
};

const AUDIT_TYPE_OPERATORS = {
  auditRan: {operator: '==', passedFn: (actual, expected) => actual === expected},
  minScore: {operator: '>=', passedFn: (actual, expected) => actual >= expected},
  maxLength: {operator: '<=', passedFn: (actual, expected) => actual <= expected},
  maxValue: {operator: '<=', passedFn: (actual, expected) => actual <= expected},
};