Make QA testing easier

[agjohnson]

trafficstars

Currently, QA testing doesn't happen consistently because it's all manual process and is mostly done locally. We also have a huge test matrix at the moment because we haven't pruned our supported dependencies in a long time. Hopefully some of our next releases will clean this up though.

const cartesian = (...a) => a.reduce((a, b) => a.flatMap(d => b.map(e => [d, e].flat())));

const variants = {
    python: [2, 3],
    sphinx: [18, 24, 35, 41],
    html: [4, 5],
    browsers: ["firefox", "chrome", "safari", "edge", "ie11"],
    viewport: ["mobile", "tablet", "desktop"],
};

const product = cartesian(
    variants.python,
    variants.sphinx,
    variants.html,
    variants.browsers,
    variants.viewport,
);

console.assert(product.length == 240);

:grimacing:

It will hopefully soon be something closer to:

const cartesian = (...a) => a.reduce((a, b) => a.flatMap(d => b.map(e => [d, e].flat())));

const variants = {
    python: [3],
    sphinx: [35, 41],
    html: [5],
    browsers: ["firefox", "chrome", "safari", "edge"],
    viewport: ["mobile", "tablet", "desktop"],
};

const product = cartesian(
    variants.python,
    variants.sphinx,
    variants.html,
    variants.browsers,
    variants.viewport,
);

console.assert(product.length == 24);

Way better!

If we're down to 24 or 12, we can easily automate tests here. Automation would be a great end goal, but even just a better story for manual testing our most important variants would be a fantastic first step.

[agjohnson]

@nienn might have some thoughts here.

One additional requirement might be screenshot output, or something reviewable by community members. We can always give community members access to our saucelab/etc account too though.

[agjohnson]

I built out a QA test site on one of our staging S3 buckets: https://readthedocs-static-dev.s3.us-east-2.amazonaws.com/qa/theme/index.html

This is currently manually updated and doesn't include PR builds. It's reachable by Saucelabs however. I see this probably be used less frequently, and maybe only for release QA testing.

The QA tox environments that I added already can serve as a preliminary, local QA test. For instance, to test a branch:

% git co agj/some-pr
% tox -e py3-sphinx41-html4
...
The HTML pages are in .tox/py3-sphinx41-html5/tmp/html.
________________ summary ________________
  py3-sphinx41-html4: commands succeeded
  congratulations :)
% firefox .tox/py3-sphinx41-html5/tmp/html/index.html