POC of using Happo for visual diff testing

[agjohnson]

We talked about setting up Happo for visual diff testing, this was a manual test that I did to demo the service. I'm happy with the output so far, it does about what I expected. This is using the most basic version of setting up the diff, which is just using a full page screenshot to produce the diff.

It's not clear how useful the diff will be as differences accumulate -- for example can we disregard known differences between html4 and html5 and those differences will be disregarded in future diffs too? The nature of the full page diff might not allow this.

Next steps are to testing out Happo against at least the RTD PR build. It would be most correct to test against our various test cases, but pretty soon that will be limited to Sphinx 3-4, docutils latest, and HTML 5 anyways.

An example diff is here: https://happo.io/a/728/p/957/compare/dev-75690771cac0c0aece71/dev-69c311fa8d5cc9ccaabf?

[agjohnson]

[nienn]

@agjohnson

can we disregard known differences between html4 and html5 and those differences will be disregarded in future diffs too?

This requires further consideration on how will the process be with our specific setup, but I believe we can: https://docs.happo.io/docs/ignoring-diffs

Edit: I'm not sure how that will work thought. If you ignore an element because it's rendered differently between html4 and html5 will that element then never be checked again for other new differences?

[agjohnson]

will that element then never be checked again for other new differences

Yeah, this is certainly the big question for me. I suppose we'll find out! I think we'll put a basic CI test of this on my next sprint, while we're still in demo mode on happo