dangerzone icon indicating copy to clipboard operation
dangerzone copied to clipboard

Verify PDF output of CI tests

Open apyrgio opened this issue 2 years ago • 1 comments

We currently assume that CI tests work because they complete successfully. However, we don't actually validate their output, and we may miss PDFs that have been converted, but are corrupted (see https://github.com/freedomofpress/dangerzone/issues/315).

In order to improve this situation, since the input is always known, we should also include the output in our tests folder, and compare it with that. The problem here is that we can't perform a byte to byte comparison. We need some sort of visual diffing, and ensure that the output PDF is not significantly different from the input PDF.

apyrgio avatar Jan 26 '23 16:01 apyrgio

TIL that this is sometimes called Pixel Tests (see this in PDFium).

deeplow avatar Nov 08 '23 09:11 deeplow