libheif icon indicating copy to clipboard operation
libheif copied to clipboard

initial version of regression test framework

Open bradh opened this issue 1 year ago • 0 comments

This is a concept draft for regression testing. It was inspired by the iden regression going unnoticed.

We do have unit testing, but it doesn't really exercise a lot of the functionality.

So this makes use of pytest (which is assumed installed - its a package on ubuntu) to do black-box testing. The main check at this stage is that heif-info produces the right result. There are a bunch of other things that could be tested.

Key decisions are:

  1. Is this worth the effort to maintain?
  2. Are the tests reliable? (I'm not sure what I have is reliable).
  3. Is there an additional / better source of test data?
  4. Do we want to do image comparisons against known-good snapshots (e.g. PNG files)

A couple of other things I noticed:

  • The output of heif-info --dump-boxes puts the MIME type, main brand and compatible brand at the start when you are using it interactively, and at the end when redirected to a file (e.g. see the last three lines in any of the .dump files).
  • The output of --help goes to stderr. That is probably OK for a mis-use, but might not be appropriate when requested, since its not really an error at this point.

Thoughts / opinions welcome.

bradh avatar Jan 18 '24 08:01 bradh