augraphy icon indicating copy to clipboard operation
augraphy copied to clipboard

Add archetype pipelines, more tests, and make pipelines directly callable

Open proofconstruction opened this issue 3 years ago • 1 comments

This PR:

  • adds the archetype pipelines mentioned in #150 to augraphy.default
  • adds more tests to run those pipelines against random images
  • makes pipelines return augmented images directly

proofconstruction avatar Feb 03 '22 00:02 proofconstruction

Keeping this a draft until we resolve this.

proofconstruction avatar Feb 03 '22 00:02 proofconstruction

@proofconstruction Do you think this approach to building out a test suite still makes the most sense?

Two directions on testing I see are:

  1. Visual Testing / Image Snapshot Drift (see Percy Visual Testing as an example) where the rendered output is compared to a baseline image and SSIM or similar metric is used to ensure not too much drift occurs
  2. Benchmark Testing where certain activities are measured based on activities performed on the output are tracked to see if the quantitative measures drift, similar to visual testing but focused on task-oriented metrics

IMO, we should probably have a snapshot image for each augmentation run in isolation, and possibly a few in combination, where we track drift from a committed baseline sample image. Is this basically where you were headed with this branch?

Running notebooks to ensure they don't bomb seems like a valid approach as well, but I think that approach is captured in our Shabby Page of the Day daily build process.

jboarman avatar Feb 18 '23 22:02 jboarman