External test data

[HeroicKatora]

This is a draft for a system that would allow external packagers to run our test suite based on the published packages, without requiring us to add that test data into the packages. This is based on a recent idea of mine which I published as xtest-data. Comment of all sorts welcome on the approach.

I'm hoping to quickly arrive at a stable version—maybe even 1.0—of the linked crate by prototyping it with image or its sub-crates. These should be a decent reflection of real world testing and we certainly have a relatively heavy test suite in terms of file sizes. One particular pain point which I haven't been able to integrate yet would be to move data out-of-tree altogether but I imagine this to be feasible with submodules—indeed, submodules should already work without requiring everyone to download all data—but more interesting would be git lfs etc.

How to run the tests

• Manually package the crate: cargo package --no-verify
• Unpack the crate at some location:

  gunzip -c target/package/image-0.23.14.crate | tar -C /tmp --extract --file -
  cd /tmp/image-0.23.14
  

• Do the dance for the crate (note that only reference_images works atm):

  # Note the error output here, no network connection yet.
  cargo test --tests check_references -- --nocapture
  # Instruct the crate to auto-setup the directories
  CARGO_XTEST_DATA_FETCH=1 cargo test --tests check_references -- --nocapture