array-api-tests icon indicating copy to clipboard operation
array-api-tests copied to clipboard

Tracker: add "repro_snippets" to failing tests

Open ev-br opened this issue 1 month ago • 0 comments

Hypothesis reported failures are sometimes hard to diagnose, even though our assertions go to incredible lengths to give useful diagnostics. It is often not easy to reproduce an exact set of parameters which trigger a test failure. As suggested in https://github.com/data-apis/array-api-tests/issues/379, we want to emit on failure the exact inputs which triggered it. [1]

A typical test case looks like this:

@given(arg=hypothesis_strategy)
def test_something(arg):
    arg2 = further_hypothesis_magic

    out = xp.function_under_test(arg1, arg2)
    assertions_on_out

and a "low-tech" solution is to convert it into

@given(arg=hypothesis_strategy)
def test_something(arg):
    arg2 = further_hypothesis_magic

    repro_snippet = f"xp.function_under_test({arg1!r}, {arg2!r})"   # exactly the r.h.s. of `out = ` below
    try:
        # as before
        out = xp.function_under_test(arg1, arg2)
        assertions_on_out
    except Exception as exc:
       # attach a failing snippet and reraise
       exc.add_note(repro_snippet)
       raise

See https://github.com/data-apis/array-api-tests/pull/384 for a worked example.

[1] gh-379 gives two ideas: declare "dependencies" of each test, and emit a "reproducible snippet" on failure. This issue only tracks progress towards the latter.


The plan is transform all tests in this manner:

  • a test module per commit
  • list the commits in .git-blame-ignore-revs for git blame to skip the whitespace changes.

Here's the tracker:

  • [ ] test_array_object.py
  • [ ] test_creation_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_data_type_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_fft.py
  • [ ] test_indexing_functions.py
  • [ ] test_linalg.py
  • [x] test_manipulation_functions.py -- gh-384
  • [ ] test_operators_and_elementwise_functions.py
  • [ ] test_searching_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_set_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_sorting_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_special_cases.py
  • [ ] test_statistical_functions.py -- https://github.com/data-apis/array-api-tests/pull/392
  • [ ] test_utility_functions.py

These test modules are different enough to not need "repro snippets", as their use of hypothesis is minimal to not at all:

  • test_signatures.py : does not use hypothesis
  • test_inspection_functions.py: does not use hypothesis
  • test_has_names.py
  • test_constants.py

cc @cbourjau @mtsokol

ev-br avatar Oct 27 '25 13:10 ev-br