array-api-tests icon indicating copy to clipboard operation
array-api-tests copied to clipboard

Reset the max-examples default back to 100

Open asmeurer opened this issue 10 months ago • 1 comments

The tests run fast enough now that there's no need to lower this to 20.

See the discussion at #246.

asmeurer avatar Apr 19 '24 19:04 asmeurer

Could we just change our own workflow to have --hypothesis-max-examples=100, instead of the default? As defo for something like JAX this makes it basically unusable on default runs :sweat_smile:

honno avatar May 10 '24 13:05 honno

JAX is the exception, not us. If JAX is slower than most libraries, they should manually lower their max examples. Library defaults should match the common case, not the least common denominator.

asmeurer avatar May 14 '24 19:05 asmeurer

I also changed the unvectorized max examples to scale linearly with max-examples instead of logarithmically. Otherwise, it's very difficult to actually run a lot of examples for unvectorized tests. We can adjust the scaling factor if 1/10 is not small enough by default.

asmeurer avatar Jun 05 '24 22:06 asmeurer