array-api-tests Incomplete tests

This is a tracking issue for tests that are currently incomplete. These are mostly notated in the code by TODO comments, but it's useful to note them all here too. I haven't checked everything throughly so if there is something missing that isn't noted in a comment it might not be listed here.

Note there are also some functions that aren't yet tested at all, but those are covered in separate issues (e.g., #248 #299). I also am not including tests that are currently skipped entirely as flaky (https://github.com/data-apis/array-api-tests/issues/300).

test_array_object.py

[ ] test_getitem_masking.py (key.ndim < x.ndim not tested)

test_creation_functions.py

The device keyword is never tested (see below)

test_data_type_functions.py

[ ] test_astype (~~no complex types~~, no values testing, no test for copy) -- gh-311
[ ] test_broadcast_arrays (no values testing)
[ ] test_broadcast_to (no values testing)
[x] test_can_cast (no complex types) -- https://github.com/data-apis/array-api-tests/pull/320
[ ] test_finfo (no values testing)
[ ] test_iinfo (no values testing)

test_fft.py

[x] test_irfftn (no shape tests) -- fixed in https://github.com/data-apis/array-api-tests/pull/309

test_has_names.py

[ ] Several names are not tested for (see https://github.com/data-apis/array-api-tests/issues/224)

test_indexing.py

[ ] test_take (negative indices, different dtypes for indices, axis=None https://github.com/data-apis/array-api-tests/issues/385)
[ ] test_take_along_axis (negative indices, different dtypes for indices, broadcast-compatible indices)

test_inspection_functions.py

Basically none of the inspection APIs are currently tested, beyond the basic signature tests. Note that for many of these functions, we may not even need to use hypothesis. Just a basic test that the function outputs the right thing and a smoke test that the data outputted is correct.

[x] https://github.com/data-apis/array-api-tests/pull/362 adds some more smoke testing of inspection capabilities.

test_linalg.py

(note: values testing here might be hard. Values testing isn't as high priority for functions that already generally work)

[ ] test_det (no values testing)
[ ] test_eigh (no values testing)
[ ] test_eigvalsh (no values testing)
[ ] test_inv (no values testing)
[ ] test_matmul (the @ __matmul__ operator methods are not tested)
[ ] test_matrix_norm (no values testing)
[ ] test_qr (no test that Q is orthonormal)
[ ] test_slogdet (no values testing)
[ ] test_svdvals (no values testing; no test that it agrees with svd())
[ ] test_trace (need to update dtype assertion, this should match test_sum)
[ ] test_vector_norm (no values testing)

Additionally, there are some limitations in the hypothesis strategies used by some of the functions. These should be made more general so that the functions are actually tested on a full gamut of possible inputs:

[ ] symmetric_matrices (does not generate complex Hermitian matrices
[ ] positive_definite_matrices (only generates stacks of identity matrices
[ ] invertable_matrices (only generates stacks of diagonal matrices)

test_manipulation_functions.py

[ ] test_tile (no values testing) -- https://github.com/data-apis/array-api-tests/pull/376

test_operators_and_elementwise_functions.py

[x] test_divide (no complex numbers testing) -- fixed in gh-306
[ ] test_negative (there is a TODO about uints)
[ ] scalar support in binary functions: most functions are only tested for a single dtype; see also https://github.com/data-apis/array-api-tests/issues/364
[ ] test_clip : no testing of an int scalar min/max and a float array

test_signatures.py

Several issues tracked in other issues: https://github.com/data-apis/array-api-tests/issues/256, https://github.com/data-apis/array-api-tests/issues/171

test_sorting_functions.py

[ ] test_argsort (no testing of signed zeros or nans)
[ ] test_sort (no testing of signed zeros or nans)

test_special_cases.py

Many special cases are not tested. This is tracked at https://github.com/data-apis/array-api-tests/issues/284

test_statistical_functions.py

See also https://github.com/data-apis/array-api-tests/issues/167

[ ] test_mean (no values testing)
[ ] test_std (no values testing)
[ ] test_var (no values testing)

test_utility_functions.py

[ ] test_diff (no values testing for n > 1)

OTHER

[ ] Tests for device support https://github.com/data-apis/array-api-tests/issues/302
[ ] Tests for dlpack

Nov 14 '24 20:11 asmeurer

test_negative (there is a TODO about uints)

The TODO is # TODO: clarify if uints are acceptable, adjust accordingly and apparently negative(uint) wraps around in at least several libraries:

In [1]: import torch

In [2]: torch.negative
Out[2]: <function torch._VariableFunctionsClass.negative>

In [3]: torch.negative(torch.as_tensor(3, dtype=torch.uint8))
Out[3]: tensor(253, dtype=torch.uint8)

In [4]: import jax.numpy as jnp

In [5]: jnp.negative(jnp.asarray(3, dtype=jnp.uint8))
Out[5]: Array(253, dtype=uint8)

In [6]: import numpy as np

In [7]: np.negative(np.asarray(3, dtype=np.uint8))
Out[7]: np.uint8(253)

In [8]: np.__version__
Out[8]: '2.1.0'

In [9]: import array_api_strict as xp

In [10]: xp.negative(xp.asarray(3, dtype=xp.uint8))
Out[10]: Array(253, dtype=array_api_strict.uint8)

The spec does not seem to fix the behavior for unsigned ints: https://data-apis.org/array-api/latest/API_specification/generated/array_api.negative.html

So maybe a first action item is to clarify the spec?

Nov 16 '24 13:11 ev-br

That sounds right. I thought I remember this already being addressed in the standard, but I guess it never was. At any rate, it's pretty clear to me that this behavior should be undefined in the standard, because we generally don't require wraparound behavior for integers to be defined. Maybe there is (or should be) something in the standard that just states that generally.

Nov 18 '24 22:11 asmeurer

Tests for DLPack are fairly high-prio in this list. It would help implementers quite a bit, since some of the logic for new keywords (max_version, dl_device, copy) is a little fiddly.

Feb 04 '25 16:02 rgommers