Versioning support, bulk of complex testing #165

honno · 2023-01-27T12:09:12Z

Resolves #20 with the introduced marker min_version(api_version), so tests only run when specified with a later version. The env var ARRAY_API_TESTS_VERSION allows one to specify a version to test with, which currently defaults to xp.__array_api_version__ and fallsback to "2021.12".

Covers complex testing for extended and new functions in the 2022.12 spec. Does not cover outside of that yet, nor complex special cases—something for a future PR.

Realise I need to update Hypothesis first so it recognizes 2022.12 before I'd want this merged.

asmeurer · 2023-01-27T22:06:21Z

array_api_tests/__init__.py

+__all__ = ["COMPLEX_VER", "api_version", "xps"]
+
+
+COMPLEX_VER: str = "2022.12"


I don't like this. There's more that was added in 2022 than just complex numbers.

Why not convert the version string to a tuple of numbers, so you can just compare like if version >= (2022,).

Ah this is just meant to represent the version where complex numbers were added, not some be-end-all. It existed as a variable for historic reasons, as it was previously "draft" and then something I'd just updated, but really I can grep-replace instances of it to "2022.12" now

I would rather just use inline version checks with the actual version. That's much more explicit to anyone reading the test code. We can include links to the spec changelog or versionadded admonitions as references if needed.

I would rather just use inline version checks with the actual version. That's much more explicit to anyone reading the test code. We can include links to the spec changelog or versionadded admonitions as references if needed.

Pushed a change which represents what I was going for, which I think is what you're saying.

* Update test_sign for complex inputs * Use result dtype for res_type inference in assert-against-refimpl utils * Test `xp.real()` and `xp.imag()`

asmeurer · 2023-02-23T23:31:58Z

Does hypothesis use the api_version keyword to __array_namespace__? https://data-apis.org/array-api/draft/API_specification/generated/array_api.array.__array_namespace__.html#array_api.array.__array_namespace__ (not that anything even has that outside of numpy.array_api)?

asmeurer · 2023-02-23T23:39:02Z

I tried running this on data-apis/array-api-compat#14 and there were several failures (outside of the expected XFAILs).

FAILED array_api_tests/test_creation_functions.py::test_asarray_arrays - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_creation_functions.py::test_full_like - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_creation_functions.py::test_ones_like - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_creation_functions.py::test_zeros_like - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_data_type_functions.py::test_result_type - TypeError: result_type() received an invalid combination of arguments - got (torch.dtype, torch.dtype), but expected one of:
FAILED array_api_tests/test_linalg.py::test_matrix_transpose - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_linalg.py::test_vecdot - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_manipulation_functions.py::test_concat - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_manipulation_functions.py::test_stack - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_add[add(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_add[__add__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_add[__iadd__(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_add[__iadd__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_divide[divide(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_divide[__truediv__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_divide[__itruediv__(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_divide[__itruediv__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_equal[equal(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_equal[__eq__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_log1p - AssertionError: out=inf, but should be roughly log1p(x)=87.49823360321435 [log1p()]
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_multiply[multiply(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_multiply[__mul__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_multiply[__imul__(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_multiply[__imul__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_not_equal[not_equal(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_not_equal[__ne__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_pow[pow(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_pow[__pow__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_pow[__ipow__(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_pow[__ipow__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_remainder[__mod__(x1, x2)] - AssertionError: out.dtype=int8, but should be int32 [__mod__(int32, int8)]
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_subtract[subtract(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_subtract[__sub__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_subtract[__isub__(x1, x2)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_operators_and_elementwise_functions.py::test_subtract[__isub__(x, s)] - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch
FAILED array_api_tests/test_searching_functions.py::test_where - hypothesis.errors.InvalidArgument: dtype=torch.complex64 not recognised in array_api_compat.torch

(there might be more too since the known xfails were all skipped).

honno · 2023-02-24T11:43:28Z

Does hypothesis use the api_version keyword to __array_namespace__?

Nope. It uses __array_namespace__() just for a throaway check is all.

several failures

Thanks for the report, I've identified a few issues before we can get this in. I'll prioritise resolving them so hopefully end of next week everythings up-to-speed.

Also ignore very large distances

honno · 2023-03-13T20:16:27Z

Thanks for the report, I've identified a few issues before we can get this in. I'll prioritise resolving them so hopefully end of next week everythings up-to-speed.

I should of mentioned last week I think I've addressed any blocking issues @asmeurer. So I'll continue to work on this branch, but if things look good to you it's okay to merge as-is on my end.

asmeurer · 2023-03-13T22:17:05Z

Let's merge this. I'll open an issue if I notice any issues with testing the compat library.

asmeurer reviewed Jan 27, 2023

View reviewed changes

honno added 11 commits February 14, 2023 11:02

Rudimentary complex testing for unary elwise functions

82e6312

Define elwise filters only for component dtypes

7a1e48e

Complex testing for all elwise funcs

5889a7e

TODOs for test_divide

ff865bc

Loose assertion of infinities to very large floats

9f03c92

Complex-related updates

1ba2efd

* Update test_sign for complex inputs * Use result dtype for res_type inference in assert-against-refimpl utils * Test `xp.real()` and `xp.imag()`

test_conj

2d6b2d8

min_version() marker and other versioning nicities

f5723ad

Try inferring api_version from xp.__array_api_version__

f34bbf5

Bump Hypothesis to >=6.68.0

e1d56a0

Remove COMPLEX_VER

03b735f

honno force-pushed the complex-support branch from dfef8b4 to 03b735f Compare February 14, 2023 11:17

honno marked this pull request as ready for review February 14, 2023 11:20

honno requested a review from asmeurer February 21, 2023 11:00

hypothesis_helper dtype strats just alias xps strats

78c57d0

honno added 11 commits February 27, 2023 11:21

Move oneway strategies to hypothesis_helpers.py

84bd3ef

Use cmath where obvious

a9506a8

Stop testing complex in test_arange

12c3aa2

Also ignore very large distances

Remove unnecessary use of hh.shared_dtypes() in test_empty

74101de

Support complex in test_full, complex dtype utilities

b41d447

Skip even not-so-very-large distances in test_linspace

90e7837

Skip testing complex dtypes in test_data_type_functions.py for now

05f2cf9

Skip testing complex numbers in test_linalg.py for now

8922b83

Remove debug print statement in test_manipulation_functions.py

33a0f6c

Change dtype helpers behaviour depending on api_version

d2267e4

Update type hints relating to complex

ef0e3b1

asmeurer mentioned this pull request Mar 6, 2023

Add functions such as take to existing Array API spec if not implemented yet? data-apis/array-api-compat#23

Closed

asmeurer merged commit a533680 into data-apis:master Mar 13, 2023

honno deleted the complex-support branch February 28, 2024 13:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Versioning support, bulk of complex testing #165

Versioning support, bulk of complex testing #165

honno commented Jan 27, 2023

asmeurer Jan 27, 2023

honno Jan 27, 2023

asmeurer Jan 27, 2023

honno Feb 14, 2023

asmeurer commented Feb 23, 2023

asmeurer commented Feb 23, 2023

honno commented Feb 24, 2023

honno commented Mar 13, 2023

asmeurer commented Mar 13, 2023

		__all__ = ["COMPLEX_VER", "api_version", "xps"]


		COMPLEX_VER: str = "2022.12"

Versioning support, bulk of complex testing #165

Versioning support, bulk of complex testing #165

Conversation

honno commented Jan 27, 2023

asmeurer Jan 27, 2023

Choose a reason for hiding this comment

honno Jan 27, 2023

Choose a reason for hiding this comment

asmeurer Jan 27, 2023

Choose a reason for hiding this comment

honno Feb 14, 2023

Choose a reason for hiding this comment

asmeurer commented Feb 23, 2023

asmeurer commented Feb 23, 2023

honno commented Feb 24, 2023

honno commented Mar 13, 2023

asmeurer commented Mar 13, 2023