ENH/TST: `xp_assert_` enhancements #267

mdhaber · 2025-04-16T17:55:03Z

Toward #17. As discussed in mdhaber/marray#110 (comment).

This adds a few of the features from SciPy's xp_assert_ functions to the xpx xp_assert functions. Tests will probably fail in this first commit; I'll comment inline.

I decided not to add a check_namespace toggle: if check_namespace is False and the namespaces don't match, other operations might fail. I didn't want to work on fixing this if it's not an important feature. There is only one file in SciPy that uses check_namespace=False, and it probably shouldn't.

Once this is looking good, I'll add xp_assert_less as a clone of xp_assert_equal.

mdhaber

Easier to review with "Hide whitespace". @lucascolley do you see what is going on with

FAILED tests/test_testing.py::test_assert_close_equal_shape[sparse-True-xp_assert_close]
FAILED tests/test_testing.py::test_assert_close_equal_dtype[sparse-True-xp_assert_close]

?

Also, it looks to me like the codecov failures are misleading. It doesn't look to me like the new code is actually missing coverage if tests are run with all backends.

mdhaber · 2025-04-16T17:55:21Z

src/array_api_extra/_lib/_testing.py

@@ -9,13 +9,15 @@
 from types import ModuleType
 from typing import cast

+import numpy as np


Is this OK? Sometimes it was imported within test functions below.

Today it is OK, as this module is not imported automatically from the outer scope.
In the long run though, we want to move this module to public at which point it won't be a good design anymore (although it remains to be seen if any Array library in real life can achieve not to have numpy as a hard dependency...)

Yes, I think it would be fine to make these public API once they are ready, with the caveat that NumPy is required. We are really striving for minimal runtime dependencies rather than test time dependencies downstream, at least for now.

This module heavily relies on np.testing.assert* anyway.
We'll just need to add a test that import array_api_extra doesn't import numpy.

We'll just need to add a test that import array_api_extra doesn't import numpy.

Note to do this in a follow-up

mdhaber · 2025-04-16T17:56:11Z

src/array_api_extra/_lib/_testing.py

+        msg = f"dtypes do not match: {actual.dtype} != {desired.dtype}"
+        assert actual.dtype == desired.dtype, msg
+
+    if is_numpy_namespace(actual_xp) and check_scalar:


Suggested change

if is_numpy_namespace(actual_xp) and check_scalar:

if is_numpy_namespace(actual_xp) and check_arrayness:

?
I seem to remember some discussion about naming this parameter when adding to SciPy (check_0d). I don't really care what it is called. check_type is also on the table.

scalar sounds fine to me.

I think I prefer check_0d, but we can consider this in a follow-up that makes these public

src/array_api_extra/_lib/_testing.py

tests/test_testing.py

src/array_api_extra/_lib/_testing.py

lucascolley · 2025-04-16T18:41:07Z

It doesn't look to me like the new code is actually missing coverage if tests are run with all backends.

Yeah, this is a constant problem. It eventually flips to passed once it registers the backends run. I think initial investigation didn't find a solution, but there is probably some sort of config/workaround to fix it.

lucascolley · 2025-04-16T18:57:04Z

The failures are XPASSes: [XPASS(strict)] sparse:no isdtype

Judging by the message, we're expecting the test to fail due to sparse not implementing isdtype, but it is now passing. Could that be because of any of your changes?

mdhaber · 2025-04-16T21:31:56Z

That's what I figured, which is why I tried to set the xfail strict=False in param_assert_equal_close. I guess that option is not respected as written.

The trouble is that when check_shape or check_dtype is False, _check_ns_shape_dtype passes, the code continues, and then we run into the AttributeError.

        xp = _check_ns_shape_dtype(actual, desired, check_dtype, check_shape, check_scalar)
    
       floating = xp.isdtype(actual.dtype, ("real floating", "complex floating"))
       # AttributeError: module 'sparse' has no attribute 'isdtype'. Did you mean: 'astype'?

So whether the test will fail with sparse arrays due to the AttributeError depends on check_shape/check_dtype. We cannot simply use the param_assert_equal_close decorator to xfail (strict) the sparse tests, nor can we remove it.

So what would you prefer:

Mark the test with pytest.xfail inside the test function considering the value of check_shape/check_dtype (in addition to the backend)
Create some alternative to param_assert_equal_close that effectively does the same thing as the previous option (but is harder for me to write)
Make the XFAIL with sparse arrays not strict (perhaps I can figure out why strict=False was not enough?)
Just skip the test with sparse arrays
Remove the parameterization and write two separate tests just to accommodate for sparse arrays
Something else?

If it were me, I would just do option 1. I have no qualms about using pytest.xfail within a function, but since someone went through the trouble of creating param_assert_equal_close, it seems that is not desirable to everyone.

lucascolley · 2025-04-16T21:49:03Z

Multiple of those options seem fine with me (at least for now), but I'll ping @crusaderky as the original author first

hameerabbasi · 2025-04-17T17:46:03Z

The failures are XPASSes: [XPASS(strict)] sparse:no isdtype

Judging by the message, we're expecting the test to fail due to sparse not implementing isdtype, but it is now passing. Could that be because of any of your changes?

We recently released 0.16.0; I believe that may have added isdtype.

mdhaber · 2025-04-17T18:26:36Z

@hameerabbasi If so, it does not appear to be documented at https://sparse.pydata.org/en/stable/api/. Is that where I should be looking?

Also - I'm not very familiar with the sparse project - is there somewhere that summarizes the current relationship between scipy.sparse and sparse projects?

hameerabbasi · 2025-04-17T20:13:29Z

Whoops: It's in a non-standard experimental part of the codebase; not exposed to the user. My bad.

lucascolley · 2025-04-17T20:27:48Z

is there somewhere that summarizes the current relationship between scipy.sparse and sparse projects?

https://labs.quansight.org/blog/sparse-array-ecosystem is probably a good start

src/array_api_extra/_lib/_testing.py

tests/test_testing.py

src/array_api_extra/_lib/_testing.py

tests/test_testing.py

mdhaber

I'm leaving the pyright failures so you can decide how you want to deal with them. I think all of them are false positives, as usual with static type checking.

src/array_api_extra/_lib/_testing.py

tests/test_testing.py

mdhaber · 2025-04-21T17:57:07Z

tests/test_testing.py

+        else nullcontext()
+    )
+    with context:
+        func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)


NaNs pass NumPy's _close, _equal, and _less checks, so using NaNs here (and below) allows us to parametrize over all three functions.

Suggested change

func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)

# note: NaNs are handled by all 3 checks

func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)

crusaderky · 2025-04-25T15:12:23Z

Fix pyright failures and some cosmetic design tweaks: mdhaber#2

src/array_api_extra/_lib/_testing.py

crusaderky · 2025-04-25T16:41:06Z

mdhaber#3

Fix failures in data-apis#267

crusaderky · 2025-04-25T17:35:45Z

coverage is complaining about cupy being untested in CI #60
LGTM and ready to be merged

lucascolley · 2025-04-27T14:23:36Z

are you looking to add anything else to the scope of this PR @mdhaber ? If not, we can probably merge and iterate on what else is needed for SciPy/scikit-learn in future PRs

crusaderky · 2025-05-07T14:59:31Z

@mdhaber gentle ping

mdhaber · 2025-05-07T16:28:01Z

Please hold off on merge, but I will get back to this shortly.

mdhaber · 2025-05-09T10:37:20Z

Never mind, we don't have to add anything here.

lucascolley

thanks both, this is very close!

lucascolley · 2025-05-10T11:22:15Z

tests/test_testing.py

+        else nullcontext()
+    )
+    with context:
+        func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)


Suggested change

func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)

# note: NaNs are handled by all 3 checks

func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)

lucascolley · 2025-05-10T11:25:15Z

src/array_api_extra/_lib/_testing.py

@@ -9,13 +9,15 @@
 from types import ModuleType
 from typing import cast

+import numpy as np


We'll just need to add a test that import array_api_extra doesn't import numpy.

Note to do this in a follow-up

lucascolley · 2025-05-10T11:27:45Z

src/array_api_extra/_lib/_testing.py

+    actual: Array,
+    desired: Array,
+    check_dtype: bool,
+    check_shape: bool,
+    check_scalar: bool,


@mdhaber please could you add the new parameters to the docstring?

lucascolley · 2025-05-10T11:28:40Z

src/array_api_extra/_lib/_testing.py

+        msg = f"dtypes do not match: {actual.dtype} != {desired.dtype}"
+        assert actual.dtype == desired.dtype, msg
+
+    if is_numpy_namespace(actual_xp) and check_scalar:


I think I prefer check_0d, but we can consider this in a follow-up that makes these public

lucascolley · 2025-05-10T11:30:17Z

src/array_api_extra/_lib/_testing.py


    return desired_xp


-def _prepare_for_test(array: Array, xp: ModuleType) -> Array:
+def as_numpy_array(array: Array, *, xp: ModuleType) -> np.typing.NDArray[Any]:  # type: ignore[explicit-any]


@jorenham I don't want to open a can of worms by looking at the typing across everywhere this PR touches (this is very close to merge, any improvements could come in a follow-up), but is this single line idiomatic np.typing use?

lucascolley · 2025-05-10T11:32:13Z

src/array_api_extra/_lib/_testing.py

+    Parameters
+    ----------
+    x, y : Array
+        The arrays to compare according to ``x < y`` (elementwise).
+    err_msg : str, optional
+        Error message to display on failure.
+    check_dtype, check_shape : bool, default: True
+        Whether to check agreement between actual and desired dtypes and shapes
+    check_scalar : bool, default: False
+        NumPy only: whether to check agreement between actual and desired types -
+        0d array vs scalar.


check_shape is missing from the docstrings here and above and below.

WIP: xp_assert enhancements

1140a1a

mdhaber commented Apr 16, 2025

View reviewed changes

mdhaber marked this pull request as draft April 16, 2025 18:17

lucascolley changed the title ~~WIP: xp_assert_ enhancements~~ WIP: ENH: xp_assert_ enhancements Apr 16, 2025

lucascolley added enhancement New feature or request xpx.testing testing labels Apr 16, 2025

lucascolley changed the title ~~WIP: ENH: xp_assert_ enhancements~~ WIP: ENH/TST: xp_assert_ enhancements Apr 16, 2025

crusaderky reviewed Apr 21, 2025

View reviewed changes

src/array_api_extra/_lib/_testing.py Show resolved Hide resolved

crusaderky mentioned this pull request Apr 21, 2025

TST: xfail_xp_backend(strict=False) #269

Merged

crusaderky reviewed Apr 21, 2025

View reviewed changes

tests/test_testing.py Show resolved Hide resolved

mdhaber added 2 commits April 21, 2025 09:52

Merge remote-tracking branch 'upstream/main' into xp_assert_enhancements

054fba0

ENH: add xp_assert_less

d7f7549

mdhaber mentioned this pull request Apr 21, 2025

API/TST: array-ness of desired and expected values does not match in several tests #271

Open

mdhaber commented Apr 21, 2025

View reviewed changes

This was referenced Apr 24, 2025

TST/DOC: lazy_xp_backends in xp_capabilities scipy/scipy#22686

Merged

Rework prepare_for_test mdhaber/array-api-extra#1

Closed

Rework prepare_for_test mdhaber/array-api-extra#2

Merged

mdhaber and others added 2 commits April 25, 2025 09:21

Merge branch 'data-apis:main' into xp_assert_enhancements

a6d2d8c

Rework prepare_for_test (#2)

cb3c2d6

crusaderky approved these changes Apr 25, 2025

View reviewed changes

src/array_api_extra/_lib/_testing.py Show resolved Hide resolved

Fix failures in data-apis#267

7347366

Merge pull request #3 from crusaderky/xp_assert3

c80dd5e

Fix failures in data-apis#267

lucascolley added this to the 0.7.2 milestone Apr 27, 2025

lucascolley mentioned this pull request May 5, 2025

Support Array API Standard for Python. magpylib/magpylib#792

Open

mdhaber marked this pull request as ready for review May 9, 2025 10:32

lucascolley changed the title ~~WIP: ENH/TST: xp_assert_ enhancements~~ ENH/TST: xp_assert_ enhancements May 9, 2025

Merge branch 'main' into xp_assert_enhancements

f43f553

lucascolley self-requested a review May 9, 2025 12:41

lucascolley requested changes May 10, 2025

View reviewed changes

lucascolley modified the milestones: 0.7.2, 0.8.0 May 11, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH/TST: `xp_assert_` enhancements #267

ENH/TST: `xp_assert_` enhancements #267

mdhaber commented Apr 16, 2025 •

edited

Loading

mdhaber left a comment •

edited

Loading

mdhaber Apr 16, 2025

crusaderky Apr 21, 2025

lucascolley Apr 21, 2025

crusaderky Apr 21, 2025 •

edited

Loading

lucascolley May 10, 2025

mdhaber Apr 16, 2025 •

edited

Loading

crusaderky Apr 21, 2025

lucascolley May 10, 2025

lucascolley commented Apr 16, 2025 •

edited

Loading

lucascolley commented Apr 16, 2025

mdhaber commented Apr 16, 2025 •

edited

Loading

lucascolley commented Apr 16, 2025 •

edited

Loading

hameerabbasi commented Apr 17, 2025

mdhaber commented Apr 17, 2025 •

edited

Loading

hameerabbasi commented Apr 17, 2025

lucascolley commented Apr 17, 2025

mdhaber left a comment •

edited

Loading

mdhaber Apr 21, 2025

lucascolley May 10, 2025

crusaderky commented Apr 25, 2025

crusaderky commented Apr 25, 2025

crusaderky commented Apr 25, 2025

lucascolley commented Apr 27, 2025

crusaderky commented May 7, 2025

mdhaber commented May 7, 2025

mdhaber commented May 9, 2025

lucascolley left a comment

lucascolley May 10, 2025

lucascolley May 10, 2025

lucascolley May 10, 2025

lucascolley May 10, 2025

lucascolley May 10, 2025

lucascolley May 10, 2025

	if is_numpy_namespace(actual_xp) and check_scalar:
	if is_numpy_namespace(actual_xp) and check_arrayness:

	func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)
	# note: NaNs are handled by all 3 checks
	func(xp.asarray([xp.nan, xp.nan]), xp.asarray(xp.nan), check_shape=check_shape)

ENH/TST: xp_assert_ enhancements #267

Are you sure you want to change the base?

ENH/TST: xp_assert_ enhancements #267

Conversation

mdhaber commented Apr 16, 2025 • edited Loading

mdhaber left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crusaderky Apr 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mdhaber Apr 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lucascolley commented Apr 16, 2025 • edited Loading

lucascolley commented Apr 16, 2025

mdhaber commented Apr 16, 2025 • edited Loading

lucascolley commented Apr 16, 2025 • edited Loading

hameerabbasi commented Apr 17, 2025

mdhaber commented Apr 17, 2025 • edited Loading

hameerabbasi commented Apr 17, 2025

lucascolley commented Apr 17, 2025

mdhaber left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

crusaderky commented Apr 25, 2025

crusaderky commented Apr 25, 2025

crusaderky commented Apr 25, 2025

lucascolley commented Apr 27, 2025

crusaderky commented May 7, 2025

mdhaber commented May 7, 2025

mdhaber commented May 9, 2025

lucascolley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ENH/TST: `xp_assert_` enhancements #267

ENH/TST: `xp_assert_` enhancements #267

mdhaber commented Apr 16, 2025 •

edited

Loading

mdhaber left a comment •

edited

Loading

crusaderky Apr 21, 2025 •

edited

Loading

mdhaber Apr 16, 2025 •

edited

Loading

lucascolley commented Apr 16, 2025 •

edited

Loading

mdhaber commented Apr 16, 2025 •

edited

Loading

lucascolley commented Apr 16, 2025 •

edited

Loading

mdhaber commented Apr 17, 2025 •

edited

Loading

mdhaber left a comment •

edited

Loading