Installable as a package?

[asmeurer]

I don't know if it would be beneficial to have the test suite be installable as a Python package (and to release it on PyPI).

For now, it is not installable. You can run the suite by cloning the repo and running. If you are an array library maintainer and having the test suite installable would make things easier for you, please let us know.

[djl11]

I would definitely find this to be the cleanest solution for keeping up to date with the latest official release of the standard in my downstream CI unit tests. However, maybe this is only useful once there are multiple released versions of the standard, with the PyPI releases aligned as such. While the standard is still in draft form I imagine it's not really needed, and cloning is the cleanest solution IMO.

[asmeurer]

Our plan is to have a single test suite that tests multiple versions of the standard. The standard version would be specified by something like an environment variable or a flag to pytest. See #20.

[asmeurer]

@djl11 how would you envision the workflow looking like for running the test suite if array-api-tests were an installable package?

[leofang]

cc: @kmaehashi

[djl11]

Our plan is to have a single test suite that tests multiple versions of the standard. The standard version would be specified by something like an environment variable or a flag to pytest. See #20.

Ah yes this makes more sense, thanks for explaining.

@djl11 how would you envision the workflow looking like for running the test suite if array-api-tests were an installable package?

I've not given this much detailed thought, but I would add the PyPI package as a requirement in my CI Dockerfile and add these tests alongside my own. One motivation for using PyPI rather than cloning would be potential improved stability of the tests. However, if the PyPI releases were based on calendar and not implementation milestones or stability then actually there might not be a huge benefit to having PyPI releases over simply cloning.

Either way I agree this is probably low priority.

[asmeurer]

I've not given this much detailed thought, but I would add the PyPI package as a requirement in my CI Dockerfile and add these tests alongside my own.

You mean you would copy the tests from the installed package in site-packages into your test suite? IMO if you are going to do this, then cloning is a better way.

One motivation for using PyPI rather than cloning would be potential improved stability of the tests.

Just to be clear, we are going to do releases by creating tags on this repo. So if you want stability, that will already be there. You would do something like

git clone https://github.com/data-apis/array-api-tests.git
cd array-api-tests
git checkout 2022.1

You can then either run the tests from the repo, or copy them into your own test suite.

The question here is whether there would be any benefit to anyone to putting the packages on PyPI in addition to the tagging.

I just made the first release tag, 2022.1, so you can start using that (CC @barry-jin).

However, if the PyPI releases were based on calendar and not implementation milestones or stability then actually there might not be a huge benefit to having PyPI releases over simply cloning.

Yes, we are using calendar versioning because every release would be expected to be "breaking", in the sense that there may be new tests added that might introduce new failures if your library doesn't conform to them.

[djl11]

This all makes total sense, thanks a lot for explaining. This is how I've done it for now, using a git submodule of my own fork of this repo in my test folder. I'll use this fork to make any PRs.

It's working well I would say! So back to the point of the issue, I don't see a huge benefit in having a PyPI package. Thanks again for the help!

[honno]

As packaging adds to the maintenance burden and there seems to no desire for this, closing (but feel free to re-open/discuss further).

[asmeurer]

I think we should still consider whether it's worth it to make a package. It's a harder process, but it's much simpler for end users, and closer to what they are used to using from other Python libraries. See also #145.