Add Pyodide support and CI jobs for Zarr

[agriyakhetarpal]

Description

This description is a stub and will be updated in due course.

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

Additional context

Here are a few xrefs to some other PRs that have been implemented for packages in the Scientific Python ecosystem:

• NumPy's Pyodide job: CI, BLD, TST: Re-enable Emscripten/Pyodide CI job for NumPy numpy/numpy#25894, by courtesy of @rgommers
• The same for PyWavelets (which is an optional dependency for scikit-image): Build and test PyWavelets Pyodide wheels in CI PyWavelets/pywt#701 and Upload dev wheels to Anaconda.org + revamp wheels publishing workflow PyWavelets/pywt#714
• For the pandas repository BLD, TST: Build and test Pyodide wheels for pandas in CI pandas-dev/pandas#57896
• Add out-of-tree Pyodide builds in CI for scikit-image scikit-image/scikit-image#7350
• and so on

The greater, long-term goal towards this change is to implement Sphinx-based interactive documentation via JupyterLite and WASM-powered, in-browser kernels, as referenced in Quansight-Labs/czi-scientific-python-mgmt#19, see also: Quansight-Labs/czi-scientific-python-mgmt#18. A pilot that can be readily tried out is available for the "API reference" pages under the PyWavelets documentation. This will be preceded by configuring a place to publish these WebAssembly/Emscripten wheels nightly or on a scheduled cadence for Zarr (which could be a third-party, PyPI-like index such as Anaconda.org) and then integrating Sphinx extensions such as jupyterlite-sphinx for hosted documentation.

[agriyakhetarpal]

Zarr v3 has crc32c and zstandard as dependencies, which were not present for v2. These packages contain compiled code and are not available in Pyodide yet (based on the logs from the workflow run). I shall try to add recipes for them Pyodide, but I am afraid I shall not have an answer for how long that can take.

[agriyakhetarpal]

Update: we added both crc32c and zstandard (please see the issues/PRs linked above). Pyodide 0.26.0 should be out very soon containing both packages plus with a bump to the Python version (to 3.12.1) and an ABI-breaking change (via Emscripten 3.1.58). I shall revisit this PR in a few days.

Next, when Zarr v3 (stable) comes out on PyPI in the next month or so, I shall bump the version/recipe for the in-tree Pyodide build as well. It would be great to have the out-of-tree build, i.e., this PR, ready for review/merging by then, and I will be committed to putting my best efforts in doing so – thanks!

[agriyakhetarpal]

The Pyodide 0.26.0 release is out, and we have been lucky to have got both crc32c and zstandard in time (but in any case, they would have got into a patch release anyway). Let me update this PR in a short moment.

[agriyakhetarpal]

I can't request a review from you as you haven't been previously involved in this PR, @ryanking13, but it would be nice if you could take a look at it as well – thank you!

[ryanking13]

How about skipping tests that require newer version of numcodecs?

[ryanking13]

Maybe we should implement it in Pyodide?

If this is only problematic in testing environment, how about moving this into conftest.py?

[agriyakhetarpal]

Maybe we should implement it in Pyodide?

I haven't thought about it, but yes, I think this should be helpful to put somewhere in pytest-pyodide, if I understand you correctly. I haven't come across this case/problem in any other downstream package, so I'm not sure (but perhaps Zarr is among the first).

If this is only problematic in testing environment, how about moving this into conftest.py?

Yes, makes sense to me! I'll move it to conftest.py and make it run only when PYTEST_CURRENT_TEST is active.

Edit: I've moved the patch to conftest.py in bad9920

[hoodmane]

I would prefer solving it in webloop rather than just in the test code if it's possible.

[agriyakhetarpal]

The tests are now passing in https://github.com/zarr-developers/zarr-python/actions/runs/15322119791/job/43108023246?pr=1903 (except the extremely slow ones – also, they don't exhibit the same level of slowness at all if you run them locally); and I've added a release note highlighting Pyodide/WebAssembly support. Codecov is going to complain because it isn't aware of this WebAssembly environment, so I'm happy to either mock it to absurdly boost the coverage value or skip those lines with a # pragma: no cover comment. Thank you!

[hoodmane]

It would be nice to make coverage work with the webassembly tests too. Though we still haven't done it for Pyodide itself.

[agriyakhetarpal]

It would be nice to make coverage work with the webassembly tests too. Though we still haven't done it for Pyodide itself.

Yes, I'd assume merely collecting or operating on coverage data should already work, as coverage and pytest-cov themselves are able to generate reports, and we can combine them outside of Codecov to fix the missing values. It doesn't work for Codecov, though.

[ryanking13]

For Pyodide-wise, it looks good to me.

[agriyakhetarpal]

@ryanking13 – I've corrected the error message based on your suggestion, thanks!

@hoodmane, surfacing your conversation from above about solving the issue in webloop.py rather than just in the test code if it's possible: I've taken a more in-depth look at this, and I think it makes sense not to do much here. The idea is that shutdown_asyncgens closes tracked async generators (or the ones it's able to track with sys.set_asyncgen_hooks() – which allows the event loop to be notified of the generators that are created, through callbacks), by calling aclose() on them during shutdown. CPython uses _asyncgen_firstiter_hook() and _asyncgen_finalizer_hook() to manage them.

Next, what I understand here is that WebLoop's lifecycle management philosophy is fundamentally different from standard asyncio loops. We're not letting pyodide.webloop manage its lifecycle deliberately because it's delegated to the browser. At most, we could add a no-op async def shutdown_asyncgens() to webloop.py, but 1. it can't stop pytest-asyncio from raising this, and 2. I've realised that pytest-asyncio is raising only a RuntimeWarning here, which pytest has been filtering to an error based on Zarr's pytest configuration (!), so there was no need to add a fixture for this – there should be no harm in ignoring the warning. Please let me know if you had something else in mind; thanks!

[ryanking13]

At most, we could add a no-op async def shutdown_asyncgens() to webloop.py, but 1. it can't stop pytest-asyncio from raising this, and 2. I've realised that pytest-asyncio is raising only a RuntimeWarning here, which pytest has been filtering to an error based on Zarr's pytest configuration (!), so there was no need to add a fixture for this – there should be no harm in ignoring the warning. Please let me know if you had something else in mind; thanks!

Thanks for the investigation. If it only raises a runtime warning, I think it is okay to leave it as-is (without the fixture) in this PR.

Apart from that, adding no-op implementation of shutdown_asyncgens in Pyodide Webloop makes sense to me, as users (not just pytest-asyncio) who might call shutdown_asyncgens will face NotImplementedError. I think no-op implementation that does nothing is better than raising an error.