diff --git a/spec/design_topics/C_API.md b/spec/design_topics/C_API.md deleted file mode 100644 index 96575ba63..000000000 --- a/spec/design_topics/C_API.md +++ /dev/null @@ -1,93 +0,0 @@ -(C-API)= - -# C API - -Use of a C API is out of scope for this array API, as mentioned in {ref}`Scope`. -There are a lot of libraries that do use such an API - in particular via Cython code -or via direct usage of the NumPy C API. When the maintainers of such libraries -want to use this array API standard to support multiple types of arrays, they -need a way to deal with that issue. This section aims to provide some guidance. - -The assumption in the rest of this section is that performance matters for the library, -and hence the goal is to make other array types work without converting to a -`numpy.ndarray` or another particular array type. If that's not the case (e.g. for a -visualization package), then other array types can simply be handled by converting -to the supported array type. - -```{note} - -Often a zero-copy conversion to `numpy.ndarray` is possible, at least for CPU arrays. -If that's the case, this may be a good way to support other array types. -The main difficulty in that case will be getting the return array type right - however, -this standard does provide a Python-level API for array construction that should allow -doing this. A relevant question is if it's possible to know with -certainty that a conversion will be zero-copy. This may indeed be -possible, see {ref}`data-interchange`. -``` - -## Example situations for C/Cython usage - -### Situation 1: a Python package that is mostly pure Python, with a limited number of Cython extensions - -```{note} - -Projects in this situation include Statsmodels, scikit-bio and QuTiP -``` - -Main strategy: documentation. The functionality using Cython code will not support other array types (or only with conversion to/from `numpy.ndarray`), which can be documented per function. - - -### Situation 2: a Python package that contains a lot of Cython code - -```{note} - -Projects in this situation include scikit-learn and scikit-image -``` - -Main strategy: add support for other array types _per submodule_. This keeps it manageable to explain to the user which functionality does and doesn't have support. - -Longer term: specific support for particular array types (e.g. `cupy.ndarray` can be supported with Python-only code via `cupy.ElementwiseKernel`). - - -### Situation 3: a Python package that uses the NumPy or Python C API directly - -```{note} - -Projects in this situation include SciPy and Astropy -``` - -Strategy: similar to _situation 2_, but the number of submodules that can support all array types may be limited. - - -## Device support - -Supporting non-CPU array types in code using the C API or Cython seems problematic, -this almost inevitably will require custom device-specific code (e.g., CUDA, ROCm) or -something like JIT compilation with Numba. - - -## Other longer-term approaches - -### Further Python API standardization - -There may be cases where it makes sense to standardize additional sets of -functions, because they're important enough that array libraries tend to -reimplement them. An example of this may be _special functions_, as provided -by `scipy.special`. Bessel and gamma functions for example are commonly -reimplemented by array libraries. This may avoid having to drop into a -particular implementation that does use a C API (e.g., one can then rely on -`arraylib.special.gamma` rather than having to use `scipy.special.gamma`). - - -### HPy - -[HPy](https://github.com/hpyproject/hpy) is a new project that will provide a higher-level -C API and ABI than CPython offers. A Cython backend targeting HPy will be provided as well. - -- Better PyPy support -- Universal ABI - single binary for all supported Python versions -- Cython backend generating HPy rather than CPython code - -HPy isn't quite ready for mainstream usage today, but once it does it may -help make supporting multiple array libraries or adding non-CPU device -support to Cython more feasible. diff --git a/spec/design_topics/C_API.rst b/spec/design_topics/C_API.rst new file mode 100644 index 000000000..ec2b721f8 --- /dev/null +++ b/spec/design_topics/C_API.rst @@ -0,0 +1,94 @@ +.. _C-API: + +C API +===== + +Use of a C API is out of scope for this array API, as mentioned in :ref:`Scope`. +There are a lot of libraries that do use such an API - in particular via Cython code +or via direct usage of the NumPy C API. When the maintainers of such libraries +want to use this array API standard to support multiple types of arrays, they +need a way to deal with that issue. This section aims to provide some guidance. + +The assumption in the rest of this section is that performance matters for the library, +and hence the goal is to make other array types work without converting to a +``numpy.ndarray`` or another particular array type. If that's not the case (e.g. for a +visualization package), then other array types can simply be handled by converting +to the supported array type. + +.. note:: + Often a zero-copy conversion to ``numpy.ndarray`` is possible, at least for CPU arrays. + If that's the case, this may be a good way to support other array types. + The main difficulty in that case will be getting the return array type right - however, + this standard does provide a Python-level API for array construction that should allow + doing this. A relevant question is if it's possible to know with + certainty that a conversion will be zero-copy. This may indeed be + possible, see :ref:`data-interchange`. + + +Example situations for C/Cython usage +------------------------------------- + +Situation 1: a Python package that is mostly pure Python, with a limited number of Cython extensions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + Projects in this situation include Statsmodels, scikit-bio and QuTiP + +Main strategy: documentation. The functionality using Cython code will not support other array types (or only with conversion to/from ``numpy.ndarray``), which can be documented per function. + + +Situation 2: a Python package that contains a lot of Cython code +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + Projects in this situation include scikit-learn and scikit-image + +Main strategy: add support for other array types *per submodule*. This keeps it manageable to explain to the user which functionality does and doesn't have support. + +Longer term: specific support for particular array types (e.g. ``cupy.ndarray`` can be supported with Python-only code via ``cupy.ElementwiseKernel``). + + +Situation 3: a Python package that uses the NumPy or Python C API directly +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + Projects in this situation include SciPy and Astropy + +Strategy: similar to *situation 2*, but the number of submodules that can support all array types may be limited. + + +Device support +-------------- + +Supporting non-CPU array types in code using the C API or Cython seems problematic, +this almost inevitably will require custom device-specific code (e.g., CUDA, ROCm) or +something like JIT compilation with Numba. + + +Other longer-term approaches +---------------------------- + +Further Python API standardization +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +There may be cases where it makes sense to standardize additional sets of +functions, because they're important enough that array libraries tend to +reimplement them. An example of this may be *special functions*, as provided +by ``scipy.special``. Bessel and gamma functions for example are commonly +reimplemented by array libraries. This may avoid having to drop into a +particular implementation that does use a C API (e.g., one can then rely on +``arraylib.special.gamma`` rather than having to use ``scipy.special.gamma``). + +HPy +~~~ + +`HPy `_ is a new project that will provide a higher-level +C API and ABI than CPython offers. A Cython backend targeting HPy will be provided as well. + +- Better PyPy support +- Universal ABI - single binary for all supported Python versions +- Cython backend generating HPy rather than CPython code + +HPy isn't quite ready for mainstream usage today, but once it does it may +help make supporting multiple array libraries or adding non-CPU device +support to Cython more feasible.