ENH: Please provide "universal2" wheels for macOS

[ronaldoussoren]

Is your feature request related to a problem?

There is a, currently formally experimental, "universal2" installer for Python on www.python.org. This natively supports both x86_64 (macOS 10.9 or later) and arm64 (macOS 11.0 or later). Using pandas with this installer currently requires building pandas from source, not from a binary wheel.

Describe the solution you'd like

A binary wheel on PyPI for the "universal2" installer, next to the "x86_64" wheel there already is.

In the longer run I'd love to see only "universal2" wheels and no "x86_64" wheels, but that at the very least requires time for support in the packaging system to further mature (support for "universal2" in pip and wheel is pretty new at this point).

[joshburt]

+1 for this request. I've been able to get many of the libraries needed for machine learning up and working natively. Would really like to have pandas too.

Thanks everyone!

[ronaldoussoren]

FWIW, the hard part of installing pandas was installing numpy (see numpy/numpy#18143 for a similar issue in the numpy repository), compiling pandas itself from source is not that hard.

And a show stopper for providing "universal2" or even "arm64" wheels is the availability of relevant hardware in the various cloud CI systems. That will improve over time, but AFAIK none of the large cloud CI providers have M1 runners.

[jorisvandenbossche]

Indeed, we use more or less the same CI infrastructure as numpy (https://github.com/MacPython/pandas-wheels), so will also be dependent on cloud support for being to build wheels.

[lithomas1]

Hi @ronaldoussoren.
Currently we have universal2 and arm64 wheels for pandas 1.3.4 for Python 3.10, but we're planning on adding x86_64 wheels for 1.3.5.
Unfortunately, though, IIRC there's no way to install universal2 wheels if x86_64 or arm64 wheels are present, but dropping either x86_64 or arm64 wheels would result in basically a doubling of package size for users on those platforms.

Quick question: Should all three(x86_64, universal2, arm64) wheels be provided or just a subset of those three wheels?
I'd be happy to add the rest of the universal2/arm64 wheels after discussing this.

[ronaldoussoren]

Sorry about the slow response.

I'd prefer to only have universal2 wheels as long as the Python.org installer is the universal2 variant. A universal2 variant is larger than an architecture specific one, but the pandas' wheel is not that large given the large disk sizes in current machines.

An important advantage of universal2 wheels is that this makes it easier to package applications using tools like py2app and PyInstaller for use on multiple machines.

Another advantage is that you can trivially download wheels and install them on other machines without having to know about the CPU in them. Although that's less interesting these days.

[rgommers]

I'd prefer to only have universal2 wheels as long as the Python.org installer is the universal2 variant. A universal2 variant is larger than an architecture specific one, but the pandas' wheel is not that large given the large disk sizes in current machines.

This seems like a bad idea to me. Wheel sizes are not insignificant. I did a quick check for NumPy, which already has both arm64 and universal2 wheels up:

• numpy-1.21.4-cp310-cp310-macosx_11_0_arm64.whl is 11.8 MiB, and 40 MiB unzipped
• numpy-1.21.4-cp310-cp310-macosx_10_9_universal2.whl is 25.9 MiB, and 116 MiB unzipped

Pandas wheel sizes are similar to NumPy's. If you install even a very basic scientific stack, you have (e.g.) NumPy, SciPy, Pandas, scikit-learn, scikit-image. That'll be like ~0.5 GB on disk, perhaps more (SciPy and scikit-image are significantly larger). So taking the approach of doubling wheel and on-disk sizes for the only reason to make it a little easier to use py2app & co does not make any sense to me.

In addition, universal2 takes much more CI time and is complicated to support (multibuild has to jump through a bunch of hoops to glue things together), and is in general just not a great idea from a technical perspective. Hence arm64 wheels should be required, and universal2 is a nice to have.

Unfortunately, though, IIRC there's no way to install universal2 wheels if x86_64 or arm64 wheels are present,

pip install pandas works on arm64 as well as on x86_64 (as do other installers), so is there any issue with this? I assume testing the wheels is done before uploading to PyPI, so that you cannot install a universal2 wheel directly from PyPI should not matter much.

[lithomas1]

At this point, I think I'm just inclined to provide follow numpy's lead and provide all three wheels(x86_64, arm64, and universal2). Although the universal2 wheels are bigger in size, I do understand that they have a valid use case for a niche group of users and will commit to trying to support them.
Since pip defaults to installing the platform specific wheel when available(as in this case), I guess users interested in universal2 will have to manually download with something like pip download --platform