API: ArrowExtensionArray._cmp_method to return pyarrow.bool_ type

[lukemanley]

• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/v2.1.0.rst file if fixing a bug or adding a new feature.

Per #51462 (comment), updating ArrowExtensionArray._cmp_method to return boolean[pyarrow] dtype instead of boolean.

[phofl]

Looks like this is failing asvs?

[lukemanley]

I've reverted this. The asv job was failing before I tried this change. This was just an attempt to find the issue.

[lukemanley]

I think(?) the first four here should have underscores. These were raising in an ASV.

[lukemanley]

for example, this raises on main:

import pandas as pd

s1 = pd.Series([True], dtype="boolean[pyarrow]")
s2 = pd.Series([False], dtype="boolean[pyarrow]")

# raises: KeyError: 'or_'
s1 | s2  

[phofl]

Good find! Could you move into a separate pr and add tests?

[lukemanley]

sure, will do

[lukemanley]

opened #51688 for this

[phofl]

the min versions failure looks legit?

[lukemanley]

the min versions failure looks legit?

yep. just pushed a fix. I think the failure is related to the deepcopy you added in #50984

It looks like that test combined with the deepcopy was hitting this bug in older versions of numpy: numpy/numpy#21883

I think this will work now. What is the reason to use deepcopy instead of ndarray.copy there? It can be very slow for an object dtype array:

import numpy as np
from copy import deepcopy

arr = np.arange(1_000_000, dtype=object)

%timeit arr.copy()
%timeit deepcopy(arr)

# 9.83 ms ± 757 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
# 298 ms ± 4.82 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)    

[phofl]

@mroeschke argued back then that this would cover eas without a copy method, but if this is slow, then I'd say we can switch to copy

[mroeschke]

Fine to switch to copy()

[mroeschke]

Thanks @lukemanley

[phofl]

@mroeschke thoughts on back porting? This would fix #51980

[mroeschke]

Sure, +1 to backport

[phofl]

@meeseeksdev backport 2.0.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.0.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 b3d18af7d7ae2c454b9dcef0f2c0c2ae89bea047

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #51643: API: ArrowExtensionArray._cmp_method to return pyarrow.bool_ type'

4. Push to a named branch:

git push YOURFORK 2.0.x:auto-backport-of-pr-51643-on-2.0.x

5. Create a PR against branch 2.0.x, I would have named this PR:

"Backport PR #51643 on branch 2.0.x (API: ArrowExtensionArray.cmp_method to return pyarrow.bool type)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.