Skip to content

Alternative implementation for .where on EA-backed Indexes #24144

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
TomAugspurger opened this issue Dec 7, 2018 · 4 comments
Closed

Alternative implementation for .where on EA-backed Indexes #24144

TomAugspurger opened this issue Dec 7, 2018 · 4 comments
Labels
ExtensionArray Extending pandas with custom dtypes or arrays. Performance Memory or execution speed performance

Comments

@TomAugspurger
Copy link
Contributor

Followup to #24114.

Right now Categorical.where and IntervalIndex.where involve converting to objects.

We could use an alternative implementation that copies the underlying array and sets the values.
#24114 (comment) should give a rough idea of the kinds of speedups you'd get.

@TomAugspurger TomAugspurger added Performance Memory or execution speed performance Effort Medium ExtensionArray Extending pandas with custom dtypes or arrays. labels Dec 7, 2018
@TomAugspurger TomAugspurger added this to the Contributions Welcome milestone Dec 7, 2018
@charlesdong1991
Copy link
Member

charlesdong1991 commented Dec 11, 2018

I am sorry to bother, could I ask you a question, each time when i tried to do the pytest, even with no change at all after git rebase, I got ImportError while loading conftest '/Users/kaiqi/pandas/pandas/tests/indexes/conftest.py'. pandas/tests/indexes/conftest.py:21: in <module> tm.makeCategoricalIndex(100), pandas/util/testing.py:1902: in makeCategoricalIndex return CategoricalIndex(np.random.choice(x, k), name=name, **kwargs) pandas/core/indexes/category.py:129: in __new__ dtype) pandas/core/indexes/category.py:192: in _create_categorical dtype=dtype) pandas/core/arrays/categorical.py:413: in __init__ raise NotImplementedError("> 1 ndim Categorical are not " E NotImplementedError: > 1 ndim Categorical are not supported at this time error, but i tested on pandas-dev/ci, there is no such problem. Do you have any idea where this problem might come from and how can I fix it locally? thank you so much!!

@TomAugspurger
Copy link
Contributor Author

Did you rebuild the C extensions? python setup.py build_ext -i

@charlesdong1991
Copy link
Member

ahh, it works... @TomAugspurger thanks so much!! i will keep this in mind!!!

@jbrockmendel
Copy link
Member

Index.where now dispatches to Index.putmask, which has a non-object-casting implementation on ExtensionIndex. closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ExtensionArray Extending pandas with custom dtypes or arrays. Performance Memory or execution speed performance
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants