PERF: Avoid materializing values in Categorical.set_categories #17515
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
TomAugspurger
added
Categorical
TomAugspurger
force-pushed
the
set_categories-perf
branch
from
d848c95 to
6d72836
Compare
|
Here's the ASV: |
TomAugspurger
force-pushed
the
set_categories-perf
branch
2 times, most recently
from
17dd9f9 to
f74da7e
Compare
|
Hello @TomAugspurger! Thanks for updating the PR. Cheers ! There are no PEP8 issues in this Pull Request. 🍻 Comment last updated on September 14, 2017 at 10:55 Hours UTC |
TomAugspurger
force-pushed
the
set_categories-perf
branch
from
f74da7e to
9b311f4
Compare
Codecov Report
@@ Coverage Diff @@
## master #17515 +/- ##
==========================================
- Coverage 91.24% 91.2% -0.05%
==========================================
Files 163 163
Lines 49582 49586 +4
==========================================
- Hits 45242 45225 -17
- Misses 4340 4361 +21
Continue to review full report at Codecov.
|
jreback
reviewed
Sep 13, 2017
|
|
||
| Examples | ||
| -------- | ||
| >>> old_cat = pd.Index(['b', 'a', 'c']) |
There was a problem hiding this comment.
isn't this just a special case of union_categoricals?
There was a problem hiding this comment.
these both remap codes; seems like they've should share
There was a problem hiding this comment.
I see. There was a little bit I could extract from union_categorical. Simplified things a bit too by using take_1d. See my latest commit.
|
I don't think so. `union_categoricals` takes the union of the two
categories. This replaces the old categories with the new.
…On Wed, Sep 13, 2017 at 6:24 PM, Jeff Reback ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In pandas/core/categorical.py
<#17515 (comment)>:
> +def _recode_for_categories(codes, old_categories, new_categories):
+ """
+ Convert a set of codes for to a new set of categories
+
+ Parameters
+ ----------
+ codes : array
+ old_categories, new_categories : Index
+
+ Returns
+ -------
+ new_codes : array
+
+ Examples
+ --------
+ >>> old_cat = pd.Index(['b', 'a', 'c'])
isn't this just a special case of union_categoricals?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#17515 (review)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ABQHIr3MDXGGPEIv_2U2nqZ4qxzcvvvRks5siGQ9gaJpZM4PWjZF>
.
|
Mater:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)];
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
HEAD:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)]
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
Closes pandas-dev#17508
TomAugspurger
force-pushed
the
set_categories-perf
branch
from
9b311f4 to
d238e3e
Compare
|
thanks @TomAugspurger |
alanbato
pushed a commit
to alanbato/pandas
that referenced
this pull request
Nov 10, 2017
Mater:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)];
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
HEAD:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)]
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
Closes pandas-dev#17508
No-Stream
pushed a commit
to No-Stream/pandas
that referenced
this pull request
Nov 28, 2017
Mater:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)];
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
HEAD:
```python
In [1]: import pandas as pd; import numpy as np
In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
size=500000)]
s = pd.Series(arr).astype('category')
In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
Closes pandas-dev#17508
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Mater:
HEAD:
Closes #17508
I'll rebase #16015 on top of this. Running an ASV now.