Skip to content

PERF: Avoid materializing values in Categorical.set_categories #17508

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
TomAugspurger opened this issue Sep 13, 2017 · 0 comments
Closed

PERF: Avoid materializing values in Categorical.set_categories #17508

TomAugspurger opened this issue Sep 13, 2017 · 0 comments
Labels
Categorical Categorical Data Type Performance Memory or execution speed performance
Milestone
0.21.0

Comments

@TomAugspurger
Copy link
Contributor

TomAugspurger commented Sep 13, 2017 •
edited
Loading

In Categorical.set_categories, we allocate an array of the values, which may be expensive:

pandas/pandas/core/categorical.py

Line 774 in 83436af

values = cat.__array__()

It should be possible to do this operation by just manipulating the codes.

In [6]: c = pd.Categorical(['a'] * 100000)

In [7]: c.set_categories(['a', 'b'])
Out[7]:
[a, a, a, a, a, ..., a, a, a, a, a]
Length: 100000
Categories (2, object): [a, b]

See 5ab0123 for how this might work, which will probably be squashed, but it's the implementation of Categorical._set_dtype in #16015

I may get to this as a followup to that PR.
@TomAugspurger TomAugspurger added the Performance Memory or execution speed performance label Sep 13, 2017
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
d848c95 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
@TomAugspurger TomAugspurger mentioned this issue Sep 13, 2017
Merged
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
6d72836 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
17dd9f9 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
f74da7e 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
9b311f4 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
0416154 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508

(cherry picked from commit 9b311f4)
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
8c740fd 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508

(cherry picked from commit 9b311f4)
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
3bb521c 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508

(cherry picked from commit 9b311f4)
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 13, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
90a10cd 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508

(cherry picked from commit 9b311f4)
@jreback jreback added this to the 0.21.0 milestone Sep 13, 2017
@jreback jreback added the Categorical Categorical Data Type label Sep 13, 2017
TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Sep 14, 2017
@TomAugspurger
PERF: Avoid values in Categorical.set_categories
d238e3e 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
jreback pushed a commit that referenced this issue Sep 14, 2017
@TomAugspurger @jreback
PERF: Avoid values in Categorical.set_categories (#17515)
0097cb7 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes #17508
alanbato pushed a commit to alanbato/pandas that referenced this issue Nov 10, 2017
@TomAugspurger @alanbato
PERF: Avoid values in Categorical.set_categories (pandas-dev#17515)
e2a417e 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
No-Stream pushed a commit to No-Stream/pandas that referenced this issue Nov 28, 2017
@TomAugspurger @No-Stream
PERF: Avoid values in Categorical.set_categories (pandas-dev#17515)
8000eee 
Mater:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)];
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
68.5 ms ± 846 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

HEAD:

```python
In [1]: import pandas as pd; import numpy as np

In [2]: arr = ['s%04d' % i for i in np.random.randint(0, 500000 // 10,
                                                      size=500000)]
s = pd.Series(arr).astype('category')

In [3]: %timeit s.cat.set_categories(s.cat.categories)
7.43 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

Closes pandas-dev#17508
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Categorical Categorical Data Type Performance Memory or execution speed performance
Projects
None yet
Milestone
0.21.0
Development

No branches or pull requests
2 participants
@jreback @TomAugspurger