BUG: Series.map using categorical Series raises AttributeError

[sinhrks]

Closes #10324. Closes #10460.

Based on #9848, using .get_values should be avoided?

[jreback]

get_values is fine
in fact this is the point of it

[sinhrks]

Thanks. Another point I forgot to refer is below case results in object dtype. Should it be category?

a = Series([1, 2, 3, 4])
b = Series(["even", "odd", "even", "odd"], dtype="category")
a.map(b)
# 0     odd
# 1    even
# 2     odd
# 3     NaN
# dtype: object

[jreback]

so the way to do this is actually
arg.values.take_nd(indexer) which will return a Categorical.

so maybe make this mod in core.common.take_nd (which is aliases to take_1d). if its a categorical_dtype, then short-circuit as I show above. (it ultimately calls take_1d as well but with the codes)

[sinhrks]

Applied a fix to common.take_nd. I will add more tests later.

[jreback]

more like return arr.values.take_1d(...)

[sinhrks]

@jreback Categorical doesn't have values, but get_values().

import pandas as pd
c = pd.Categorical(['A', 'B', 'A', 'A'])
c
# [A, B, A, A]
# Categories (2, object): [A, B]
c.values
# AttributeError: 'Categorical' object has no attribute 'values'

[jreback]

no, you send this from a higher level (e.g. arr is actually a Series/CategoricalIndex object when you start).

You must use the take_1d in Categorical, and NOT create one here, this is moving too specific knowledge to this function

something like:

if is_categorical_dtype(arr):
    arr = getattr(arr,'value',arr)
    return arr.take_1d(....)

...

[sinhrks]

@jreback I think I could understand. Changed to use take_nd, as Categorical doesn't have take_1d.

[sinhrks]

BTW, I've noticed Categorical has its own take_nd. I think we can use this, but it raises IndexError for non-existing key.

import pandas as pd
c = pd.Categorical(['A', 'B', 'A', 'A'])
c.take_nd([2, 1])
# [A, B]
# Categories (2, object): [A, B]

c.take_nd([2, 1, 4])
# IndexError: Out of bounds on buffer access (axis 0)

So another point of the original issue is the behavior when the mapped category doesn't contain all the mappings. Should raise or padding with np.nan (create a new category implicitly)?

# this should work because all values are mapped to existing categories
pd.Series([0, 1, 2, 3]).map(pd.Series(["even", "odd", "even", "odd"], dtype="category"))
# 0    even
# 1     odd
# 2    even
# 3     odd
# dtype: category 

pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"], dtype="category"))
# raise or padding with np.nan?

[jreback]

These should be the same as a post-convert to category
The nans are NOT part of the categorical, rather missing in the Series themselves.

In [5]: a = pd.Series([0, 1, 2, 3]).map(pd.Series(["even", "odd", "even", "odd"]))                   

In [6]: b = pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"]))

In [7]: a.astype('category')
Out[7]: 
0    even
1     odd
2    even
3     odd
dtype: category
Categories (2, object): [even, odd]

In [8]: b.astype('category')
Out[8]: 
0     odd
1    even
2     odd
3     NaN
dtype: category
Categories (2, object): [even, odd]

[sinhrks]

I see. Then, current impl works as expected. I'll consider if it can be better.

b = pd.Series([1, 2, 3, 4]).map(pd.Series(["even", "odd", "even", "odd"], dtype='category'))
b
# 0     odd
# 1    even
# 2     odd
# 3     NaN
# dtype: category
# Categories (2, object): [even, odd]

[jreback]

exactly!

[jreback]

looks good. go ahead and merge when green.

[sinhrks]

@jreback Thanks for review. Merging.