BUG: astype(CategoricalDtype) has no effect

[arita37]

The following fails silently to convert the dtype:

In [14]: s = pd.Series(['a', 'b', 'c'])

In [15]: s
Out[15]: 
0    a
1    b
2    c
dtype: object

In [16]: s.astype(pd.types.dtypes.CategoricalDtype)
Out[16]: 
0    a
1    b
2    c

I would think this either to work or either to raise an error that the dtype is not undertood.

---

Code Sample

When using dictionnary of dtype, it does not convert the dataframe :
Types are not modified whereas df['col']= df['col].astype(type1) works....

dtype0= {'brand': np.dtype('int64'),
 'category': np.dtype('int64'),
 'chain': np.dtype('int64'),
 'company': np.dtype('int64'),
 'date': np.dtype('O'),
 'dept':  pandas.types.dtypes.CategoricalDtype,
 'id': np.dtype('int64')}
df= df.astype(dtype0)
df.dtypes

Problem description

When using dictionnary of dtype, it does not convert the dataframe :
Types are not modified whereas df['col']= df['col].astype(type1) works....

Expected Output

columns are converted into the desire types.

Output of pd.show_versions()

Pandas 0.19.2

# Paste the output here pd.show_versions() here

[jorisvandenbossche]

Can you show a reproducible example that shows the problem?

For me this works with a simple example:

In [1]: pd.__version__
Out[1]: '0.19.2'

In [2]: df = pd.DataFrame({'a':[1,2,3], 'b':['a','b', 'c']})

In [3]: df.dtypes
Out[3]: 
a     int64
b    object
dtype: object

In [4]:  df.astype({'a': 'float64', 'b': 'category'}).dtypes
Out[4]: 
a     float64
b    category
dtype: object

[arita37]

Your solution is working.
It seems pandas.types.dtypes.CategoricalDtype is not recognized when doing the casting,
so better to use 'category'

[jorisvandenbossche]

Hmm, that should actually work I think. Or otherwise an error. I reopened the issue and updated the top post with an example.

[jreback]

maybe though this is an internal type (and not directly exposed to the user)

[jorisvandenbossche]

Yeah, and it should not necessarily work for me, but if we don't accept it as a dtype, then it should raise an error IMO.

[jreback]

this is with our new default
errors='raise'?

[jorisvandenbossche]

It does not depend on that (and the default was already to raise, the keyword name changed but not the default value).

The reason is that the provided dtype is converted to a numpy dtype:

In [24]: np.dtype(list)
Out[24]: dtype('O')

In [25]: np.dtype(pd.types.dtypes.CategoricalDtype)
Out[25]: dtype('O')

For that reason also something like s.astype(list) does work but does not do anything (for an object series).

It can just a bit confusing to users I think, as an instantiated CategoricalDtype actually works:

In [26]: s.astype(pd.types.dtypes.CategoricalDtype())
Out[26]: 
0    a
1    b
2    c
dtype: category
Categories (3, object): [a, b, c]

In [27]: s.astype(pd.types.dtypes.CategoricalDtype)
Out[27]: 
0    a
1    b
2    c
dtype: object

which comes down to this difference:

In [28]: pd.types.common.is_categorical_dtype(pd.types.dtypes.CategoricalDtype)
Out[28]: False

In [29]: pd.types.common.is_categorical_dtype(pd.types.dtypes.CategoricalDtype())
Out[29]: True

[arita37]

Additionnally, There are numpy dtypes and Pandas types... We spent some time to find where is the real object of 'category' as this is not really mentionned in the documentation. Joris Van den Bossche wrote:

…

It does not depend on that (and the default /was/ already to raise, the keyword name changed but not the default value). The reason is that the provided dtype is converted to a numpy dtype: |In [24]: np.dtype(list) Out[24]: dtype('O') In [25]: np.dtype(pd.types.dtypes.CategoricalDtype) Out[25]: dtype('O') | For that reason also something like |s.astype(list)| does work but does not do anything (for an object series). It can just a bit confusing to users I think, as an instantiated CategoricalDtype actually works: |In [26]: s.astype(pd.types.dtypes.CategoricalDtype()) Out[26]: 0 a 1 b 2 c dtype: category Categories (3, object): [a, b, c] In [27]: s.astype(pd.types.dtypes.CategoricalDtype) Out[27]: 0 a 1 b 2 c dtype: object | which comes down to this difference: |In [28]: pd.types.common.is_categorical_dtype(pd.types.dtypes.CategoricalDtype) Out[28]: False In [29]: pd.types.common.is_categorical_dtype(pd.types.dtypes.CategoricalDtype()) Out[29]: True | — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#15078 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AR10p_PdX0ZjZjjg3e7uPFPiJTq1pDYgks5rP8GAgaJpZM4LdWAs>.

[TomAugspurger]

We spent some time to find where is the real object of 'category' as
this is not really mentionned in the documentation.

At the moment, the CategoricalDtype isn't part of the public API. There are plans to refactor it a bit before exposing it.