object of type 'CategoricalDtype' has no len()

[teto]

Code Sample, a copy-pastable example if possible

Using 0.23.3

My goal: I have an enum with 2 values Client/Server and would like to make use of the categorical dtype for performance reasons.

import pandas as pd
from enum import IntEnum, auto

# Your code here
class ConnectionRoles(IntEnum):
    Client = auto()
    Server = auto()

    def __str__(self):
        # Note that defining __str__ is required to get ArgumentParser's help output to include the human readable (values) of Color
        return self.name

dtype_role = pd.api.types.CategoricalDtype(categories=[ x.name for x in ConnectionRoles], ordered=True)

data  = pd.DataFrame()

# I want to add a new column "tcpdest" full of nans but with a specific dtype to the dataframe "data"
new = pd.DataFrame(dtype= { "tcpdest": dtype_role })
data = pd.concat([ data, new ],)

triggers

  File "/home/teto/mptcpanalyzer/mptcpanalyzer/data.py", line 480, in load_into_pandas
    "tcpdest": dtype_role
  File "/nix/store/gcshh2ivxrbjgbx6xidx25cp2shlbjd6-python3.6-pandas-0.23.3/lib/python3.6/site-packages/pandas/core/frame.py", line 339, in __init__
    dtype = self._validate_dtype(dtype)
  File "/nix/store/gcshh2ivxrbjgbx6xidx25cp2shlbjd6-python3.6-pandas-0.23.3/lib/python3.6/site-packages/pandas/core/generic.py", line 168, in _validate_dtype
    dtype = pandas_dtype(dtype)
  File "/nix/store/gcshh2ivxrbjgbx6xidx25cp2shlbjd6-python3.6-pandas-0.23.3/lib/python3.6/site-packages/pandas/core/dtypes/common.py", line 2013, in pandas_dtype
    npdtype = np.dtype(dtype)
  File "/nix/store/fqkicsyl8a0sahg5k6axcd1kcklmpjzz-python3.6-numpy-1.14.5/lib/python3.6/site-packages/numpy/core/_internal.py", line 62, in _usefields
    names, formats, offsets, titles = _makenames_list(adict, align)
  File "/nix/store/fqkicsyl8a0sahg5k6axcd1kcklmpjzz-python3.6-numpy-1.14.5/lib/python3.6/site-packages/numpy/core/_internal.py", line 30, in _makenames_list
    n = len(obj)
TypeError: object of type 'CategoricalDtype' has no len()

Problem description

[this should explain why the current behaviour is a problem and why the expected output is a better solution.]
I want to specify the dtype otherwise pandas convert my enums into their integer values. I prefer to keep enums since when printing the dataframe it's easier to read "server" than "2".

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]

INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.14.24
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8
LOCALE: fr_FR.UTF-8

pandas: 0.23.3
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: 0.28.3
numpy: 1.14.5
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: 1.2.1
tables: 3.4.4
numexpr: 2.6.6
feather: None
matplotlib: 2.2.2
openpyxl: 2.5.4
xlrd: 0.9.4
xlwt: 1.3.0
xlsxwriter: None
lxml: 4.2.3
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.10
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

Can you make a reproducible example? http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports

[teto]

@TomAugspurger thanks for the interest, snippet can now be run as is and will trigger the same error.

[TomAugspurger]

Can you paste the full traceback running your snippet? The line

new = pd.DataFrame(dtype= { "tcpdest": dtype_role })

is invalid, since dtype should be a scalar dtype, not a dict

dtype : dtype, default None
    Data type to force. Only a single dtype is allowed. If None, infer

[teto]

arf indeed, I am used to using read_csv which accept a dtypes parameter. I must confess I don't understand the documentation for dtype. A dataframe can have multiple columns with different dtype, why can't I pass several dtypes as in read_csv ? apparently my solution is to use astype instead.

[TomAugspurger]

dataframe can have multiple columns with different dtype, why can't I pass several dtypes as in read_csv ?

I think there's an enhancement request for that.