Core dump when converting to category with duplicate column names

[amueller]

Code Sample, a copy-pastable example if possible

import pandas as pd
import numpy as np
X = np.random.normal(size=(100, 2))
X_df = pd.DataFrame(X, columns=[0, 0])
X_df[0] = X_df[0].astype('category')

Problem description

Stack overflow in the Python interpreter

Expected Output

No core dump.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.7.1.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-46-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: 3.6.2
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.29.2
numpy: 1.16.2
scipy: 1.2.1
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: 1.8.2
patsy: 0.5.0
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.7.1
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: 2.7.7 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[amueller]

Oh, looks like this is fixed in 0.24.2?

[jschendel]

This is a dupe of #24704 and has been partially fixed in 0.24.x.

The crash was fixed by #24717, but setting with duplicate column names is buggy for extension dtypes and results in object dtype instead. Using a simplified example on master:

In [1]: import pandas as pd; pd.__version__
Out[1]: '0.25.0.dev0+299.ge31b4f46df'

In [2]: df = pd.DataFrame([[1, 2], [1, 1], [3, 2]], columns=['a', 'a'])

In [3]: df['a'] = df['a'].astype('category')

In [4]: df.dtypes
Out[4]:
a    object
a    object
dtype: object

If your DataFrame just consists of the duplicate columns you can get around this by reassigning the whole DataFrame:

In [5]: df = df.astype('category')

In [6]: df.dtypes
Out[6]:
a    category
a    category
dtype: object

If you have other columns and just want to convert the dupes, a potential work around is to concat the converted dupes with the non-dupe columns:

In [7]: df = df.astype('int64').assign(b=3.14)

In [8]: df.dtypes
Out[8]:
a      int64
a      int64
b    float64
dtype: object

In [9]: df = pd.concat([df['a'].astype('category'), df.drop('a', axis=1)], axis=1)

In [10]: df.dtypes
Out[10]:
a    category
a    category
b     float64
dtype: object

There are probably other workarounds but the above is the first thing that comes to mind.

[amueller]

Thanks for the detailed explanation!