Major Bug: Concatenating Categorical Columns Yields Jumbled Result

[sam-cohan]

Code Sample

import pandas as pd
cat1 = pd.Categorical(
    ["a", "b",], 
    categories=["a", "b"],
    ordered=False)
cat2 = pd.Categorical(
    ["a", "b"], 
    categories=["b", "a"],
    ordered=False)

df1 = pd.DataFrame({"A": ["a", "b"], "B": cat1})
df2 = pd.DataFrame({"A": ["a", "b"], "B": cat2})
expected = pd.DataFrame({"A": ["a", "b", "a", "b"], 
                         "B": ["a", "b", "a", "b"]})
res = pd.concat([df1, df2], ignore_index=True)
print(res)
assert np.all(res == expected), "expected:\n{}\n res:\n{}".format(expected, res)

	A	B
0	a	a
1	b	b
2	a	b
3	b	a

Problem description

In some previous versions, merging two categorical columns where the category and label did not agree would give the expected result of converting to string and using the values rather than the categories. In the new version, the merged columns keep the categories instead of the values. This is a major problem because it causes unexpected results in workflows which require reading from multiple parts, converting some of their columns to categories and then merging them.

Expected Output

	A	B
0	a	a
1	b	b
2	a	a
3	b	b

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-1049-aws
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.22.0
pytest: 3.2.5
pip: 9.0.2
setuptools: 38.5.1
Cython: 0.25.1
numpy: 1.13.3
scipy: 0.18.1
pyarrow: None
xarray: None
IPython: 5.1.0
sphinx: None
patsy: None
dateutil: 2.7.0
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: 2.6.1
feather: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.1.2
pymysql: None
psycopg2: 2.6.2 (dt dec pq3 ext lo64)
jinja2: 2.8
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.5.0

[TomAugspurger]

Sorry about that, already fixed in master, #16339 I think. We're hoping to have a release out in a couple weeks.

For now, you can work around by ensuring that your unordered categories have the same categories order. (if they have different categories, that's fine too. This is only an issue when the categories are unordered, the same set, but in a different order).

cat2 = cat2.set_categories(cat1.categories)