df.append should retain columns type if same type

[topper-123]

Currently df.append loses columns index type, if the columns is a CategoricalIndex:

>>> idx = pd.CategoricalIndex('a b'.split())
>>> df = pd.DataFrame([[1, 2]], columns=idx)
>>> ser = pd.Series([3, 4], index=idx, name=1)
>>> df.append(ser).columns
Index(['a', 'b'], dtype='object')

df.append(ser).columns should return a CategoricalIndex equal to idx.

pandas 0.21 has the new CategoricalDtype, so it's now easy to compare CategoricalIndex instances for strict type equality. Hence this issue should be much easier to solve than previously.

Solution proposal

In frame.py::DataFrame.append there is this line:

combined_columns = self.columns.tolist() + self.columns.union(
                    other.index).difference(self.columns).tolist()

This line converts CategoricalIndex columns to normal indexes. So by making some checks for types and dtypes it should be easy return the correct index. So if the above would be something like this instead:

same_types = type(self.columns) == type(other.index)
same_dtypes = self.columns.dtype == other.index.dtype
if same_types and same_dtypes:
    combined_columns = self.columns.union(other.index)
else:
    combined_columns = self.columns.tolist() + self.columns.union(
        other.index).difference(self.columns).tolist()

and I think this issue can be solved (haven't checked yet all details, maybe some adjustments have to be made). I'd appreciate comments if this approach is ok.

[jreback]

yeah this is kind of messy. this should all use Index.append then none of this is an issue. we shouldn't be using .tolist() at all.

[topper-123]

Hi, I've started to look into this.

ATM it seems like .union actually is very robust, and I'm leaning towards that simply combined_columns = self.columns.union(other.index) is possible, but I wonder why you pointed to Index.append. Did you mean Index.Union?

[jreback]

no i meant append; you need to append the union of differences (i think this is the symmetric_didferenev)

[topper-123]

symmetric_difference doesn't work:

>>> d = pd.api.types.CategoricalDtype('A B C'.split())
>>> c1 = pd.CategoricalIndex('A B'.split(), dtype=d)
>>> c2 = pd.CategoricalIndex('B C'.split(), dtype=d)
>>> c1.symmetric_difference(c2)
Index(['A', 'C'], dtype='object')  # notice index type and also values are not good to be appended

Just difference is good:

>>> c2.append(c1.difference(c2))
CategoricalIndex(['B', 'C', 'A'], categories=['A', 'B', 'C'], ordered=False, dtype='category')

Which gives the same result (in this case, maybe generally) as union:

>>> c2.union(c1)
CategoricalIndex(['B', 'C', 'A'], categories=['A', 'B', 'C'], ordered=False, dtype='category')

[avnishbm]

This issue still exists with dataframe.append, it happens to change the index type to pandas.core.indexes.base.Index, even though both the dataframes being merged had index type as pandas.core.indexes.datetimes.DatetimeIndex! Can this be looked into please?

Have also tried using inplace=True for the append call but it was of no use!

[jreback]

@avnishbm this is a very old issue

if you think u have a bug then pls show a reproducible example in a new issue testing against the latest released version

[avnishbm]

@jreback have raised a new issue providing the details: #40435