BUG: CategoricalDtype is not refresh after index categories set

[Yikun]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
pdf = pd.DataFrame(
          {
              "a": pd.Categorical([1, 2, 3, 1, 2, 3]),
          },
          index=pd.Categorical([10, 20, 30, 20, 30, 10], categories=[30, 10, 20], ordered=True),
      )
pidx = pdf.index
pidx.categories = ["z", "y", "x"]
pidx.dtype

Issue Description

categories set failed

>>> pidx.dtype
CategoricalDtype(categories=[30, 10, 20], ordered=True)

Expected Behavior

>>> pidx.dtype
CategoricalDtype(categories=['z', 'y', 'x'], ordered=True)

(also a behavior before 1.4.x)

Installed Versions

1.4.0+

[Yikun]

First bad commit: 126a19d

also cc @jbrockmendel

[samukweku]

@Yikun it seems to work with set_categories right?

[Yikun]

@samukweku Yes, I can make it work as workaround:

pidx = pidx.set_categories(pidx.categories)
pdf.index = pidx

But I didn't need to do by this way before 1.4.0

[jreback]

you are trying to modify an immutable object - it's possible it accidentally worked before but in no way is this correct

[Yikun]

@jreback Would we consider supporting index categories setters in pandas as a way to simplify set_categories in the future ?

[Yikun]

@jreback @jbrockmendel @samukweku Any thoughts? Thank!

[jbrockmendel]

we're actually looking at deprecating all the in-place category-setting behaviors xref #37643, so unless im misunderstanding what you're asking for, im -1 on adding this setter

[Yikun]

@jbrockmendel @jreback Thanks! And sorry for late reply.

Should we consider deprecating this setter or raise unsupported error? Because when the category is set (Yep, this is unexpected set to immutable object), the dtype is not refreshed, we are also no plan to fix it.

[1] https://github.com/pandas-dev/pandas/blob/main/pandas/core/arrays/categorical.py#L742-L743

[simonjayhawkins]

you are trying to modify an immutable object

yes. it appears that the index is updated (which should probably raise) but the dtype attribute is not updated.

pidx = pdf.index
print(pidx)
pidx.categories = ["z", "y", "x"]
print(pidx)
print(repr(pidx.dtype))

CategoricalIndex([10, 20, 30, 20, 30, 10], categories=[30, 10, 20], ordered=True, dtype='category')
CategoricalIndex(['y', 'x', 'z', 'x', 'z', 'y'], categories=['z', 'y', 'x'], ordered=True, dtype='category')
CategoricalDtype(categories=[30, 10, 20], ordered=True)

First bad commit: 126a19d

what code sample was used and what was the behavior change? I'm seeing the same behavior with the snippet above back to pandas 1.0.5

in pandas 0.25.3, it raised AttributeError: can't set attribute

[Yikun]

what code sample was used and what was the behavior change?

@simonjayhawkins

import pandas as pd
import pandas._testing as tm
import numpy as np

pdf = pd.DataFrame(
          {
              "a": pd.Categorical([1, 2, 3, 1, 2, 3]),
          },
          index=pd.Categorical([10, 20, 30, 20, 30, 10], categories=[30, 10, 20], ordered=True),
      )
pidx = pdf.index
pidx.categories = ["z", "y", "x"]
# Check `pidx.dtype.categories` is refreshed or not
tm.assert_index_equal(pidx.dtype.categories, pidx.categories)
tm.assert_numpy_array_equal(pidx.dtype.categories._data, pidx.categories._data)