BUG: adding a value not in the Categories does not raise a ValueError on a Series when adding the value to a new index

[arnaudlegout]

• [x ] I have checked that this issue has not already been reported.

• [ x] I have confirmed this bug exists on the latest version of pandas.

• [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

s = pd.Series(['a' , 'b'], dtype='category')
>>>
0    a
1    b
dtype: category
Categories (2, object): [a, b]

# this is expected, cannot assign a value not in the category
s.loc[1] = 'c'
>>>
ValueError: Cannot setitem on a Categorical with a new category, set the categories first

# this is NOT expected. not only it assigns a value not the in the category, but it also implicitly
# change the dtype
s.loc[2] = 'c'
>>>
0    a
1    b
2    c
dtype: object

Problem description

Assigning a value not in the category to an index already in the Series show a different behavior than when the index is not already in the Series.

This looks like an inconsistent behavior that is hard to explain.

The expected behavior is that assigning a value not in a category will always raise a ValueError Exception whether the assignement is in an existing index or in a new index.

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.7.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 1.0.3
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.1.3.post20200330
Cython : 0.29.15
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.13.0
pandas_datareader: 0.8.1
bs4 : None
bottleneck : 1.3.2
fastparquet : None
gcsfs : None
lxml.etree : 4.5.0
matplotlib : 3.1.3
numexpr : 2.7.1
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : 1.3.15
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : 0.48.0

[MarcoGorelli]

Thanks @arnaudlegout for the report

Agree that this seems like inconsistent behaviour - are you interested in submitting a pull request?

[arnaudlegout]

I never contributed to core pandas, and I am not sure I know where to start looking at. My guess is that the buggy logic is in the setattr method of the CategoricalDtype.

Also I don't have any experience in cython.

If I find some time, I will update this bug report with what I find. But, don't count too much on me.

[chrispe]

I'd be willing to give this a try.

[MarcoGorelli]

Sure, PRs are welcome

[arnaudlegout]

@MarcoGorelli I never contributed to core pandas, and I don't know much the code structure. However, setting up the pandas-dev environment was quite easy. So I made some tests.

@chrispe92 Let me share the very fast investigation I did. I found that the __setitem__ method in categorical.py is correctly called when we try to change a value for an index already existing in the Series. This is where the ValueError exception comes from.

However, when we set a value for an index not already in the Series, __setitem__ from categorical.py is not called. I assume that adding an index to a Series goes on a different code path than changing a value for an existing index. This code path is may be managed in the ExtensionArray

[chrispe]

@MarcoGorelli I never contributed to core pandas, and I don't know much the code structure. However, setting up the pandas-dev environment was quite easy. So I made some tests.

@chrispe92 Let me share the very fast investigation I did. I found that the __setitem__ method in categorical.py is correctly called when we try to change a value for an index already existing in the Series. This is where the ValueError exception comes from.

However, when we set a value for an index not already in the Series, __setitem__ from categorical.py is not called. I assume that adding an index to a Series goes on a different code path than changing a value for an existing index. This code path is may be managed in the ExtensionArray

Hi @arnaudlegout, thanks for the insights. I also did start with that and ended up to the same conclusions as yours. To which I may add that the actual code path for a new index value is done by __setattr__ in the generic.NDFrame.

[arnaudlegout]

I tested the inconsistent behavior for both string arrays and boolean arrays (from the new extension types)

They both have the same issue, they validate the type of the value set in the Series if the index is already existing, but not if the index is new.

[simonjayhawkins]

@arnaudlegout Thanks for the report.

Just to be clear, the conversion to object dtype happens even if the value is in the categories when setting with enlargement, see #25383

>>> import pandas as pd
>>>
>>> pd.__version__
'1.1.0.dev0+1500.ge9b019b65'
>>>
>>> s = pd.Series(["a", "b"], dtype="category")
>>> print(s)
0    a
1    b
dtype: category
Categories (2, object): [a, b]
>>>
>>>
>>> s.loc[2] = "a"
>>> print(s)
0    a
1    b
2    a
dtype: object

Closing as duplicate.