BUG: pd.replace changes category dtype to object when using a dict to replace values

[mathurk1]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

>>> import pandas as pd
>>> pd.__version__
'1.0.5'
>>> input_dict = {"col1": ["a"], "col2": ["obj1"], "col3": ["cat1"]}
>>> input_df = pd.DataFrame(data=input_dict).astype({"col1": "category", "col2": "category", "col3": "category"})
>>> input_df
  col1  col2  col3
0    a  obj1  cat1
>>> input_df.dtypes
col1    category
col2    category
col3    category
dtype: object
>>> input_df = input_df.replace({"a": "z", "obj1": "obj9", "cat1": "catX"})
>>> input_df
  col1  col2  col3
0    z  obj9  catX
>>> input_df.dtypes
col1    object
col2    object
col3    object
dtype: object

Problem description

Currently, the category dtypes are lost when we use pandas replace. This loss of category dtype is observed only when a dictionary is used to replace multiple values.

I was able to recreate the issue for pd.__version__ == '1.1.0.dev0+2073.g280efbfcc' as well. Also, please see related issue: #23305

Expected Output

I would expect the categorical columns to keep their category dtype after replace is called on the dataframe. Thanks!

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Darwin
OS-release : 19.5.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.19.0
pytz : 2020.1
dateutil : 2.8.1
pip : 19.0.3
setuptools : 40.8.0
Cython : None
pytest : 5.4.3
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.3
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[tsibley]

This is true for other input dtypes as well, such as the new string dtype, and is demonstrable with Series.replace as well as DataFrame.replace.

>>> import pandas as pd
>>> pd.__version__
'1.1.0'

>>> s = pd.Series(["a", "", "b"], dtype="string")
>>> s
0    a
1
2    b
dtype: string

>>> s.replace("", pd.NA)
0       a
1    <NA>
2       b
dtype: string

>>> s.replace({"": pd.NA})
0       a
1    <NA>
2       b
dtype: object

I believe the issue is with the underlying Block.replace method. There are a couple places which explicitly convert to an object dtype. Maybe those are triggering too eagerly? For example:

pandas/pandas/core/internals/blocks.py

Lines 754 to 762 in 3b1d4f1

	if not self.is_extension:
	# TODO: https://github.com/pandas-dev/pandas/issues/32586
	# Need an ExtensionArray._can_hold_element to indicate whether
	# a scalar value can be placed in the array.
	assert not self._can_hold_element(value), value
	# try again with a compatible block
	block = self.astype(object)
	return block.replace(

Note the comment about extension dtypes, which seems particularly applicable.

[tsibley]

I also meant to point out that using two-arg replace (as shown in my REPL example) doesn't exhibit this behaviour (at least in my example). This is particularly odd, as I'd (naively, perhaps) expect two-arg replace and one-dict-arg replace to both map to the same codepath internally!

[tsibley]

The Period dtype also seems to suffer this issue: #34871.

[samc1213]

See #36867 for an idea of how to fix this. Basically I think that CategoricalBlock needs to properly implement _can_hold_element in order to fix this

[jbrockmendel]

closed by #44940