BUG: replace fails with IndexError when regex parameter is passed a dictionary with multiple items

[dmonder]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np

data = df = pd.DataFrame({ 'a_str' : ['A1','A2','A3'],
                           'b_int' : ['1,000','200','3'],
                           'c_str' : ['C1','C2','C3'],
                           'd_date' : ['2021-01-01','','2021-03-03']})
non_string_columns = ['b_int','d_date']

df[non_string_columns] = df[non_string_columns].replace(regex={'':np.nan,',':''})

Traceback (most recent call last):
  File "c:\Users\donder\.vscode\extensions\ms-python.python-2021.1.502429796\pythonFiles\lib\python\debugpy\_vendored\pydevd\_pydevd_bundle\pydevd_vars.py", line 416, in evaluate_expression
    compiled = compile(_expression_to_evaluate(expression), '<string>', 'eval')
  File "<string>", line 1
    df[non_string_columns] = df[non_string_columns].replace(regex={'':np.nan,',':''})
                           ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "E:\Python\Python38\lib\site-packages\pandas\core\frame.py", line 4521, in replace
    return super().replace(
  File "E:\Python\Python38\lib\site-packages\pandas\core\generic.py", line 6842, in replace
    return self.replace(
  File "E:\Python\Python38\lib\site-packages\pandas\core\frame.py", line 4521, in replace
    return super().replace(
  File "E:\Python\Python38\lib\site-packages\pandas\core\generic.py", line 6891, in replace
    new_data = self._mgr.replace_list(
  File "E:\Python\Python38\lib\site-packages\pandas\core\internals\managers.py", line 664, in replace_list
    bm = self.apply(
  File "E:\Python\Python38\lib\site-packages\pandas\core\internals\managers.py", line 427, in apply
    applied = getattr(b, f)(**kwargs)
  File "E:\Python\Python38\lib\site-packages\pandas\core\internals\blocks.py", line 901, in _replace_list
    result = blk._replace_coerce(
  File "E:\Python\Python38\lib\site-packages\pandas\core\internals\blocks.py", line 1643, in _replace_coerce
    return self._replace_regex(
  File "E:\Python\Python38\lib\site-packages\pandas\core\internals\blocks.py", line 844, in _replace_regex
    replace_regex(new_values, rx, value, mask)
  File "E:\Python\Python38\lib\site-packages\pandas\core\array_algos\replace.py", line 133, in replace_regex
    values[mask] = f(values[mask])
IndexError: boolean index did not match indexed array along dimension 0; dimension is 1 but corresponding boolean dimension is 2

Problem description

In the dataframe columns specified by the non_string_column variable, the above code should be replacing all the blanks with np.nan and removing all the commas.

If I change the replace to look like the following, the dataframe is correct.

df[non_string_columns] = df[non_string_columns].replace(regex={'':np.nan})
df[non_string_columns] = df[non_string_columns].replace(regex={',':''})

Expected Output

I expected the dataframe to look like the following:

  a_str b_int c_str      d_date
0    A1  1000    C1  2021-01-01
1    A2   200    C2         NaN
2    A3     3    C3  2021-03-03

I get this error whether I am using Python 3.8.3 or 3.9.1. Also, the numpy version under Python 3.9.1 is 1.19.5.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 9d598a5
python : 3.8.3.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.17763
machine : AMD64
processor : Intel64 Family 6 Model 58 Stepping 0, GenuineIntel
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : English_United States.1252

pandas : 1.2.1
numpy : 1.18.2
pytz : 2020.1
dateutil : 2.8.1
pip : 20.3.3
setuptools : 41.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.18.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.15
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None
None

[kidrahahjo]

@phofl I'd like to contribute here. This will be my first time with pandas-dev, can you help me out?

[phofl]

Hey,

this is a tricky bug. The actual replace is done on object Blocks (this is an internal structure). At the beginning we have an object block representing both columns. After replacing the empty string with nan, the object block is split into two Object blocks, this causes the indexing error. I thinkt the split should not happen, but I am not an expert here either.

[kidrahahjo]

Interesting, I'll have a look at the source code and see if I can solve this.

[kidrahahjo]

@phofl I saw that there is a split of ObjectBlock instance happening. The logic clearly states to split if ndim == 2. I am not familiar with the internal working of pandas so I don't think I should make any comment on merging the split or not splitting in the ObjectBlock in the first place.

[phofl]

Haven't looked deeply enough, but this seemed odd to me. When you create a DataFrame with nans in object columns this is stored as one object block, for example

df = DataFrame({"a": ["x", "y"], "b": [np.nan, "z"]})

has one object block. You can see this with df._mgr. In some cases the nan may change the dtype of the block in replace, maybe this is the reason?

df = DataFrame({"a": ["x", "y"], "b": ["test", 1]})
df.replace({"test": np.nan}, inplace=True)

Ths casts the second column to float, hence there is a split necessary.

[simonjayhawkins]

working in 1.1.5, labelling as regression

>>> import pandas as pd
>>> import numpy as np
>>>
>>> data = df = pd.DataFrame({ 'a_str' : ['A1','A2','A3'],
...                            'b_int' : ['1,000','200','3'],
...                            'c_str' : ['C1','C2','C3'],
...                            'd_date' : ['2021-01-01','','2021-03-03']})
>>> non_string_columns = ['b_int','d_date']
>>>
>>> df[non_string_columns] = df[non_string_columns].replace(regex={'':np.nan,',':''})
>>> df
  a_str b_int c_str      d_date
0    A1  1000    C1  2021-01-01
1    A2   200    C2         NaN
2    A3     3    C3  2021-03-03
>>> pd.__version__
'1.1.5'
>>>

[simonjayhawkins]

working in 1.1.5, labelling as regression

starting failing with

first bad commit: [4f2251c] REF: implement Block._replace_list (#36020) cc @jbrockmendel

but at this point the traceback was

2021-01-26T14:47:49.7247948Z 1.2.0.dev0+210.g4f2251c6e6
2021-01-26T14:47:49.7281618Z Traceback (most recent call last):
2021-01-26T14:47:49.7282316Z   File "../39338.py", line 16, in <module>
2021-01-26T14:47:49.7283045Z     df[non_string_columns] = df[non_string_columns].replace(regex={"": np.nan, ",": ""})
2021-01-26T14:47:49.7283903Z   File "/home/runner/work/pandas/pandas/pandas/core/frame.py", line 4418, in replace
2021-01-26T14:47:49.7284592Z     return super().replace(
2021-01-26T14:47:49.7285312Z   File "/home/runner/work/pandas/pandas/pandas/core/generic.py", line 6546, in replace
2021-01-26T14:47:49.7286013Z     return self.replace(
2021-01-26T14:47:49.7286717Z   File "/home/runner/work/pandas/pandas/pandas/core/frame.py", line 4418, in replace
2021-01-26T14:47:49.7287398Z     return super().replace(
2021-01-26T14:47:49.7288107Z   File "/home/runner/work/pandas/pandas/pandas/core/generic.py", line 6589, in replace
2021-01-26T14:47:49.7288838Z     new_data = self._mgr.replace_list(
2021-01-26T14:47:49.7289640Z   File "/home/runner/work/pandas/pandas/pandas/core/internals/managers.py", line 633, in replace_list
2021-01-26T14:47:49.7290374Z     bm = self.apply(
2021-01-26T14:47:49.7291103Z   File "/home/runner/work/pandas/pandas/pandas/core/internals/managers.py", line 393, in apply
2021-01-26T14:47:49.7291851Z     applied = getattr(b, f)(**kwargs)
2021-01-26T14:47:49.7292622Z   File "/home/runner/work/pandas/pandas/pandas/core/internals/blocks.py", line 808, in _replace_list
2021-01-26T14:47:49.7293388Z     m = masks[i][blk.mgr_locs.indexer]
2021-01-26T14:47:49.7294685Z AttributeError: 'list' object has no attribute 'mgr_locs'

[simonjayhawkins]

moving to 1.2.3