BUG: DataFrame.replace fails to replace value when column contains pd.NA #47480

pschwientek · 2022-06-23T01:34:40Z

Pandas version checks

I have checked that this issue has not already been reported.
I have confirmed this bug exists on the latest version of pandas.
I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

df = pd.DataFrame({'A': [0, 1, 2]})
print(df)
#    A
# 0  0
# 1  1
# 2  2
df['A'].replace(to_replace=2, value=99, inplace=True)
print(df)
#     A
# 0   0
# 1   1
# 2  99
df.at[0, 'A'] = pd.NA
df['A'].replace(to_replace=1, value=100, inplace=True)
print(df)
#       A
# 0  <NA>
# 1     1  <-- should be 100
# 2    99

Issue Description

Pandas replace function does not seem to work on a column if the column contains at least one pd.NA value

Expected Behavior

replace function should work even if pd.NA values are in the column

Installed Versions

INSTALLED VERSIONS

commit : 66e3805
python : 3.10.0.final.0
python-bits : 64
OS : Linux
OS-release : 5.16.19-76051619-generic
Version : #202204081339~~1649696161~~20.04~~091f44b~~dev-Ubuntu SMP PREEMPT Tu
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.5
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.2.4
setuptools : 58.0.4
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 3.0.3
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.2
IPython : 7.29.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.5.1
numexpr : None
odfpy : None
openpyxl : 3.0.9
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

The text was updated successfully, but these errors were encountered:

phofl · 2022-06-24T13:52:38Z

Hi, thanks for your report.

Did you check on 1.4.2 and main? Because this works on both for me.

phofl · 2022-07-02T23:09:34Z

might need tests

johnmantios · 2022-07-04T10:05:10Z

Hi @phofl. I tried replicating the issue in 1.4.3 on my local machine and I got the following error:

df["A"].replace(to_replace=1, value=100, inplace=True)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/series.py", line 4960, in replace
    return super().replace(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/generic.py", line 6747, in replace
    new_data = self._mgr.replace(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 441, in replace
    return self.apply(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/internals/blocks.py", line 683, in replace
    mask = missing.mask_missing(values, to_replace)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pandas/core/missing.py", line 98, in mask_missing
    new_mask = new_mask.to_numpy(dtype=bool, na_value=False)
AttributeError: 'bool' object has no attribute 'to_numpy'

What versions do you use in your own environment? I'd be curious to know since you say it works fine for you. Mine are:

INSTALLED VERSIONS

commit : e8093ba
python : 3.10.5.final.0
python-bits : 64
OS : Darwin
OS-release : 17.7.0
Version : Darwin Kernel Version 17.7.0: Fri Oct 30 13:34:27 PDT 2020; root:xnu-4570.71.82.8~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : el_GR.UTF-8
LOCALE : el_GR.UTF-8

pandas : 1.4.3
numpy : 1.23.0
pytz : 2022.1
dateutil : 2.8.2
setuptools : 58.1.0
pip : 22.0.4
Cython : 0.29.30
pytest : 7.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

phofl · 2022-07-04T10:20:00Z

Hmm good point, does not work on 1.4.2, but on main

MikiPWata · 2022-07-11T22:13:30Z

Hi there,
I'm interested in working on this issue(this will be my first contribution). I'm guessing I'm supposed to write some unit tests related to this bug?

NaveenKaidbettu · 2022-07-12T14:38:04Z

Hi @phofl,

I have similar error as as @johnmantios , the issue seem to be coming even from 1.4.3 as well. The issue is being thrown on line 100 here of ..\pandas\core\missing.py. There are no accessible definition for to_numpy based on any of the imports.

jcbedoyam · 2022-07-13T18:03:26Z

I tried solving this issue by overloading the comparation operator __eq__ for the NA class since comparisons return again <NA> instead of a boolean. However this fails the arithmetics comparison tests. Is there a reason why the comparison for this class is implemented this way or can I rewrite the comparison tests to fit the new comparison functionality?

phofl · 2022-07-13T18:27:30Z

NA = NA is again NA, this happens on purpose

pspiagicw · 2022-08-08T04:59:12Z

Is anybody working on this ? Can I investigate ?

Shadimrad · 2022-08-08T20:34:43Z

NA = NA is again NA, this happens on purpose

Just out of curiosity may I ask why? @phofl

phofl · 2022-08-08T20:36:00Z

https://en.m.wikipedia.org/wiki/Three-valued_logic

Kleene logic

Shadimrad · 2022-08-08T20:40:09Z

Thank you so much!

…

On Mon, 8 Aug 2022, 1:36 pm Patrick Hoefler, ***@***.***> wrote: https://en.m.wikipedia.org/wiki/Three-valued_logic Kleene logic — Reply to this email directly, view it on GitHub <#47480 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWI7Q4YTNTVMBUCE6XAYK2LVYFVTZANCNFSM5ZSRIGKA> . You are receiving this because you commented.Message ID: ***@***.***>

phofl · 2022-08-15T17:36:54Z

Works on main

df = pd.DataFrame({'A': [0, 1, 2]}, dtype="Int64")
df.at[0, 'A'] = pd.NA
df['A'].replace(to_replace=1, value=100, inplace=True)

returns

yuanx749 · 2022-08-30T14:13:18Z

I would like to take this. @phofl
The bug exists if not specifying dtype in df = pd.DataFrame({"A": [pd.NA, 1, 2]}).

AkshayJain1995 · 2022-09-12T08:37:17Z

Hi @phofl , If the issue still exists can I pick this up?
Please assign this to me.
Thanks!

Lahiry · 2022-10-20T13:50:37Z

I just did use your reproducible example and it just works fine for me. I guess you can mark this issue as closed as this is probably fixed by now. But please let me know if you continue having the issue, because I´m interested in helping!

phofl · 2022-10-20T13:51:33Z

We try to add tests if something was fixed without getting closed

vsbits · 2022-11-18T22:24:02Z

take

vsbits · 2022-11-20T01:21:39Z

Tested using the Docker image and the bug still exists. It seems to happen only when pd.Series has dtype object. If it contains NAs but has dtype declared as Float64 or Int64 it runs just fine.

From what I found, the problem is at the function mask_missing, at .\pandas\core\missing.py (as @NaveenKaidbettu stated). On line:

new_mask = arr == x

In this situation arr has type numpy.ndarray and x is int. The evaluation is expected to return a BooleanArray, but is returning a single bool instead and raising the exeception @johnmantios posted above. Still don't know why.

Should I send pull request with failing tests?

pschwientek added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 23, 2022

phofl added good first issue Needs Tests Unit test(s) needed to prevent regressions and removed Closing Candidate May be closeable, needs more eyeballs labels Jul 2, 2022

jcbedoyam mentioned this issue Jul 12, 2022

BUG/TST fix replace with panda NAType #47480 #47688

Closed

5 tasks

phofl added Bug NA - MaskedArrays Related to pd.NA and nullable extension arrays good first issue and removed good first issue Needs Tests Unit test(s) needed to prevent regressions Bug labels Aug 8, 2022

phofl added the Needs Tests Unit test(s) needed to prevent regressions label Aug 15, 2022

github-actions bot assigned vsbits Nov 18, 2022

vsbits mentioned this issue Nov 19, 2022

TST: Test for Dataframe.replace when column contains pd.NA (#47480) #49783

Closed

3 tasks

vsbits mentioned this issue Nov 20, 2022

TST: Tests for replace method when column contains pd.NA (#47480) #49805

Closed

4 tasks

MarceloCMiguel mentioned this issue Dec 1, 2022

Fix dataframe.replace when columns contain pd.na #49988

Closed

2 tasks

vsbits removed their assignment Jan 15, 2023

phofl mentioned this issue Jan 15, 2023

BUG: replace raising RecursionError #50760

Merged

6 tasks

mroeschke closed this as completed in #50760 Jan 16, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: DataFrame.replace fails to replace value when column contains pd.NA #47480

BUG: DataFrame.replace fails to replace value when column contains pd.NA #47480

pschwientek commented Jun 23, 2022 •

edited

Loading

INSTALLED VERSIONS

phofl commented Jun 24, 2022

phofl commented Jul 2, 2022

johnmantios commented Jul 4, 2022

phofl commented Jul 4, 2022

MikiPWata commented Jul 11, 2022

NaveenKaidbettu commented Jul 12, 2022 •

edited

Loading

jcbedoyam commented Jul 13, 2022 •

edited

Loading

phofl commented Jul 13, 2022

pspiagicw commented Aug 8, 2022 •

edited

Loading

Shadimrad commented Aug 8, 2022

phofl commented Aug 8, 2022

Shadimrad commented Aug 8, 2022 via email

phofl commented Aug 15, 2022

yuanx749 commented Aug 30, 2022

AkshayJain1995 commented Sep 12, 2022

Lahiry commented Oct 20, 2022

phofl commented Oct 20, 2022

vsbits commented Nov 18, 2022

vsbits commented Nov 20, 2022

BUG: DataFrame.replace fails to replace value when column contains pd.NA #47480

BUG: DataFrame.replace fails to replace value when column contains pd.NA #47480

Comments

pschwientek commented Jun 23, 2022 • edited Loading

Pandas version checks

Reproducible Example

Issue Description

Expected Behavior

Installed Versions

INSTALLED VERSIONS

phofl commented Jun 24, 2022

phofl commented Jul 2, 2022

johnmantios commented Jul 4, 2022

INSTALLED VERSIONS

phofl commented Jul 4, 2022

MikiPWata commented Jul 11, 2022

NaveenKaidbettu commented Jul 12, 2022 • edited Loading

jcbedoyam commented Jul 13, 2022 • edited Loading

phofl commented Jul 13, 2022

pspiagicw commented Aug 8, 2022 • edited Loading

Shadimrad commented Aug 8, 2022

phofl commented Aug 8, 2022

Shadimrad commented Aug 8, 2022 via email

phofl commented Aug 15, 2022

yuanx749 commented Aug 30, 2022

AkshayJain1995 commented Sep 12, 2022

Lahiry commented Oct 20, 2022

phofl commented Oct 20, 2022

vsbits commented Nov 18, 2022

vsbits commented Nov 20, 2022

pschwientek commented Jun 23, 2022 •

edited

Loading

NaveenKaidbettu commented Jul 12, 2022 •

edited

Loading

jcbedoyam commented Jul 13, 2022 •

edited

Loading

pspiagicw commented Aug 8, 2022 •

edited

Loading