Skip to content

BUG: RecursionError when replace(np.nan, value=None) encounters a NaN #48231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
TheNeuralBit opened this issue Aug 24, 2022 · 3 comments
Closed
2 of 3 tasks
Labels
Bug Regression Functionality that used to work in a prior pandas version replace replace method
Milestone

Comments

@TheNeuralBit
Copy link
Contributor

TheNeuralBit commented Aug 24, 2022

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
ok = pd.Series([1.0, 2.0])
ok.replace(np.nan, value=None)
bad = pd.Series([1.0, np.nan])
bad.replace(np.nan, value=None)
# RecursionError

Issue Description

Replacing NaNs with None encounters infinite recursion. I've only reproduced this in 1.5.0rc0, I confirmed it's not reproducible on 1.4.5, have not checked main.

Full stacktrace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/series.py", line 5380, in replace
    return super().replace(
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/util/_decorators.py", line 317, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/generic.py", line 7251, in replace
    new_data = self._mgr.replace(
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 468, in replace
    return self.apply(
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 348, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 613, in replace
    return blk.replace(
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 613, in replace
    return blk.replace(
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 613, in replace
    return blk.replace(
  [Previous line repeated 978 more times]
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 612, in replace
    blk = self.coerce_to_target_dtype(value)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 451, in coerce_to_target_dtype
    return self.astype(new_dtype, copy=False)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 527, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 299, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/astype.py", line 217, in astype_array
    if is_datetime64tz_dtype(dtype) and is_datetime64_dtype(values.dtype):
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/common.py", line 388, in is_datetime64tz_dtype
    return DatetimeTZDtype.is_dtype(arr_or_dtype)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/base.py", line 312, in is_dtype
    if isinstance(dtype, (ABCSeries, ABCIndex, ABCDataFrame, np.dtype)):
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 47, in _instancecheck
    return _check(inst) and not isinstance(inst, type)
  File "/usr/local/google/home/bhulette/.pyenv/versions/beam/lib/python3.8/site-packages/pandas/core/dtypes/generic.py", line 41, in _check
    return getattr(inst, attr, "_typ") in comp
RecursionError: maximum recursion depth exceeded while calling a Python object

Expected Behavior

The same behavior as pandas 1.4.5:

>>> import pandas as pd
>>> import numpy as np
>>> pd.Series([1.0, np.nan]).replace(np.nan, value=None)
0     1.0
1    None
dtype: object

Installed Versions

INSTALLED VERSIONS

commit : 224458e
python : 3.8.6.final.0
python-bits : 64
OS : Linux
OS-release : 5.17.11-1rodete2-amd64
Version : #1 SMP PREEMPT Debian 5.17.11-1rodete2 (2022-06-09)
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.5.0rc0
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
setuptools : 49.2.1
pip : 20.2.1
Cython : None
pytest : 7.1.2
hypothesis : None
sphinx : 5.0.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : 1.0.2
psycopg2 : 2.9.3
jinja2 : 3.1.1
IPython : 8.3.0
pandas_datareader: None
bs4 : 4.11.1
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 7.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.8.0
snappy : None
sqlalchemy : 1.4.35
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : 0.18.0
tzdata : None

@phofl
Copy link
Member

phofl commented Aug 24, 2022

Does not work on main either, labelling as regression then

@phofl phofl added Regression Functionality that used to work in a prior pandas version replace replace method and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 24, 2022
@phofl phofl added this to the 1.5 milestone Aug 24, 2022
@simonjayhawkins
Copy link
Member

duplicate of #45725

@TheNeuralBit
Copy link
Contributor Author

Thank you @simonjayhawkins and @phofl. I appreciate the quick response (as usual) :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Regression Functionality that used to work in a prior pandas version replace replace method
Projects
None yet
Development

No branches or pull requests

3 participants