Skip to content

BUG: shift with large periods creates mulfunctioning dtypes with master branch #44978

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
wjsi opened this issue Dec 20, 2021 · 3 comments · Fixed by #45005
Closed
2 of 3 tasks

BUG: shift with large periods creates mulfunctioning dtypes with master branch #44978

wjsi opened this issue Dec 20, 2021 · 3 comments · Fixed by #45005
Labels
Bug Regression Functionality that used to work in a prior pandas version Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Milestone

Comments

@wjsi
Copy link
Contributor

wjsi commented Dec 20, 2021

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.rand(5, 3))
shifted = df.shift(6, axis=1, fill_value=None)
shifted.dtypes

Issue Description

As the DataFrame has only 3 columns, shifting with periods=6 produces an mulfunctioning dtypes attribute and error below is shown:

Traceback (most recent call last):
  File "/Users/wenjun/miniconda3/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-11-251a20aaa112>", line 1, in <module>
    shifted.dtypes
  File "/Users/wenjun/Code/pandas/pandas/core/generic.py", line 5716, in dtypes
    data = self._mgr.get_dtypes()
  File "/Users/wenjun/Code/pandas/pandas/core/internals/managers.py", line 228, in get_dtypes
    return dtypes.take(self.blknos)
  File "/Users/wenjun/Code/pandas/pandas/core/internals/managers.py", line 168, in blknos
    self._rebuild_blknos_and_blklocs()
  File "pandas/_libs/internals.pyx", line 723, in pandas._libs.internals.BlockManager._rebuild_blknos_and_blklocs
IndexError: Out of bounds on buffer access (axis 0)

The same code works under pandas 1.3.x.

I don't know if it is right to submit issues for a working branch, but this issue seems exists for a long time.

Expected Behavior

shifted.dtypes shall display a same Series as df.dtypes, or a ValueError shall be raised when periods out of bounds are provided.

Installed Versions

INSTALLED VERSIONS

commit : 3e9f09f
python : 3.9.5.final.0
python-bits : 64
OS : Darwin
OS-release : 21.1.0
Version : Darwin Kernel Version 21.1.0: Wed Oct 13 17:33:23 PDT 2021; root:xnu-8019.41.5~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : None
LOCALE : zh_CN.UTF-8
pandas : 1.4.0.dev0+1432.g3e9f09fa63
numpy : 1.21.2
pytz : 2021.3
dateutil : 2.8.2
pip : 21.1.3
setuptools : 52.0.0.post20210125
Cython : 0.29.24
pytest : 6.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.2
IPython : 7.29.0
pandas_datareader: None
bs4 : None
bottleneck : 1.3.2
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : 2.7.3
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.7.1
sqlalchemy : 1.4.27
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

@wjsi wjsi added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 20, 2021
@wjsi wjsi changed the title BUG: shift with large periods creates mulfunctioning dtypes BUG: shift with large periods creates mulfunctioning dtypes in master branch Dec 20, 2021
@wjsi wjsi changed the title BUG: shift with large periods creates mulfunctioning dtypes in master branch BUG: shift with large periods creates mulfunctioning dtypes with master branch Dec 20, 2021
@simonjayhawkins
Copy link
Member

Expected Behavior

shifted.dtypes shall display a same Series as df.dtypes, or a ValueError shall be raised when periods out of bounds are provided.

Not sure if it's the dtype attribute access that's the issue since simply printing the result of the shift operation gives the same IndexError

shifted = df.shift(6, axis=1, fill_value=None)
print(shifted)

simonjayhawkins added a commit to simonjayhawkins/pandas that referenced this issue Dec 20, 2021
@simonjayhawkins
Copy link
Member

The same code works under pandas 1.3.x.

first bad commit: [7a9975f] BUG: DataFrame.shift with axis=1 and mismatched fill_value (#44564)

cc @jbrockmendel

@simonjayhawkins simonjayhawkins added Regression Functionality that used to work in a prior pandas version Reshaping Concat, Merge/Join, Stack/Unstack, Explode and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Dec 21, 2021
@simonjayhawkins simonjayhawkins added this to the 1.4 milestone Dec 21, 2021
@simonjayhawkins
Copy link
Member

labelling as 1.4 to track the regression for 1.4.x if not fixed before. not blocker for rc.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Regression Functionality that used to work in a prior pandas version Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants