Skip to content

BUG:unexpected behavior of pandas 1.1.1 dt.normalize() on pre-epoch dates #36294

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
koopmatt opened this issue Sep 11, 2020 · 2 comments · Fixed by #36557
Closed
2 of 3 tasks

BUG:unexpected behavior of pandas 1.1.1 dt.normalize() on pre-epoch dates #36294

koopmatt opened this issue Sep 11, 2020 · 2 comments · Fixed by #36557
Labels
Datetime Datetime data dtype Regression Functionality that used to work in a prior pandas version
Milestone

Comments

@koopmatt
Copy link

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample, a copy-pastable example

import pandas as pd
print(pd.__version__)
1.1.1

df = pd.DataFrame({'year': [1969, 2016],'month': [1, 1],'day': [1, 1],'hour': [9, 9]})
df=pd.to_datetime(df)

df

Out[10]: 
0   1969-01-01 09:00:00
1   2016-01-01 09:00:00
dtype: datetime64[ns]

df.dt.normalize()

Out[11]: 
0   1969-01-02
1   2016-01-01
dtype: datetime64[ns]

Problem description

I would expect pre-epoch behavior to be the same as post-epoch

Expected Output

Out[10]: 
0   1969-01-01 09:00:00
1   2016-01-01 09:00:00
dtype: datetime64[ns]

df.dt.normalize()

Out[11]: 
0   1969-01-01
1   2016-01-01
dtype: datetime64[ns]

Output of pd.show_versions()

commit : f2ca0a2 python : 3.6.10.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.17763 machine : AMD64 processor : Intel64 Family 6 Model 85 Stepping 4, GenuineIntel byteorder : little LC_ALL : None LANG : en LOCALE : None.None

pandas : 1.1.1
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.2
setuptools : 49.6.0.post20200814
Cython : 0.29.21
pytest : 6.0.1
hypothesis : None
sphinx : 3.2.1
blosc : None
feather : None
xlsxwriter : 1.3.3
lxml.etree : 4.5.2
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : 1.3.2
fsspec : 0.8.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.1
numexpr : 2.7.1
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.2
sqlalchemy : 1.3.19
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : 1.3.0
numba : 0.50.1

@koopmatt koopmatt added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 11, 2020
@dsaxton dsaxton added Regression Functionality that used to work in a prior pandas version Datetime Datetime data dtype and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Sep 11, 2020
@dsaxton
Copy link
Member

dsaxton commented Sep 11, 2020

6010141 is the first bad commit
commit 6010141
Author: jbrockmendel [email protected]
Date: Tue Jun 9 10:59:26 2020 -0700

PERF: normalize_i8_timestamps (#34672)

pandas/_libs/tslibs/conversion.pyx | 35 +++++++++++++++--------------------
1 file changed, 15 insertions(+), 20 deletions(-)

#34672 cc @jbrockmendel

@jbrockmendel
Copy link
Member

Speculation: looks like in _normalize_i8_stamp we use % without specifying @cython.cdivision(False), as a result of which we are getting c semantics for floordiv which round towards zero instead of down.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Datetime Datetime data dtype Regression Functionality that used to work in a prior pandas version
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants