Skip to content

BUG: Date offset on numpy datetime64 does not work for week and month offsets #41617

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
yohplala opened this issue May 22, 2021 · 6 comments · Fixed by #45766
Closed
2 of 3 tasks

BUG: Date offset on numpy datetime64 does not work for week and month offsets #41617

yohplala opened this issue May 22, 2021 · 6 comments · Fixed by #45766
Assignees
Labels
Bug Frequency DateOffsets good first issue Needs Tests Unit test(s) needed to prevent regressions Numeric Operations Arithmetic, Comparison, and Logical operations
Milestone

Comments

@yohplala
Copy link

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Code Sample

import numpy as np
import pandas as pd

# test data: a numpy Datetime64
ts_np = np.Datetime64('2021-01-01T08:00:00.00')

# 1 hour offset: works as expected (1 day offset also works as expected)
do=pd.tseries.frequencies.to_offset('1h')
ts_np + do

Result

Timestamp('2021-01-01 09:00:00')
# 1 week offset returns an error
do=pd.tseries.frequencies.to_offset('1W')
ts_np + do

Result

ts_np+do
Traceback (most recent call last):

  File "<ipython-input-11-e20c02eb9ea8>", line 1, in <module>
    ts_np+do

UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ms]') and dtype('O')
# 1 month begin offset returns an error
do=pd.tseries.frequencies.to_offset('1MS')

Result

ts_np+do
Traceback (most recent call last):

  File "<ipython-input-13-e20c02eb9ea8>", line 1, in <module>
    ts_np+do

UFuncTypeError: ufunc 'add' cannot use operands with types dtype('<M8[ms]') and dtype('O')

Problem description

Applying week or month offset on numpy Datetime64 does not work.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.8.0-53-generic
Version : #60~20.04.1-Ubuntu SMP Thu May 6 09:52:46 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : fr_FR.UTF-8
LOCALE : fr_FR.UTF-8

pandas : 1.2.4
numpy : 1.20.1
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.1
setuptools : 52.0.0.post20210125
Cython : 0.29.23
pytest : 6.2.3
hypothesis : None
sphinx : 4.0.1
blosc : None
feather : None
xlsxwriter : 1.3.8
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 3.0.0
IPython : 7.22.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.9.0
fastparquet : 0.6.0
gcsfs : None
matplotlib : 3.3.4
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : 2.0.0
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.15
tables : 3.6.1
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.51.2

@yohplala yohplala added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels May 22, 2021
@ghost
Copy link

ghost commented May 27, 2021

A possible workaround to this can be to use the timedelta64 function of numpy:

import numpy as np
import pandas as pd

ts_np = np.Datetime64('2021-01-01T08:00:00.00')
do = np.timedelta64(1, 'W')
ts_np + do

Output:

2021-01-08T08:00:00.000

@jbrockmendel
Copy link
Member

jbrockmendel commented Jun 6, 2021

This should be fixed by defining __array_priority__ on BaseOffset. PR would be welcome.

@jbrockmendel jbrockmendel added Frequency DateOffsets and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Jun 6, 2021
@mroeschke mroeschke added the Numeric Operations Arithmetic, Comparison, and Logical operations label Aug 21, 2021
@jbrockmendel
Copy link
Member

#44532 fixed this behavior but only implemented tests for ops with td64. A PR adding tests from the OP would close this.

@jbrockmendel jbrockmendel added the Needs Tests Unit test(s) needed to prevent regressions label Dec 21, 2021
@dhruvsamdani
Copy link
Contributor

When performing this code, it does not give the correct result.

import pandas as pd
import numpy as np

ts_np = np.Datetime64('2021-01-01T08:00:00.00')

do=pd.tseries.frequencies.to_offset('1W')
ts_np + do

Result:

2021-01-03 08:00:00

This is only two days and not the full week. Is this how it is supposed to work?

@jbrockmendel
Copy link
Member

do here is a Week offset with weekday=6, so ts_np + do is giving you the next date with res.weekday() ==6. I think you want either to_offset("7D") or pd.offsets.Week(n=1, weekday=None)

@ChrisAlbertsen
Copy link
Contributor

take

ChrisAlbertsen added a commit to ChrisAlbertsen/pandas that referenced this issue Feb 1, 2022
@mroeschke mroeschke added this to the 1.5 milestone Feb 10, 2022
mroeschke pushed a commit that referenced this issue Feb 10, 2022
* TST: add test for compatibility with numpy datetime64 #41617

* TST; add test for compatibility with numpy datetime64 (#41617)

* Fixed a fat finger error in one of the expected values

* Fixes from pre-commit [automated commit]

* changed numpy import
phofl pushed a commit to phofl/pandas that referenced this issue Feb 14, 2022
* TST: add test for compatibility with numpy datetime64 pandas-dev#41617

* TST; add test for compatibility with numpy datetime64 (pandas-dev#41617)

* Fixed a fat finger error in one of the expected values

* Fixes from pre-commit [automated commit]

* changed numpy import
yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this issue Jul 13, 2022
* TST: add test for compatibility with numpy datetime64 pandas-dev#41617

* TST; add test for compatibility with numpy datetime64 (pandas-dev#41617)

* Fixed a fat finger error in one of the expected values

* Fixes from pre-commit [automated commit]

* changed numpy import
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Frequency DateOffsets good first issue Needs Tests Unit test(s) needed to prevent regressions Numeric Operations Arithmetic, Comparison, and Logical operations
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants