Skip to content

merge_asof() tolerance cannot be datetime.timedelta #28098

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ianzur opened this issue Aug 22, 2019 · 0 comments · Fixed by #28241
Closed

merge_asof() tolerance cannot be datetime.timedelta #28098

ianzur opened this issue Aug 22, 2019 · 0 comments · Fixed by #28241
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timedelta Timedelta data type

Comments

@ianzur
Copy link
Contributor

ianzur commented Aug 22, 2019

Code Sample

import pandas as pd
import numpy as np
from datetime import timedelta

left = pd.DataFrame(
    list(zip([0, 5, 10, 15, 20, 25], [0, 1, 2, 3, 4, 5])),
    columns=["time", "left"],
)
left["time"] = pd.to_timedelta(left["time"], "ms")

right = pd.DataFrame(
    list(zip([0, 3, 9, 12, 15, 18], [0, 1, 2, 3, 4, 5])),
    columns=["time", "right"],
)
right["time"] = pd.to_timedelta(right["time"], "ms")

expected = pd.DataFrame(
    list(
        zip(
            [0, 5, 10, 15, 20, 25],
            [0, 1, 2, 3, 4, 5],
            [0, np.nan, 2, 4, np.nan, np.nan],
        )
    ),
    columns=["time", "left", "right"],
)
expected["time"] = pd.to_timedelta(expected["time"], "ms")
print(expected)

# this throws MergeError: incompatible tolerance <class 'datetime.timedelta'>, must be compat with type dtype('<m8[ns]')
result = pd.merge_asof(left, right, on="time", tolerance=timedelta(microseconds=1000), direction="nearest")

# this works!
# result = pd.merge_asof(left, right, on="time", tolerance=pd.Timedelta("1ms"), direction="nearest")

print(result)

Problem description

I can use pd.Timedelta as tolerance, but not datetime.timedelta.

Expected Output

No error

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : None python : 3.7.3.final.0 python-bits : 64 OS : Linux OS-release : 4.19.0-5-amd64 machine : x86_64 processor : byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 0.25.0+222.gdef01cf7b
numpy : 1.17.0
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.2
setuptools : 41.2.0
Cython : 0.29.13
pytest : 5.1.1
hypothesis : 4.33.1
sphinx : 1.8.5
blosc : 1.8.1
feather : None
xlsxwriter : 1.1.9
lxml.etree : 4.4.1
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.7.0
pandas_datareader: None
bs4 : 4.8.0
bottleneck : 1.2.1
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.4.1
matplotlib : 3.1.1
numexpr : 2.7.0
odfpy : None
openpyxl : 2.6.3
pandas_gbq : None
pyarrow : 0.14.1
pytables : None
s3fs : 0.3.3
scipy : 1.3.1
sqlalchemy : 1.3.7
tables : 3.5.2
xarray : 0.12.3
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.1.9

@jreback jreback added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timedelta Timedelta data type labels Sep 2, 2019
mroeschke pushed a commit that referenced this issue Sep 25, 2019
* whatsnew

* 🐛🔧

* BUG: convert datetime.timedelta to nanoseconds to fix (#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (#28098)

* don't change this file

* CLN: sort imports

* BUG: (#28098) changes

* BUG (#28098) some more changes

* update whatsnew

* BUG: pass timedelta and datetime.timedelta to timedelta
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
* whatsnew

* 🐛🔧

* BUG: convert datetime.timedelta to nanoseconds to fix (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* don't change this file

* CLN: sort imports

* BUG: (pandas-dev#28098) changes

* BUG (pandas-dev#28098) some more changes

* update whatsnew

* BUG: pass timedelta and datetime.timedelta to timedelta
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
* whatsnew

* 🐛🔧

* BUG: convert datetime.timedelta to nanoseconds to fix (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* don't change this file

* CLN: sort imports

* BUG: (pandas-dev#28098) changes

* BUG (pandas-dev#28098) some more changes

* update whatsnew

* BUG: pass timedelta and datetime.timedelta to timedelta
bongolegend pushed a commit to bongolegend/pandas that referenced this issue Jan 1, 2020
* whatsnew

* 🐛🔧

* BUG: convert datetime.timedelta to nanoseconds to fix (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* BUG: mergeasof() now accepts datetime.timedelta as tolerance kwarg (pandas-dev#28098)

* don't change this file

* CLN: sort imports

* BUG: (pandas-dev#28098) changes

* BUG (pandas-dev#28098) some more changes

* update whatsnew

* BUG: pass timedelta and datetime.timedelta to timedelta
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timedelta Timedelta data type
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants