BUG: Incompatible tolerance error in merge_asof with comaptible types.

[jkibele]

• [X ] I have checked that this issue has not already been reported.

• [X ] I have confirmed this bug exists on the latest version of pandas.
  Crashes in 1.1.0, but works in 1.0.5

• [X ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

dr1 = pd.date_range(start='1/1/2020', end='2/1/2020')
dr2 = pd.date_range(start='1/1/2020', end='1/20/2020', freq='2D') + pd.Timedelta(seconds=0.4)

df1 = pd.DataFrame({'val1': 'foo'}, index=pd.DatetimeIndex(dr1))
df2 = pd.DataFrame({'val2': 'bar'}, index=pd.DatetimeIndex(dr2))

td = pd.Timedelta(seconds=0.5)
pd.merge_asof(df2, df1, left_index=True, right_index=True, tolerance=td)

Problem description

When trying to use merge_asof in 1.1.0 (and in master), I get message about "incompatible tolerance" even though I'm using pandas.DatetimeIndex and pandas.Timedelta for the tolerance.

Well, technically, I actually get UnbondLocalError because a local variable is referenced before assignment while trying to generate the error message, but it looks like the underlying problem is something to do with an incompatible tolerance error.

Here's the traceback:

---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
<ipython-input-4-c18851904e91> in <module>
      1 td = pd.Timedelta(seconds=0.5)
----> 2 pd.merge_asof(df2, df1, left_index=True, right_index=True, tolerance=td)

~/miniconda3/envs/pandas_problem/lib/python3.8/site-packages/pandas/core/reshape/merge.py in merge_asof(left, right, on, left_on, right_on, left_index, right_index, by, left_by, right_by, suffixes, tolerance, allow_exact_matches, direction)
    544     4 2016-05-25 13:30:00.048   AAPL   98.00       100     NaN     NaN
    545     """
--> 546     op = _AsOfMerge(
    547         left,
    548         right,

~/miniconda3/envs/pandas_problem/lib/python3.8/site-packages/pandas/core/reshape/merge.py in __init__(self, left, right, on, left_on, right_on, left_index, right_index, by, left_by, right_by, axis, suffixes, copy, fill_method, how, tolerance, allow_exact_matches, direction)
   1571         self.direction = direction
   1572 
-> 1573         _OrderedMerge.__init__(
   1574             self,
   1575             left,

~/miniconda3/envs/pandas_problem/lib/python3.8/site-packages/pandas/core/reshape/merge.py in __init__(self, left, right, on, left_on, right_on, left_index, right_index, axis, suffixes, copy, fill_method, how)
   1465 
   1466         self.fill_method = fill_method
-> 1467         _MergeOperation.__init__(
   1468             self,
   1469             left,

~/miniconda3/envs/pandas_problem/lib/python3.8/site-packages/pandas/core/reshape/merge.py in __init__(self, left, right, how, on, left_on, right_on, axis, left_index, right_index, sort, suffixes, copy, indicator, validate)
    650             self.right_join_keys,
    651             self.join_names,
--> 652         ) = self._get_merge_keys()
    653 
    654         # validate the merge keys dtypes. We may need to coerce

~/miniconda3/envs/pandas_problem/lib/python3.8/site-packages/pandas/core/reshape/merge.py in _get_merge_keys(self)
   1667 
   1668             msg = (
-> 1669                 f"incompatible tolerance {self.tolerance}, must be compat "
   1670                 f"with type {repr(lk.dtype)}"
   1671             )

UnboundLocalError: local variable 'lk' referenced before assignment

Expected Output

Here's what I get in version 1.0.4, and what I think I should get.

                         val2 val1
2020-01-01 00:00:00.400  bar  foo
2020-01-03 00:00:00.400  bar  foo
2020-01-05 00:00:00.400  bar  foo
2020-01-07 00:00:00.400  bar  foo
2020-01-09 00:00:00.400  bar  foo
2020-01-11 00:00:00.400  bar  foo
2020-01-13 00:00:00.400  bar  foo
2020-01-15 00:00:00.400  bar  foo
2020-01-17 00:00:00.400  bar  foo
2020-01-19 00:00:00.400  bar  foo

Output of pd.show_versions()

INSTALLED VERSIONS

commit : a4203cf
python : 3.8.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.0-7634-generic
Version : #38159534531720.04~a8480ad-Ubuntu SMP Wed Jul 22 15:13:45 UTC
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.2.0.dev0+29.ga4203cf8d
numpy : 1.19.1
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 49.2.0.post20200714
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

I hope I did this right. Just trying to help. Thanks.

[TomAugspurger]

Thanks for the report. Looks like a bad substitution at https://github.com/pandas-dev/pandas/pull/31556/files#diff-14fc0a514ad9ce2a73eeaa9ec3fb9406R1647.

In

pandas/pandas/core/reshape/merge.py

Line 1670 in 3701a9b

f"with type {repr(lk.dtype)}"

we should change the lk to lt.

[simonjayhawkins]

can confirm that before changes in #31556 was giving expected output

>>> pd.__version__
'1.1.0.dev0+325.g4fa44b77d'
>>>
>>> dr1 = pd.date_range(start='1/1/2020', end='2/1/2020')
>>> dr2 = pd.date_range(start='1/1/2020', end='1/20/2020', freq='2D') + pd.Timedelta(seconds=0.4)
>>>
>>> df1 = pd.DataFrame({'val1': 'foo'}, index=pd.DatetimeIndex(dr1))
>>> df2 = pd.DataFrame({'val2': 'bar'}, index=pd.DatetimeIndex(dr2))
>>>
>>> td = pd.Timedelta(seconds=0.5)
>>> pd.merge_asof(df2, df1, left_index=True, right_index=True, tolerance=td)
                        val2 val1
2020-01-01 00:00:00.400  bar  foo
2020-01-03 00:00:00.400  bar  foo
2020-01-05 00:00:00.400  bar  foo
2020-01-07 00:00:00.400  bar  foo
2020-01-09 00:00:00.400  bar  foo
2020-01-11 00:00:00.400  bar  foo
2020-01-13 00:00:00.400  bar  foo
2020-01-15 00:00:00.400  bar  foo
2020-01-17 00:00:00.400  bar  foo
2020-01-19 00:00:00.400  bar  foo
>>>

[ikedaosushi]

May I take the issue?

[simonjayhawkins]

go for it @ikedaosushi . will also need a test. maybe need to simplify code sample.

[ikedaosushi]

Thanks. I understand. I'm gonna write it as well as fixing.