Bug in merging datetime64[ns, tz] dtypes #11405

[jreback]

closes #11405

[kawochen]

defaultdict(list) avoids the need for a lambda.

[jreback]

yep

[dflatow]

@jreback Is this still an issue merging on multiple tz-aware columns?

pandas version:
0.18.1
A:

            timestamp_start             timestamp_end  data
0 2016-12-08 17:08:49+00:00 2016-12-08 17:08:55+00:00     1
1 2016-12-08 17:08:55+00:00 2016-12-08 17:09:04+00:00     1
2 2016-12-08 17:09:04+00:00 2016-12-08 17:09:06+00:00     1
3 2016-12-08 17:09:06+00:00 2016-12-08 17:09:11+00:00     1
4 2016-12-08 17:09:11+00:00 2016-12-08 17:09:13+00:00     1

A dtypes:

timestamp_start    datetime64[ns, UTC]
timestamp_end      datetime64[ns, UTC]
data                             int64
dtype: object

B:

        timestamp_end     timestamp_start
0 2016-12-08 17:18:30 2016-12-08 17:18:21
1 2016-12-08 18:50:39 2016-12-08 18:50:13
2 2016-12-08 19:20:58 2016-12-08 19:20:50
3 2016-12-08 22:24:38 2016-12-08 22:24:24
4 2016-12-08 22:26:22 2016-12-08 22:24:41
5 2016-12-08 22:28:49 2016-12-08 22:28:41
6 2016-12-08 22:53:01 2016-12-08 22:52:22
7 2016-12-08 22:57:14 2016-12-08 22:57:00

B dtypes:

timestamp_end      datetime64[ns]
timestamp_start    datetime64[ns]
dtype: object

A.merge(B, on=["timestamp_start", "timestamp_end"])

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-119-18bced4bc0e5> in <module>()
      4 print "B:\n",B
      5 print "\nB dtypes:\n", B.dtypes
----> 6 A.merge(B, on=["timestamp_start", "timestamp_end"])

/venv/lib/python2.7/site-packages/pandas/core/frame.pyc in merge(self, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator)
   4435                      right_on=right_on, left_index=left_index,
   4436                      right_index=right_index, sort=sort, suffixes=suffixes,
-> 4437                      copy=copy, indicator=indicator)
   4438 
   4439     def round(self, decimals=0, *args, **kwargs):

/venv/lib/python2.7/site-packages/pandas/tools/merge.pyc in merge(left, right, how, on, left_on, right_on, left_index, right_index, sort, suffixes, copy, indicator)
     37                          right_index=right_index, sort=sort, suffixes=suffixes,
     38                          copy=copy, indicator=indicator)
---> 39     return op.get_result()
     40 if __debug__:
     41     merge.__doc__ = _merge_doc % '\nleft : DataFrame'

/venv/lib/python2.7/site-packages/pandas/tools/merge.pyc in get_result(self)
    215                 self.left, self.right)
    216 
--> 217         join_index, left_indexer, right_indexer = self._get_join_info()
    218 
    219         ldata, rdata = self.left._data, self.right._data

/venv/lib/python2.7/site-packages/pandas/tools/merge.pyc in _get_join_info(self)
    351              right_indexer) = _get_join_indexers(self.left_join_keys,
    352                                                  self.right_join_keys,
--> 353                                                  sort=self.sort, how=self.how)
    354             if self.right_index:
    355                 if len(self.left) > 0:

/venv/lib/python2.7/site-packages/pandas/tools/merge.pyc in _get_join_indexers(left_keys, right_keys, sort, how)
    544 
    545     # get left & right join labels and num. of levels at each location
--> 546     llab, rlab, shape = map(list, zip(* map(fkeys, left_keys, right_keys)))
    547 
    548     # get flat i8 keys from label lists

TypeError: type object argument after * must be a sequence, not itertools.imap

[jreback]

that need a nicer error message, but why would you think that should work at all? you cannot compare a tz-aware type with a tz-naive type

[dflatow]

yup, just realized that one was tz-aware and the other not. Sorry for that.

…

On Fri, Mar 24, 2017 at 10:21 AM, Jeff Reback ***@***.***> wrote: that need a nicer error message, but why would you think that should work at all? you cannot compare a tz-aware type with a tz-naive type — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#11410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFQjo1aa3nm9Qi1y73KK1nWiSWXS9d8aks5ro_umgaJpZM4GTtlE> .

-- David

[jreback]

@dflatow its actually a good catch on the error message :> see #15800 if you want to submit a fix!