Skip to content

implement assert_tzawareness_compat for DatetimeIndex #18376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 37 commits into from
Jan 6, 2018
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
97c32fd
implement tzawareness_compat for datetimeIndex
jbrockmendel Nov 19, 2017
2f19f51
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 19, 2017
f6a0e64
convert before asserting tzawareness_compat
jbrockmendel Nov 20, 2017
b18a006
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 23, 2017
623681f
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 24, 2017
28f8896
Update test to disallow tzaware vs tznaive comparison
jbrockmendel Nov 24, 2017
0f2287c
flake8 fixup
jbrockmendel Nov 24, 2017
171238d
Fix typo
jbrockmendel Nov 25, 2017
e95d75e
Move part of test per reviewer request
jbrockmendel Nov 25, 2017
ba65aaf
update test to include scalar comparisons
jbrockmendel Nov 25, 2017
49821d8
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 25, 2017
2be828e
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 25, 2017
b9e7c6d
revert hack, xfail test
jbrockmendel Nov 25, 2017
eefadf7
parametrize tests, comment for xfail
jbrockmendel Nov 26, 2017
470bd60
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Nov 26, 2017
abe65f8
remove debugging prints
jbrockmendel Dec 6, 2017
7cba103
rearrange isinstance checks per suggestion
jbrockmendel Dec 6, 2017
92dfed4
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 6, 2017
2d68f01
add whatsnew note
jbrockmendel Dec 6, 2017
712096b
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 7, 2017
f0c76ac
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 7, 2017
ddf1cea
fix duplicate line from merge mixup
jbrockmendel Dec 7, 2017
3b4a01e
put comment into xfail message
jbrockmendel Dec 7, 2017
aeef3d7
Fixup missing keyword
jbrockmendel Dec 7, 2017
cc5976a
Merge branch 'master' into PR_TOOL_MERGE_PR_18376
jreback Dec 12, 2017
662c447
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 12, 2017
123a52d
Merge branch 'tzbugs' of https://github.com/jbrockmendel/pandas into …
jbrockmendel Dec 12, 2017
909d07a
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 14, 2017
f21f025
give strings a free pass from tzawareness compat
jbrockmendel Dec 16, 2017
ead6e3a
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Dec 21, 2017
a6008e0
add list test cases
jbrockmendel Jan 4, 2018
0233a48
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Jan 4, 2018
e2611dc
follow precedent to separate out xfailed tests
jbrockmendel Jan 4, 2018
82ead56
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Jan 5, 2018
5653c61
move whatsnew to conversion
jbrockmendel Jan 5, 2018
615aad5
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Jan 5, 2018
31b7336
Merge branch 'master' of https://github.com/pandas-dev/pandas into tz…
jbrockmendel Jan 5, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions pandas/core/indexes/datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

from pandas.core.dtypes.common import (
_NS_DTYPE, _INT64_DTYPE,
is_object_dtype, is_datetime64_dtype,
is_object_dtype, is_datetime64_dtype, is_datetime64tz_dtype,
is_datetimetz, is_dtype_equal,
is_integer, is_float,
is_integer_dtype,
Expand Down Expand Up @@ -105,8 +105,11 @@ def _dt_index_cmp(opname, nat_result=False):

def wrapper(self, other):
func = getattr(super(DatetimeIndex, self), opname)
if (isinstance(other, datetime) or
isinstance(other, compat.string_types)):
if isinstance(other, (datetime, compat.string_types)):
if isinstance(other, compat.string_types):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would put this block outside the top-level if, then you know that you have datetimes

# We need to convert in order to assert tzawareness
other = Timestamp(other)
self._assert_tzawareness_compat(other)
other = _to_m8(other, tz=self.tz)
result = func(other)
if isna(other):
Expand All @@ -116,6 +119,7 @@ def wrapper(self, other):
other = DatetimeIndex(other)
elif not isinstance(other, (np.ndarray, Index, ABCSeries)):
other = _ensure_datetime64(other)
self._assert_tzawareness_compat(other)
result = func(np.asarray(other))
result = _values_from_object(result)

Expand Down Expand Up @@ -649,6 +653,20 @@ def _simple_new(cls, values, name=None, freq=None, tz=None,
result._reset_identity()
return result

def _assert_tzawareness_compat(self, other):
# adapted from _Timestamp._assert_tzawareness_compat
other_tz = getattr(other, 'tzinfo', None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is odd to use tzinfo, these are already wrapped scalars, or an index type, so .tz is appropriate

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was my first thought too but other could be a raw datetime.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use .tz, you can simply wrap other = pd.Timestamp(other)

if is_datetime64tz_dtype(other):
# Get tzinfo from Series dtype
other_tz = other.dtype.tz
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think would make sense to make a single routine in tslib/timezones.pyx which does the tzawareness check (make it a function), then call it both from Timestamp and here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the appeal in this but have some reservations. Do you mind saving this for the next round so I can give it some more thought?

if self.tz is None:
if other_tz is not None:
raise TypeError('Cannot compare tz-naive and tz-aware '
'datetime-like objects.')
elif other_tz is None:
raise TypeError('Cannot compare tz-naive and tz-aware '
'datetime-like objects')

@property
def tzinfo(self):
"""
Expand Down
18 changes: 14 additions & 4 deletions pandas/core/internals.py
Original file line number Diff line number Diff line change
Expand Up @@ -3486,10 +3486,20 @@ def replace_list(self, src_list, dest_list, inplace=False, regex=False,
# figure out our mask a-priori to avoid repeated replacements
values = self.as_matrix()

def comp(s):
if isna(s):
return isna(values)
return _maybe_compare(values, getattr(s, 'asm8', s), operator.eq)
if is_datetime64tz_dtype(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not the right place to fix if this is even an issue

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tend to agree, could use advice on where is the right place. Without this change there is a test failure in tests.indexing.test_coercion.TestReplaceSeriesCoercion.test_replace_series_datetime64tz with traceback

    def test_replace_series_datetime64tz(self):
        from_key = 'datetime64[ns, US/Eastern]'
        for to_key in self.rep:
>           self._assert_replace_conversion(from_key, to_key, how='dict')

pandas/tests/indexing/test_coercion.py:1317: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/tests/indexing/test_coercion.py:1239: in _assert_replace_conversion
    result = obj.replace(replacer)
pandas/core/generic.py:4514: in replace
    limit=limit, regex=regex)
pandas/core/generic.py:4563: in replace
    regex=regex)
pandas/core/internals.py:3504: in replace_list
    masks = [comp(s) for i, s in enumerate(src_list)]
pandas/core/internals.py:3502: in comp
    operator.eq)
pandas/core/internals.py:4954: in _maybe_compare
    result = op(a, b)
pandas/core/indexes/datetimes.py:122: in wrapper
    self._assert_tzawareness_compat(other)

where the relevant obj.replace(replacer) call has

index = pd.Index([3, 4], name='xxx')
data = [Timestamp('2011-01-01 00:00:00-0500', tz='US/Eastern'), Timestamp('2011-01-03 00:00:00-0500', tz='US/Eastern')]
obj = pd.Series(data, index=index, name='yyy')
replacer = {Timestamp('2011-01-01 00:00:00-0500', tz='US/Eastern'): 1.1, Timestamp('2011-01-03 00:00:00-0500', tz='US/Eastern'): 2.2}

The problem is that in internals replace_list is converting to m8 which drops tz.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not the place for this, might need to fixe .replace in the block itself.

# need to be careful not to compare tznaive to tzaware
tz = values.dtype.tz
def comp(s):
if isna(s):
return isna(values)
return _maybe_compare(values, s, operator.eq)
# TODO: Is just not-converting `s` the right thing to do here?
else:
def comp(s):
if isna(s):
return isna(values)
return _maybe_compare(values, getattr(s, 'asm8', s),
operator.eq)

masks = [comp(s) for i, s in enumerate(src_list)]

Expand Down
32 changes: 32 additions & 0 deletions pandas/tests/indexes/datetimes/test_datetime.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import operator

import pytest

import numpy as np
Expand Down Expand Up @@ -223,6 +225,36 @@ def test_append_join_nondatetimeindex(self):
# it works
rng.join(idx, how='outer')

def test_comparison_tzawareness_compat(self):
# GH#18162
dr = pd.date_range('2016-01-01', periods=6)
dz = dr.tz_localize('US/Pacific')

ops = [operator.eq, operator.ne,
operator.gt, operator.ge,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parametrize this you can make multiple tests if thats easier

operator.lt, operator.le]

for left, right in [(dr, dz), (dz, dr)]:
for op in ops:
with pytest.raises(TypeError):
op(left, right)

# Check that there isn't a problem aware-aware and naive-naive do not
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these tests should be in series (the latter ones). alternatively a better place might be tin test_base for these, where we already handle series & index tests.

# raise
assert (dr == dr).all()
assert (dz == dz).all()

naive_series = pd.Series(dr)
aware_series = pd.Series(dz)
for op in ops:
with pytest.raises(TypeError):
op(dz, naive_series)
with pytest.raises(TypeError):
op(dr, aware_series)

# TODO: implement _assert_tzawareness_compat for the reverse
# comparison with the Series on the left-hand side

def test_comparisons_coverage(self):
rng = date_range('1/1/2000', periods=10)

Expand Down