Skip to content

BUG: Timestamp comparison with barely-out-of-bounds datetime64 #39221

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jan 21, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.3.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,7 @@ Datetimelike
- Bug in :meth:`DatetimeIndex.intersection`, :meth:`DatetimeIndex.symmetric_difference`, :meth:`PeriodIndex.intersection`, :meth:`PeriodIndex.symmetric_difference` always returning object-dtype when operating with :class:`CategoricalIndex` (:issue:`38741`)
- Bug in :meth:`Series.where` incorrectly casting ``datetime64`` values to ``int64`` (:issue:`37682`)
- Bug in :class:`Categorical` incorrectly typecasting ``datetime`` object to ``Timestamp`` (:issue:`38878`)
- Bug in comparisons between :class:`Timestamp` object and ``datetime64`` objects just outside the implementation bounds for nanosecond ``datetime64`` (:issue:`39221`)
- Bug in :meth:`Timestamp.round`, :meth:`Timestamp.floor`, :meth:`Timestamp.ceil` for values near the implementation bounds of :class:`Timestamp` (:issue:`39244`)
- Bug in :func:`date_range` incorrectly creating :class:`DatetimeIndex` containing ``NaT`` instead of raising ``OutOfBoundsDatetime`` in corner cases (:issue:`24124`)

Expand Down
29 changes: 26 additions & 3 deletions pandas/_libs/tslibs/timestamps.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,16 @@ from cpython.datetime cimport ( # alias bc `tzinfo` is a kwarg below
time,
tzinfo as tzinfo_type,
)
from cpython.object cimport Py_EQ, Py_NE, PyObject_RichCompare, PyObject_RichCompareBool
from cpython.object cimport (
Py_EQ,
Py_GE,
Py_GT,
Py_LE,
Py_LT,
Py_NE,
PyObject_RichCompare,
PyObject_RichCompareBool,
)

PyDateTime_IMPORT

Expand Down Expand Up @@ -295,6 +304,9 @@ cdef class _Timestamp(ABCTimestamp):
try:
ots = type(self)(other)
except ValueError:
if is_datetime64_object(other):
# cast non-nano dt64 to pydatetime
other = other.astype(object)
return self._compare_outside_nanorange(other, op)

elif is_array(other):
Expand Down Expand Up @@ -349,12 +361,23 @@ cdef class _Timestamp(ABCTimestamp):
cdef bint _compare_outside_nanorange(_Timestamp self, datetime other,
int op) except -1:
cdef:
datetime dtval = self.to_pydatetime()
datetime dtval = self.to_pydatetime(warn=False)

if not self._can_compare(other):
return NotImplemented

return PyObject_RichCompareBool(dtval, other, op)
if self.nanosecond == 0:
return PyObject_RichCompareBool(dtval, other, op)

# otherwise we have dtval < self
if op == Py_NE:
return True
if op == Py_EQ:
return False
if op == Py_LE or op == Py_LT:
return other.year <= self.year
if op == Py_GE or op == Py_GT:
return other.year >= self.year

cdef bint _can_compare(self, datetime other):
if self.tzinfo is not None:
Expand Down
21 changes: 20 additions & 1 deletion pandas/tests/scalar/timestamp/test_comparisons.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
from datetime import datetime
from datetime import datetime, timedelta
import operator

import numpy as np
Expand Down Expand Up @@ -244,6 +244,25 @@ def test_timestamp_compare_with_early_datetime(self):
assert stamp < datetime(2700, 1, 1)
assert stamp <= datetime(2700, 1, 1)

other = Timestamp.min.to_pydatetime(warn=False)
assert other - timedelta(microseconds=1) < Timestamp.min

def test_timestamp_compare_oob_dt64(self):
us = np.timedelta64(1, "us")
other = np.datetime64(Timestamp.min).astype("M8[us]")

# This may change if the implementation bound is dropped to match
# DatetimeArray/DatetimeIndex GH#24124
assert Timestamp.min > other
# Note: numpy gets the reversed comparison wrong

other = np.datetime64(Timestamp.max).astype("M8[us]")
assert Timestamp.max > other # not actually OOB
assert other < Timestamp.max

assert Timestamp.max < other + us
# Note: numpy gets the reversed comparison wrong

def test_compare_zerodim_array(self):
# GH#26916
ts = Timestamp.now()
Expand Down