Skip to content

BUG: Fix localize_pydatetime using meta datetimes as Timestamp (#25734) #25746

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.25.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -136,7 +136,7 @@ Categorical
Datetimelike
^^^^^^^^^^^^

-
- Bug with :class:`Timestamp` and :class:`CustomBusinessDay` arithmetic throwing an exception with datetime subclasses (:issue:`25734`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we don't throw an exception, rather we raise an exception. Can you even be more explicit here, what does this have to do with CBD?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to scope this description to more what was in the original issue, but really it is due to freezegun monkeypatching datetime with a subclass of datetime. A few of the offsets.py classes turn things into datetime in their apply functions. That's why this affects CustomBusinessDay and not, for instance, BusinessDay. After a brief look, a few other classes might be affected by this bug, like Easter and FY5253.

However, in writing a custom datetime class for testing, another bug was discovered in convert_datetime_to_tsobject which prevents any subclassed datetime from being converted into a Timestamp. freezegun was not affected by this by chance because it happened to implement the nanosecond property in their custom datetime class, but that fix has been included in this PR as mentioned in another comment.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, so say this affects subclasses of datetime. note btw, we don't provide any compat with this, but as usual will take patches.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fact that this affects datetime subclasses is already in the release notes, I'm not sure what you would like changed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:class:CustomBusinessDay arithmetic is misleading

-
-

Expand Down
16 changes: 11 additions & 5 deletions pandas/_libs/tslibs/conversion.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -380,9 +380,12 @@ cdef _TSObject convert_datetime_to_tsobject(datetime ts, object tz,
obj.value -= int(offset.total_seconds() * 1e9)

if not PyDateTime_CheckExact(ts):
# datetime instance but not datetime type --> Timestamp
obj.value += ts.nanosecond
obj.dts.ps = ts.nanosecond * 1000
try:
obj.value += ts.nanosecond
obj.dts.ps = ts.nanosecond * 1000
except AttributeError:
# probably a subclass of datetime
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is this probably?

pass

if nanos:
obj.value += nanos
Expand Down Expand Up @@ -608,8 +611,11 @@ cpdef inline datetime localize_pydatetime(datetime dt, object tz):
if tz is None:
return dt
elif not PyDateTime_CheckExact(dt):
# i.e. is a Timestamp
return dt.tz_localize(tz)
try:
return dt.tz_localize(tz)
except AttributeError:
# probably a subclass of datetime
pass
elif is_utc(tz):
return _localize_pydatetime(dt, tz)
try:
Expand Down
38 changes: 38 additions & 0 deletions pandas/tests/arithmetic/test_datetime64.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
from datetime import datetime, timedelta
from itertools import product, starmap
import operator
import sys
import warnings

import numpy as np
Expand All @@ -23,6 +24,8 @@
from pandas.core.indexes.datetimes import _to_M8
import pandas.util.testing as tm

from pandas.tseries.offsets import CustomBusinessDay


def assert_all(obj):
"""
Expand Down Expand Up @@ -2350,3 +2353,38 @@ def test_shift_months(years, months):
for x in dti]
expected = DatetimeIndex(raw)
tm.assert_index_equal(actual, expected)


def test_add_with_monkeypatched_datetime(monkeypatch):
# GH 25734

class MetaDatetime(type):
@classmethod
def __instancecheck__(self, obj):
return isinstance(obj, datetime)

class FakeDatetime(MetaDatetime("NewBase", (datetime,), {})):
pass
Copy link
Contributor Author

@ArtificialQualia ArtificialQualia Mar 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A metaclass is needed in this test case to be able to override __instancecheck__


with monkeypatch.context() as m:
# monkeypatch datetime everywhere
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you expand this comment to describe what freezegun (and other libraries) are doing?

for mod_name, module in list(sys.modules.items()):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least some level of monkeypatching is necessary for the original issue to show up.

However, this entire for block could be replaced with m.setattr("pandas.tseries.offsets.datetime", FakeDatetime), but I wanted to future proof this test better and replicate what freezegun and other libraries that might monkeypatch datetime would do.

try:
if (mod_name == __name__ or
module.__name__ in ('datetime',)):
continue
module_attributes = dir(module)
except (ImportError, AttributeError, TypeError):
continue
for attribute_name in module_attributes:
try:
attribute_value = getattr(module, attribute_name)
except (ImportError, AttributeError, TypeError):
continue
if id(datetime) == id(attribute_value):
m.setattr(module, attribute_name, FakeDatetime)

dt = FakeDatetime(2000, 1, 1, tzinfo=pytz.UTC)
result = Timestamp(dt) + CustomBusinessDay()
expected = Timestamp("2000-01-03", tzinfo=pytz.UTC)
assert result == expected
23 changes: 22 additions & 1 deletion pandas/tests/tslibs/test_conversion.py
Original file line number Diff line number Diff line change
@@ -1,13 +1,15 @@
# -*- coding: utf-8 -*-

from datetime import datetime

import numpy as np
import pytest
from pytz import UTC

from pandas._libs.tslib import iNaT
from pandas._libs.tslibs import conversion, timezones

from pandas import date_range
from pandas import Timestamp, date_range
import pandas.util.testing as tm


Expand Down Expand Up @@ -66,3 +68,22 @@ def test_length_zero_copy(dtype, copy):
arr = np.array([], dtype=dtype)
result = conversion.ensure_datetime64ns(arr, copy=copy)
assert result.base is (None if copy else arr)


class FakeDatetime(datetime):
pass


@pytest.mark.parametrize("dt, expected", [
pytest.param(Timestamp("2000-01-01"),
Timestamp("2000-01-01", tz=UTC), id="timestamp"),
pytest.param(datetime(2000, 1, 1),
datetime(2000, 1, 1, tzinfo=UTC),
id="datetime"),
pytest.param(FakeDatetime(2000, 1, 1),
FakeDatetime(2000, 1, 1, tzinfo=UTC),
id="fakedatetime")])
def test_localize_pydatetime_dt_types(dt, expected):
# GH 25734
result = conversion.localize_pydatetime(dt, UTC)
assert result == expected