Skip to content

BUG: rendering dt64tz values with non-pytz #49684

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Nov 16, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v2.0.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -610,6 +610,8 @@ Datetimelike
- Bug in :class:`DatetimeIndex` constructor failing to raise when ``tz=None`` is explicitly specified in conjunction with timezone-aware ``dtype`` or data (:issue:`48659`)
- Bug in subtracting a ``datetime`` scalar from :class:`DatetimeIndex` failing to retain the original ``freq`` attribute (:issue:`48818`)
- Bug in ``pandas.tseries.holiday.Holiday`` where a half-open date interval causes inconsistent return types from :meth:`USFederalHolidayCalendar.holidays` (:issue:`49075`)
- Bug in rendering :class:`DatetimeIndex` and :class:`Series` and :class:`DataFrame` with timezone-aware dtypes with ``dateutil`` or ``zoneinfo`` timezones near daylight-savings transitions (:issue:`49684`)
-

Timedelta
^^^^^^^^^
Expand Down
1 change: 0 additions & 1 deletion pandas/_libs/tslibs/vectorized.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,6 @@ def get_resolution(
def ints_to_pydatetime(
arr: npt.NDArray[np.int64],
tz: tzinfo | None = ...,
fold: bool = ...,
box: str = ...,
reso: int = ..., # NPY_DATETIMEUNIT
) -> npt.NDArray[np.object_]: ...
Expand Down
4 changes: 2 additions & 2 deletions pandas/_libs/tslibs/vectorized.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,6 @@ def tz_convert_from_utc(ndarray stamps, tzinfo tz, NPY_DATETIMEUNIT reso=NPY_FR_
def ints_to_pydatetime(
ndarray stamps,
tzinfo tz=None,
bint fold=False,
str box="datetime",
NPY_DATETIMEUNIT reso=NPY_FR_ns,
) -> np.ndarray:
Expand Down Expand Up @@ -136,6 +135,7 @@ def ints_to_pydatetime(
tzinfo new_tz
bint use_date = False, use_ts = False, use_pydt = False
object res_val
bint fold = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So fold never needs to be set to 1 for the repr?

Also fold in the docstring can be removed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So fold never needs to be set to 1 for the repr?

it doesn't matter in pytz cases and it gets set via pointer in non-pytz cases

Also fold in the docstring can be removed

Will do in follow-up if thats OK

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah follow up PR is good


# Note that `result` (and thus `result_flat`) is C-order and
# `it` iterates C-order as well, so the iteration matches
Expand Down Expand Up @@ -168,7 +168,7 @@ def ints_to_pydatetime(

else:

local_val = info.utc_val_to_local_val(utc_val, &pos)
local_val = info.utc_val_to_local_val(utc_val, &pos, &fold)
if info.use_pytz:
# find right representation of dst etc in pytz timezone
new_tz = tz._tzinfos[tz._transition_info[pos]]
Expand Down
40 changes: 40 additions & 0 deletions pandas/tests/arrays/test_datetimes.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,11 @@
from datetime import timedelta
import operator

try:
from zoneinfo import ZoneInfo
except ImportError:
ZoneInfo = None

import numpy as np
import pytest

Expand Down Expand Up @@ -706,3 +711,38 @@ def test_tz_localize_t2d(self):

roundtrip = expected.tz_localize("US/Pacific")
tm.assert_datetime_array_equal(roundtrip, dta)

easts = ["US/Eastern", "dateutil/US/Eastern"]
if ZoneInfo is not None:
try:
easts.append(ZoneInfo("US/Eastern"))
except KeyError:
# No tzdata
pass

@pytest.mark.parametrize("tz", easts)
def test_iter_zoneinfo_fold(self, tz):
# GH#49684
utc_vals = np.array(
[1320552000, 1320555600, 1320559200, 1320562800], dtype=np.int64
)
utc_vals *= 1_000_000_000

dta = DatetimeArray(utc_vals).tz_localize("UTC").tz_convert(tz)

left = dta[2]
right = list(dta)[2]
assert str(left) == str(right)
# previously there was a bug where with non-pytz right would be
# Timestamp('2011-11-06 01:00:00-0400', tz='US/Eastern')
# while left would be
# Timestamp('2011-11-06 01:00:00-0500', tz='US/Eastern')
# The .value's would match (so they would compare as equal),
# but the folds would not
assert left.utcoffset() == right.utcoffset()

# The same bug in ints_to_pydatetime affected .astype, so we test
# that here.
right2 = dta.astype(object)[2]
assert str(left) == str(right2)
assert left.utcoffset() == right2.utcoffset()