Skip to content

BUG: preserve fold in Timestamp.replace #37644

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Nov 8, 2020
Merged

BUG: preserve fold in Timestamp.replace #37644

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
30f7275
BUG: preserve fold in Timestamp.replace
AlexKirko Nov 5, 2020
b2745e1
DOC: add whatsnew
AlexKirko Nov 5, 2020
096ee9b
TST: add test
AlexKirko Nov 5, 2020
b9fa491
DOC: mirror doc change for NaT
AlexKirko Nov 5, 2020
b4c4127
TST: move the test
AlexKirko Nov 5, 2020
7c2f782
TST: improve test readability
AlexKirko Nov 5, 2020
2ca31ca
TST: incorporate reviewer suggestions
AlexKirko Nov 5, 2020
4719697
TST: replace second instead of fold
AlexKirko Nov 5, 2020
ada4d94
TST: tzinfo instead of tz in the test
AlexKirko Nov 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v1.2.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,7 @@ Timezones
^^^^^^^^^

- Bug in :func:`date_range` was raising AmbiguousTimeError for valid input with ``ambiguous=False`` (:issue:`35297`)
-
- Bug in :meth:`Timestamp.replace` was losing fold information (:issue:`37610`)


Numeric
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/nattype.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -774,7 +774,7 @@ default 'raise'
microsecond : int, optional
nanosecond : int, optional
tzinfo : tz-convertible, optional
fold : int, optional, default is 0
fold : int, optional

Returns
-------
Expand Down
9 changes: 7 additions & 2 deletions pandas/_libs/tslibs/timestamps.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1374,7 +1374,7 @@ default 'raise'
microsecond=None,
nanosecond=None,
tzinfo=object,
fold=0,
fold=None,
):
"""
implements datetime.replace, handles nanoseconds.
Expand All @@ -1390,7 +1390,7 @@ default 'raise'
microsecond : int, optional
nanosecond : int, optional
tzinfo : tz-convertible, optional
fold : int, optional, default is 0
fold : int, optional

Returns
-------
Expand All @@ -1407,6 +1407,11 @@ default 'raise'
# set to naive if needed
tzobj = self.tzinfo
value = self.value

# GH 37610. Preserve fold when replacing.
if fold is None:
fold = self.fold

if tzobj is not None:
value = tz_convert_from_utc_single(value, tzobj)

Expand Down
10 changes: 10 additions & 0 deletions pandas/tests/scalar/timestamp/test_unary_ops.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -424,3 +424,13 @@ def test_timestamp(self):
# should agree with datetime.timestamp method
dt = ts.to_pydatetime()
assert dt.timestamp() == ts.timestamp()


def test_replace_preserves_fold():
# GH 37610. Check that replace preserves Timestamp fold property
tz = gettz("Europe/Moscow")

result = Timestamp(1256427000000000000, tz=tz, unit="ns").replace(tzinfo=tz).fold
Copy link
Contributor

@pganssle pganssle Nov 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few things:

  1. It should be much more obvious that this thing started with the fold set correctly. This could fail for reasons unrelated to the thing you are testing.
  2. You should test that it works both ways fold=0 → fold=0 and fold=1 → fold=1.
  3. When you do replace(tzinfo=tz), you are creating an identical timestamp. I suspect that it would be very reasonable for a future version of Python or Pandas to optimize that case and simply return the original Timestamp/datetime if nothing is changed, making this not an ideal test.
  4. This method-chaining style is not going to be great in terms of tracebacks if the test fails. The same line covers three separate things that could fail: constructing the Timestamp, executing replace and accessing the .fold property. I would do these on separate lines.

Probably something more like this:

zone = gettz("America/New_York")

ts = Timestamp(2020, 11, 1, 1, 30, fold=fold, tzinfo=zone)
ts_replaced = ts.replace(microseconds=1)
assert ts_replaced.fold == fold

Where fold is parameterized over [0, 1].

Copy link
Member Author

@AlexKirko AlexKirko Nov 5, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pganssle Thanks! I had made changes to the test, instead testing our behavior versus datetime, but your suggestion makes more sense. This way, we don't care whether datetime works properly (on the slim chance that it breaks). Introduced the changes. Hope you don't mind me keeping the OP timezone.

Please take a look at the new version.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can make the replacing of tzinfo work as a reasonable test if you change to the other timezone with the same time of DST change, e.g Moscow to Novosibirsk

Copy link
Member Author

@AlexKirko AlexKirko Nov 6, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexeyDmitriev That would work, but also make the test a bit less readable (we'd need to comment that we are swapping between two DST-zones, and both times are in a fold). I think leaving the current test should be okay.

expected = 1

assert result == expected