Skip to content

Improve to_datetime bounds checking #50183

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Jan 20, 2023
Merged

Improve to_datetime bounds checking #50183

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
f3d0d15
add test for float to_datetime near overflow bounds
rebecca-palmer Dec 11, 2022
dff5a3e
fix float to_datetime near overflow bounds
rebecca-palmer Dec 11, 2022
9fe5091
fix typo and formatting
rebecca-palmer Dec 11, 2022
bac5f68
fix formatting
rebecca-palmer Dec 11, 2022
535a4ec
fix test to not fail on rounding differences
rebecca-palmer Dec 11, 2022
e0b9a8f
Merge branch 'main' into datetimenearbounds
rebecca-palmer Dec 11, 2022
a5012a9
don't use approximate comparison on datetimes, it doesn't work
rebecca-palmer Dec 11, 2022
088eda5
also can't convert datetime to float
rebecca-palmer Dec 11, 2022
8b9c050
match dtypes
rebecca-palmer Dec 11, 2022
400bb22
Merge branch 'main' into datetimenearbounds
rebecca-palmer Jan 18, 2023
8ebc910
TST: don't try to use non-integer years (see #50301)
rebecca-palmer Jan 18, 2023
5ccb96a
Merge branch 'main' into datetimenearbounds
rebecca-palmer Jan 19, 2023
c17883e
TST: don't cross an integer
rebecca-palmer Jan 19, 2023
4bd1e2a
PERF: remove unnecessary copy
rebecca-palmer Jan 19, 2023
bf5542e
add whatsnew
rebecca-palmer Jan 19, 2023
2795783
Merge branch 'main' into datetimenearbounds
rebecca-palmer Jan 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 11 additions & 5 deletions pandas/_libs/tslib.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,14 +294,22 @@ def array_with_unit_to_datetime(
# if we have nulls that are not type-compat
# then need to iterate

if values.dtype.kind in ["i", "f", "u"]:
if values.dtype.kind in ["i", "u"]:
iresult = values.astype("i8", copy=False)
# fill missing values by comparing to NPY_NAT
mask = iresult == NPY_NAT
iresult[mask] = 0
# for bounds checking, which can't use (integer) iresult * mult
# because it needs arithmetic overflow to not wrap around
fvalues = iresult.astype("f8") * mult
need_to_iterate = False

if values.dtype.kind in ["f",]:
mask = (values != values) | (values == NPY_NAT) # first is NaNs
fvalues = (values * mult).astype("f8")
fvalues[mask] = 0
need_to_iterate = False

if not need_to_iterate:
# check the bounds
if (fvalues < Timestamp.min.value).any() or (
Expand All @@ -313,11 +321,9 @@ def array_with_unit_to_datetime(
result = (iresult * mult).astype("M8[ns]")

elif values.dtype.kind == "f":
fresult = (values * mult).astype("f8")
fresult[mask] = 0
if prec:
fresult = round(fresult, prec)
result = fresult.astype("M8[ns]", copy=False)
fvalues = round(fvalues, prec)
result = fvalues.astype("M8[ns]", copy=False)

iresult = result.view("i8")
iresult[mask] = NPY_NAT
Expand Down
21 changes: 21 additions & 0 deletions pandas/tests/tools/test_to_datetime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1629,6 +1629,27 @@ def test_to_timestamp_unit_coerce(self, bad_val):
result = to_datetime([1, 2, bad_val], unit="D", errors="coerce")
tm.assert_index_equal(result, expected)

def test_float_to_datetime_raise_near_bounds(self):
mag="cannot convert input with unit 'Y'"
oneyear_in_ns = 1e9 * 60 * 60 * 24 * 365.2425
tsmax_in_years = 2**63 / oneyear_in_ns # 2**63 ns, in years
should_succeed = Series(
[0, tsmax_in_years - 0.05, -tsmax_in_years + 0.05],
dtype=float
)
should_fail1 = Series([0, tsmax_in_years + 0.05], dtype=float)
should_fail2 = Series([0, -tsmax_in_years - 0.05], dtype=float)

result1=to_datetime(should_succeed, unit="Y", errors='raise')
result2=to_datetime(should_succeed, unit="Y", errors='coerce')
result3=to_datetime(should_succeed, unit="Y", errors='ignore')
tm.assert_series_equal(result1, result2)
tm.assert_series_equal(result1, result3)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A check that compared to a non to_datetime correct value would be better, but there don't seem to be any "allclose" checks in this file, and using exact equality risks rounding error failing it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. (Possibly the reason there weren't any allclose assert_almost_equal here was that that doesn't work on datetime arrays: pandas/_testing/asserters.py:731 calls assert_numpy_array_equal, which doesn't have an atol/rtol.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reported that as #50191

with pytest.raises(OutOfBoundsDatetime, match=msg):
to_datetime(should_fail1, unit="Y", errors='raise')
with pytest.raises(OutOfBoundsDatetime, match=msg):
to_datetime(should_fail2, unit="Y", errors='raise')


class TestToDatetimeDataFrame:
@pytest.fixture
Expand Down