Skip to content

BUG to_timedelta was raising ValueError w pd.NA #53022

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 7 commits into from
May 3, 2023
Merged

BUG to_timedelta was raising ValueError w pd.NA #53022

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
b6e5195
BUG fixing timedelta
DeaMariaLeon Apr 28, 2023
b123153
BUG timedelta raising w pd.NA
DeaMariaLeon May 1, 2023
75eee4a
Merge remote-tracking branch 'upstream/main' into timedelta
DeaMariaLeon May 1, 2023
8fe3270
BUG removed not needed fail on test .. ?
DeaMariaLeon May 2, 2023
322a24d
Merge remote-tracking branch 'upstream/main' into timedelta
DeaMariaLeon May 2, 2023
3367689
BUG Parametrised test
DeaMariaLeon May 2, 2023
e614d54
Merge remote-tracking branch 'upstream/main' into timedelta
DeaMariaLeon May 2, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v2.0.2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,11 @@ Bug fixes
~~~~~~~~~
- Bug in :func:`api.interchange.from_dataframe` was returning :class:`DataFrame`'s of incorrect sizes when called on slices (:issue:`52824`)
- Bug in :func:`api.interchange.from_dataframe` was unnecessarily raising on bitmasks (:issue:`49888`)
- Bug in :func:`to_timedelta` was raising ``ValueError`` with ``pandas.NA`` (:issue:`52909`)
- Bug in :meth:`DataFrame.convert_dtypes` ignores ``convert_*`` keywords when set to False ``dtype_backend="pyarrow"`` (:issue:`52872`)
- Bug in :meth:`Series.describe` treating pyarrow-backed timestamps and timedeltas as categorical data (:issue:`53001`)
- Bug in :meth:`pd.array` raising for ``NumPy`` array and ``pa.large_string`` or ``pa.large_binary`` (:issue:`52590`)
-


.. ---------------------------------------------------------------------------
.. _whatsnew_202.other:
Expand Down
6 changes: 3 additions & 3 deletions pandas/_libs/tslibs/timedeltas.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ import_datetime()


cimport pandas._libs.tslibs.util as util
from pandas._libs.missing cimport checknull_with_nat_and_na
from pandas._libs.tslibs.base cimport ABCTimestamp
from pandas._libs.tslibs.conversion cimport (
cast_from_unit,
Expand Down Expand Up @@ -341,8 +342,7 @@ cdef convert_to_timedelta64(object ts, str unit):
Return an ns based int64
"""
# Caller is responsible for checking unit not in ["Y", "y", "M"]

if checknull_with_nat(ts):
if checknull_with_nat_and_na(ts):
return np.timedelta64(NPY_NAT, "ns")
elif isinstance(ts, _Timedelta):
# already in the proper format
Expand Down Expand Up @@ -1808,7 +1808,7 @@ class Timedelta(_Timedelta):
# unit=None is de-facto 'ns'
unit = parse_timedelta_unit(unit)
value = convert_to_timedelta64(value, unit)
elif checknull_with_nat(value):
elif checknull_with_nat_and_na(value):
return NaT
else:
raise ValueError(
Expand Down
2 changes: 1 addition & 1 deletion pandas/tests/series/methods/test_astype.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,7 +471,7 @@ class TestAstypeString:
def test_astype_string_to_extension_dtype_roundtrip(
self, data, dtype, request, nullable_string_dtype
):
if dtype == "boolean" or (dtype == "timedelta64[ns]" and NaT in data):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just for my understanding, why did this one change?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah it's because

ser.astype(nullable_string_dtype)

introduces '<NA>', and then .astype('timedelta64[ns]') couldn't handle that (but it can after this PR!)

if dtype == "boolean":
mark = pytest.mark.xfail(
reason="TODO StringArray.astype() with missing values #GH40566"
)
Expand Down
6 changes: 5 additions & 1 deletion pandas/tests/tools/test_to_timedelta.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -214,11 +214,15 @@ def test_to_timedelta_on_missing_values(self):
actual = to_timedelta(ser)
tm.assert_series_equal(actual, expected)

@pytest.mark.parametrize("val", [np.nan, pd.NaT])
@pytest.mark.parametrize("val", [np.nan, pd.NaT, pd.NA])
def test_to_timedelta_on_missing_values_scalar(self, val):
actual = to_timedelta(val)
assert actual._value == np.timedelta64("NaT").astype("int64")

def test_to_timedelta_on_missing_values_list(self):
actual = to_timedelta([pd.NA])
assert actual[0]._value == np.timedelta64("NaT").astype("int64")

def test_to_timedelta_float(self):
# https://github.com/pandas-dev/pandas/issues/25077
arr = np.arange(0, 1, 1e-6)[-10:]
Expand Down