Skip to content

ENH: Fix total seconds in timedelta #45129

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v1.4.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,8 @@ Other enhancements
- :class:`ExtensionDtype` and :class:`ExtensionArray` are now (de)serialized when exporting a :class:`DataFrame` with :meth:`DataFrame.to_json` using ``orient='table'`` (:issue:`20612`, :issue:`44705`).
- Add support for `Zstandard <http://facebook.github.io/zstd/>`_ compression to :meth:`DataFrame.to_pickle`/:meth:`read_pickle` and friends (:issue:`43925`)
- :meth:`DataFrame.to_sql` now returns an ``int`` of the number of written rows (:issue:`23998`)

- :meth:`Timedelta.total_seconds()` now properly taking into account any nanoseconds contribution (:issue:`40946`)
-

.. ---------------------------------------------------------------------------

Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/conversion.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -504,7 +504,7 @@ cdef _TSObject convert_datetime_to_tsobject(datetime ts, tzinfo tz,

if obj.tzinfo is not None and not is_utc(obj.tzinfo):
offset = get_utcoffset(obj.tzinfo, ts)
obj.value -= int(offset.total_seconds() * 1e9)
obj.value -= int(offset.total_seconds() * 1_000_000_000)

if isinstance(ts, ABCTimestamp):
obj.value += <int64_t>ts.nanosecond
Expand Down
26 changes: 25 additions & 1 deletion pandas/_libs/tslibs/nattype.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,31 @@ class NaTType(_NaT):
Monday == 1 ... Sunday == 7.
""",
)
total_seconds = _make_nan_func("total_seconds", timedelta.total_seconds.__doc__)
total_seconds = _make_nan_func(
"total_seconds",
"""
Total seconds in the duration with default us precision
(for compatibility with `datetime.timedelta`).

Parameters
----------
ns_precision : bool, default False
Return the duration with ns precision.

Examples
--------
>>> td = pd.Timedelta(days=6, minutes=50, seconds=3,
... milliseconds=10, microseconds=10, nanoseconds=12)
>>> td
Timedelta('6 days 00:50:03.010010012')

>>> td.total_seconds()
521403.01001

>>> td.total_seconds(ns_precision=True)
521403.010010012
"""
)
month_name = _make_nan_func(
"month_name",
"""
Expand Down
28 changes: 28 additions & 0 deletions pandas/_libs/tslibs/timedeltas.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -1539,6 +1539,34 @@ class Timedelta(_Timedelta):
div = other // self
return div, other - div * self

# GH45129
def total_seconds(self, *, ns_precision=False) -> float:
"""
Total seconds in the duration with default us precision
(for compatibility with `datetime.timedelta`).

Parameters
----------
ns_precision : bool, default False
Return the duration with ns precision.

Examples
--------
>>> td = pd.Timedelta(days=6, minutes=50, seconds=3,
... milliseconds=10, microseconds=10, nanoseconds=12)
>>> td
Timedelta('6 days 00:50:03.010010012')

>>> td.total_seconds()
521403.01001

>>> td.total_seconds(ns_precision=True)
521403.010010012
"""
if ns_precision:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

umm what is this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I could not find a way to prevent the problems that the higher precision causes in timezone-related-info (see link from @mroeschke) and because of that I am keeping the super implementation the default. However, having a total_seconds method unable to output nanosecond precision is not consistent. My solution for this was to override the method with this version that has a documented kwarg such that anybody can then know about it (possibly after a bad surprise) without having to check source code.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

adding a keyword like this is not going to fly. if you want to try to fix that issue then happy to incorporate here (or in a pre-cursor). that looks like the test case needs to be updated.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. The signatures need to be kept compatible with the datetime.timedelta methods as Timedelta is a subclass

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. Will retry at a later time.

return self.value / 1_000_000_000
return super().total_seconds()


cdef bint is_any_td_scalar(object obj):
"""
Expand Down
21 changes: 21 additions & 0 deletions pandas/tests/tslibs/test_timedeltas.py
Original file line number Diff line number Diff line change
Expand Up @@ -46,3 +46,24 @@ def test_huge_nanoseconds_overflow():
# GH 32402
assert delta_to_nanoseconds(Timedelta(1e10)) == 1e10
assert delta_to_nanoseconds(Timedelta(nanoseconds=1e10)) == 1e10


# GH40946
@pytest.mark.parametrize(
"obj, expected_ns, expected_us",
[
(Timedelta("1us"), 1e-6, 1e-6),
(Timedelta("500ns"), 5e-7, 0.0),
(Timedelta(nanoseconds=500), 5e-7, 0.0),
(Timedelta(seconds=1, nanoseconds=500), 1 + 5e-7, 1.0),
(Timedelta(seconds=1e-9, milliseconds=1e-5, microseconds=1e-1), 111e-9, 0.0),
(
Timedelta(days=1, seconds=1e-9, milliseconds=1e-5, microseconds=1e-1),
24 * 3600 + 111e-9,
24 * 3600,
),
],
)
def test_total_seconds(obj: Timedelta, expected_ns, expected_us):
assert obj.total_seconds() == expected_us
assert obj.total_seconds(ns_precision=True) == expected_ns