ENH: Fix total seconds in timedelta

[deponovo]

• closes BUG: total_seconds() method returns zero for timedeltas smaller then 1 microsecond #40946
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

As of now, I could not figure out what's leading to test failing in pandas\tests\scalar\timestamp\test_constructors.py:609.

[jreback]

needs a test case

[jreback]

also some tests are failing

[deponovo]

also some tests are failing

Yes, I referred that in the entry PR message. I could not figure out what the reason is and I am lacking the time to further seek the source. Maybe somebody already knows the issue or could help?
The hour difference from a given timezone flips ahead (but short) of the real timestamp flip to the next hour.

cc @mroeschke

[mroeschke]

This comment might potentially be relevant https://github.com/pandas-dev/pandas/pull/31155/files#r370721655

[jreback]

umm what is this?

[deponovo]

I could not find a way to prevent the problems that the higher precision causes in timezone-related-info (see link from @mroeschke) and because of that I am keeping the super implementation the default. However, having a total_seconds method unable to output nanosecond precision is not consistent. My solution for this was to override the method with this version that has a documented kwarg such that anybody can then know about it (possibly after a bad surprise) without having to check source code.

[jreback]

adding a keyword like this is not going to fly. if you want to try to fix that issue then happy to incorporate here (or in a pre-cursor). that looks like the test case needs to be updated.

[mroeschke]

Agreed. The signatures need to be kept compatible with the datetime.timedelta methods as Timedelta is a subclass

[deponovo]

Ok. Will retry at a later time.

[deponovo]

@mroeschke Now back to the original method signature:

    def total_seconds(self):
        """Total seconds in the duration."""
        return self.value / 1_000_000_000

This is puzzling. I can't still figure out the source of the prob. I modified the nanosecond variable from _Timestamp in pandas/_libs/tslibs/timestamps.pxd to be public (just for testing purposes) and set it to 0 by default. Then tested a boundary condition:

>>> td = pd.Timestamp(1552211999999999999,  tz="dateutil/America/Los_Angeles")
>>> td
Timestamp('2019-03-10 01:59:59.999999-0800', tz='dateutil/US/Pacific')
>>> td.value
1552211999999999999
>>> td.nanosecond
0
>>> td.nanosecond = 999  # actually just assigning the nanosecond part of the original value
>>> td  # note the false -7
Timestamp('2019-03-10 01:59:59.999999999-0700', tz='dateutil/US/Pacific')
>>> td.value
1552211999999999999
>>> td.nanosecond
999

[mroeschke]

@mroeschke Now back to the original method signature:

    def total_seconds(self):
        """Total seconds in the duration."""
        return self.value / 1_000_000_000

This is puzzling. I can't still figure out the source of the prob. I modified the nanosecond variable from _Timestamp in pandas/_libs/tslibs/timestamps.pxd to be public (just for testing purposes) and set it to 0 by default. Then tested a boundary condition:

>>> td = pd.Timestamp(1552211999999999999,  tz="dateutil/America/Los_Angeles")
>>> td
Timestamp('2019-03-10 01:59:59.999999-0800', tz='dateutil/US/Pacific')
>>> td.value
1552211999999999999
>>> td.nanosecond
0
>>> td.nanosecond = 999  # actually just assigning the nanosecond part of the original value
>>> td  # note the false -7
Timestamp('2019-03-10 01:59:59.999999999-0700', tz='dateutil/US/Pacific')
>>> td.value
1552211999999999999
>>> td.nanosecond
999

Looking into https://github.com/pandas-dev/pandas/pull/31155/files#r370721655 might solve this issue

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

[mroeschke]

Thanks for the pull request but appears to have gone stale. If you're interested in continuing please merge in main, address the comments and we can reopen.