Skip to content

BUG: Fixed timedelta numeric operations #18892

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v0.22.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -338,7 +338,8 @@ Reshaping
Numeric
^^^^^^^

-
- Fixed ``std`` and ``var`` computations for timedelta arrays not returning results in timedelta units (:issue:`18880`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a fix for 0.22? or for 0.23? (IOW do you need to have this for nansum)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was was specific to nanstd, so I think I can work around it. The skipna stuff affected all the ops I think. I'll see how hard that is to work around.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah rather leave this alone for now (IOW do it in 0.23).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rebase on master. can you move to 0.23 (docs were renamed), prob easiest to just check this file from master and past in new one

- Fixed ``skipna`` handling for some operations like ``sum`` on timedelta arrays (:issue:`18880`)
-
-

Expand Down
10 changes: 8 additions & 2 deletions pandas/core/nanops.py
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,9 @@ def _get_values(values, skipna, fill_value=None, fill_value_typ=None,
elif copy:
values = values.copy()

if is_timedelta64_dtype(values) and not skipna:
values = values.astype('float64')
values[mask] = np.nan
values = _view_if_needed(values)

# return a platform independent precision dtype
Expand Down Expand Up @@ -406,7 +409,10 @@ def _get_counts_nanvar(mask, axis, ddof, dtype=float):
@disallow('M8')
@bottleneck_switch(ddof=1)
def nanstd(values, axis=None, skipna=True, ddof=1):
result = np.sqrt(nanvar(values, axis=axis, skipna=skipna, ddof=ddof))
var_ = nanvar(values, axis=axis, skipna=skipna, ddof=ddof)
if is_timedelta64_dtype(values):
var_ = var_.value
result = np.sqrt(var_)
return _wrap_results(result, values.dtype)


Expand Down Expand Up @@ -448,7 +454,7 @@ def nanvar(values, axis=None, skipna=True, ddof=1):
# precision as the original values array.
if is_float_dtype(dtype):
result = result.astype(dtype)
return _wrap_results(result, values.dtype)
return _wrap_results(result, dtype)


@disallow('M8', 'm8')
Expand Down
33 changes: 33 additions & 0 deletions pandas/tests/series/test_analytics.py
Original file line number Diff line number Diff line change
Expand Up @@ -1215,6 +1215,39 @@ def test_timedelta64_analytics(self):
expected = Timedelta('1 days')
assert result == expected

def test_timedelta64_sum(self):
# https://github.com/pandas-dev/pandas/issues/18880
s = pd.Series(pd.timedelta_range(0, periods=12, freq='S'))
s[0] = np.nan

result = s.sum(skipna=False)
assert result is pd.NaT

result = s.sum()
assert result == pd.Timedelta(minutes=1, seconds=6)

@pytest.mark.parametrize('method', [
'sum', 'mean', 'min', 'max', 'median',
'std', 'var',
])
def test_timedelta64_many(self, method):
s_float = pd.Series(np.arange(12) * 1e3)
s_timed = pd.Series(pd.timedelta_range(0, periods=12, freq='us'))

expected = pd.Timedelta(getattr(s_float, method)())
result = getattr(s_timed, method)()
if pd.isna(expected):
assert pd.isna(result)
else:
assert result == expected

s_float[0] = np.nan
s_timed[0] = np.nan
result = getattr(s_timed, method)(skipna=False)
expected = getattr(s_float, method)(skipna=False)
assert pd.isna(result)
assert pd.isna(expected)

def test_idxmin(self):
# test idxmin
# _check_stat_op approach can not be used here because of isna check.
Expand Down
13 changes: 13 additions & 0 deletions pandas/tests/test_nanops.py
Original file line number Diff line number Diff line change
Expand Up @@ -888,6 +888,19 @@ def test_nanstd_roundoff(self):
result = data.std(ddof=ddof)
assert result == 0.0

def test_nanvar_timedelta(self):
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know this should be done up in TestnanopsDataFrame but I cannot figure out what's going on in those tests. test_nanstd and test_nanvar both have allow_tdelta=True, but were not catching the bug in #18880

result = pd.Series(dtype='m8[ns]').var()
assert result is pd.NaT

result = pd.Series([1, 1], dtype='m8[ns]').var()
assert result == pd.Timedelta(0)

result = pd.Series([10, 20], dtype='m8[ns]').var()
assert result == pd.Timedelta(50)

result = pd.Series([np.nan, 10, 20, np.nan], dtype='m8[ns]').var()
assert result == pd.Timedelta(50)

@property
def prng(self):
return np.random.RandomState(1234)
Expand Down