ENH: Implement weighted rolling var and std #27682
Conversation
| result = 0 | ||
| else: | ||
| result = t * win / ((win - <float64_t>ddof) * sum_w) | ||
| if result < 0: |
There was a problem hiding this comment.
Would this only be negative if the weight was negative which we should validate much earlier?
There was a problem hiding this comment.
I guess it might be if one weight is negative and another is positive but not necessarily
pandas/core/window.py
Outdated
| @@ -173,6 +173,17 @@ def __getattr__(self, attr): | |||
| def _dir_additions(self): | |||
| return self.obj._dir_additions() | |||
|
|
|||
| def _get_kwargs(self, **kwargs) -> Tuple[dict, dict]: | |||
There was a problem hiding this comment.
Can you import Dict from typing and add subscription if possible?
There was a problem hiding this comment.
Only annotated separated kwargs of window. The other one is remaining of kwargs from apply function which seems intractable. I guess we don't need to annotate empty dict nor it is possible in python
pandas/core/window.py
Outdated
|
|
||
| def _get_roll_func( | ||
| self, cfunc: Callable, check_minp: Callable, index: np.ndarray, **kwargs | ||
| ) -> Callable: | ||
| def func(arg, window, min_periods=None, closed=None): | ||
| minp = check_minp(min_periods, len(window)) | ||
| return cfunc(arg, window, minp) | ||
| return cfunc(arg, window, minp, **kwargs) |
There was a problem hiding this comment.
rather than allowing arbitrary kwargs, can we just add a default weights=None arg?
There was a problem hiding this comment.
kwargs here is for ddof which is passed to var and std functions
|
@jreback could you review this? |
|
can you merge master and will review |
ihsansecer
force-pushed
the
weighted-roll-var
branch
from
cd7b86d to
b260ff7
Compare
|
@jreback merged |
|
can you merge master and will have another look |
|
@jreback merged |
pandas/tests/window/test_moments.py
Outdated
| # invalid method | ||
| with pytest.raises(AttributeError): | ||
| (DataFrame(vals).rolling(5, win_type="boxcar", center=True).std()) | ||
| # std |
There was a problem hiding this comment.
can you move these to a new test (may also need to rename the original to _mean)
There was a problem hiding this comment.
I parameterized the test instead. Is that alright?
|
@ihsansecer I think this is good; can you merge master and fix up conflict? |
|
thanks @ihsansecer nice patch. can you do a followup to make these nogil (@jbrockmendel recently added most window functions like this, e.g. see roll_sum) |
|
@jreback sure but I have been a bit busy these days. I will do it when I have time. |
black pandasgit diff upstream/master -u -- "*.py" | flake8 --diffUsed West's online algorithm. Here is an explanation of the algorithm and link to pdf of the paper. Tested implementation with
win_type="boxcar"comparing it with the result ofRolling.std.Additionaly,
_get_windowfunction is split into two._get_kwargsfunction is used to split kwargs for window function and rolling function (since std and var takes an additionalddofargument).Shared docs for
stdandvarare moved to be used withWindowfunctions