ENH: Min_weight for Rolling #12750

max-sixty · 2016-03-30T16:57:48Z

Initial implementation for #11167.

Needs some additional tests - for dataframes, other axis values etc, but want to get this out there and get feedback.

Note that the implemenation is different than that for min_periods. This uses a single function to mask all rolling calculations. min_periods is implemented separately for each function. I imagine the latter is a bit faster, although also more complicated.

closes ENH: min_weight in addition to min_periods for ewma #11167
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

jreback · 2016-03-30T17:04:55Z

is this in concert with min_periods? (e.g. looks like this), or better as either/or?

max-sixty · 2016-03-30T17:53:21Z

While you could use them together, most cases are going to be an either / or.

If it's just either / or, we could implement within min_periods - a value < 1 going to min_weight and otherwise going to min_periods. But I think less explicit.

jreback · 2016-03-30T18:00:00Z

min_periods=None is the default, just wondering if it makes any sense to do it and/or raise if BOTH min_weight is not None and min_periods. You don't need to have special values.

Furthermore does the order of application of min_periods and min_weight matter?

give this some tests, if its too complicated / not worth it, ok w/o supporting both simultaneously

max-sixty · 2016-03-30T20:33:08Z

wondering if it makes any sense to do it and/or raise if BOTH min_weight is not None and min_periods.

I think it's fine to have both - it will put a NaN in if either test fails. They are independent, so order doesn't matter.

give this some tests, if its too complicated / not worth it, ok w/o supporting both simultaneously

My thought exactly

jreback · 2016-04-01T13:44:14Z

pandas/core/window.py

@@ -1373,6 +1386,19 @@ def _check_func(minp, window):
    return _check_func


+def _min_weight_mask(rolling, min_weight):
+    data = rolling.obj


can you add a comment explaining what this is doing/why

jreback · 2016-04-01T13:47:47Z

update the doc-strings (Window/Rolling/Expanding/EWM), with versionadded tags.
whatsnew sub-section to show mini-example and point to updated docs
add to docs
more tests. Test for some edge cases (empty, 1 point, all-nan)

max-sixty · 2016-04-01T15:20:05Z

Yes will do - this was WIP. Thanks for the comments

jreback · 2016-04-04T12:27:48Z

pandas/tests/test_window.py

+        result = rolling.mean()
+
+        # check that all points between 25 and 90 are NaN
+        self.assertTrue(result.iloc[24:89].isnull().all())


construct the actual result

jreback · 2016-05-07T18:52:09Z

can you rebase / update

codecov-io · 2016-05-17T04:32:50Z

Current coverage is 84.65% (diff: 100%)

Merging #12750 into master will increase coverage by <.01%

@@             master     #12750   diff @@
==========================================
  Files           144        144          
  Lines         51025      51038    +13   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43191      43205    +14   
+ Misses         7834       7833     -1   
  Partials          0          0

Powered by Codecov. Last update 4c3d4d4...3c42153

jreback · 2016-05-20T13:29:38Z

pandas/core/generic.py

@@ -5175,30 +5175,34 @@ def _add_series_or_dataframe_operations(cls):
        from pandas.core import window as rwindow

        @Appender(rwindow.rolling.__doc__)
-        def rolling(self, window, min_periods=None, freq=None, center=False,


add to the doc-string min_weight with a versionadded tag (I don't remember exactly where these are defined)

max-sixty · 2016-07-14T18:25:35Z

I've been remiss about the final fixes here. Will need a couple of weeks on other commitments but I will get to this

jreback · 2016-09-09T22:38:28Z

can you rebase / update?

jreback · 2016-11-16T22:21:45Z

if you can update

max-sixty · 2016-11-16T23:29:07Z

Yes, apologies, this is another I'll attempt over / after Thanksgiving. Appreciate the patience!

jreback · 2016-12-21T23:03:31Z

looks like you rebased, great.

needs some docs (computation.rst), whatsnew (you can do a sub-section or just put a ref to the docs) & doc-string examples.

jreback · 2017-01-21T23:17:28Z

status?

jreback · 2017-03-20T13:54:55Z

can you update

max-sixty · 2017-03-20T15:16:48Z

Yes, thanks for your patience. I'll wrap most of these up over the next month

jreback · 2017-05-06T23:20:29Z

@MaximilianR can you rebase / update

max-sixty · 2017-05-11T20:45:13Z

Forgive the extreme delay for this + a couple others. I need a few more weeks on other stuff, and then I'll give a push to finishing these up.

jreback · 2017-07-26T23:56:52Z

pls rebase / update.

jreback · 2017-08-18T00:58:04Z

love the idea, but needs rebase and updating.

jreback added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Reshaping Concat, Merge/Join, Stack/Unstack, Explode Numeric Operations Arithmetic, Comparison, and Logical operations labels Mar 30, 2016

max-sixty force-pushed the rolling-min-weight branch 2 times, most recently from 579af97 to 3146b2f Compare April 1, 2016 00:25

jreback reviewed Apr 1, 2016
View reviewed changes

jreback reviewed Apr 4, 2016
View reviewed changes

max-sixty force-pushed the rolling-min-weight branch from 3146b2f to 5f9801c Compare April 6, 2016 23:11

max-sixty force-pushed the rolling-min-weight branch from 5f9801c to 0ca3efc Compare May 17, 2016 04:32

jreback reviewed May 20, 2016
View reviewed changes

max-sixty force-pushed the rolling-min-weight branch 4 times, most recently from 1d2a591 to 4c40a10 Compare November 28, 2016 16:05

initial implementation of min_weight

3c42153

max-sixty force-pushed the rolling-min-weight branch from 4c40a10 to 3c42153 Compare December 21, 2016 03:33

jreback closed this Aug 18, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: Min_weight for Rolling #12750

ENH: Min_weight for Rolling #12750

max-sixty commented Mar 30, 2016

jreback commented Mar 30, 2016

max-sixty commented Mar 30, 2016

jreback commented Mar 30, 2016

max-sixty commented Mar 30, 2016

jreback Apr 1, 2016

jreback commented Apr 1, 2016

max-sixty commented Apr 1, 2016

jreback Apr 4, 2016

jreback commented May 7, 2016

codecov-io commented May 17, 2016 •

edited

Loading

jreback May 20, 2016

max-sixty commented Jul 14, 2016

jreback commented Sep 9, 2016

jreback commented Nov 16, 2016

max-sixty commented Nov 16, 2016

jreback commented Dec 21, 2016

jreback commented Jan 21, 2017

jreback commented Mar 20, 2017

max-sixty commented Mar 20, 2017

jreback commented May 6, 2017

max-sixty commented May 11, 2017

jreback commented Jul 26, 2017

jreback commented Aug 18, 2017

ENH: Min_weight for Rolling #12750

ENH: Min_weight for Rolling #12750

Conversation

max-sixty commented Mar 30, 2016

jreback commented Mar 30, 2016

max-sixty commented Mar 30, 2016

jreback commented Mar 30, 2016

max-sixty commented Mar 30, 2016

jreback Apr 1, 2016

Choose a reason for hiding this comment

jreback commented Apr 1, 2016

max-sixty commented Apr 1, 2016

jreback Apr 4, 2016

Choose a reason for hiding this comment

jreback commented May 7, 2016

codecov-io commented May 17, 2016 • edited Loading

Current coverage is 84.65% (diff: 100%)

jreback May 20, 2016

Choose a reason for hiding this comment

max-sixty commented Jul 14, 2016

jreback commented Sep 9, 2016

jreback commented Nov 16, 2016

max-sixty commented Nov 16, 2016

jreback commented Dec 21, 2016

jreback commented Jan 21, 2017

jreback commented Mar 20, 2017

max-sixty commented Mar 20, 2017

jreback commented May 6, 2017

max-sixty commented May 11, 2017

jreback commented Jul 26, 2017

jreback commented Aug 18, 2017

codecov-io commented May 17, 2016 •

edited

Loading