BUG: RollingGroupBy.quantile ignores interpolation keyword argument

[pwsiegel]

• closes RollingGroupBy.quantile ignores interpolation argument #28779
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[pwsiegel]

Hello all! My build failed and threw the following error during CI:

all_parsers = <pandas.tests.io.parser.conftest.PythonParser object at 0x7fc4298d4860>

    def test_chunks_have_consistent_numerical_type(all_parsers):
        parser = all_parsers
        integers = [str(i) for i in range(499999)]
        data = "a\n" + "\n".join(integers + ["1.0", "2.0"] + integers)
    
        # Coercions should work without warnings.
        with tm.assert_produces_warning(None):
>           result = parser.read_csv(StringIO(data))

pandas/tests/io/parser/test_common.py:1204: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <contextlib._GeneratorContextManager object at 0x7fc3ebc8a438>
type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsm'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsx'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsm'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsx'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='testmultiindex.xlsm'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='testmultiindex.xlsm'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsx'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927), ('ResourceWarning', ResourceWarning("unclosed file <_io.BufferedReader name='test1.xlsm'>",), '/home/vsts/work/1/s/pandas/io/parsers.py', 2927)].

../../../miniconda3/envs/pandas-dev/lib/python3.6/contextlib.py:89: AssertionError

My changes didn't touch the reader / parser part of the codebase, so I'm wondering if this problem crept into the build from elsewhere? Any suggestions for what I can do?

[WillAyd]

Looks pretty good. Can you try merging master and pushing again? I think should fix your CI issues

[WillAyd]

Either here or in a follow up could you parametrize this test with these values rather than the loop within?

[jreback]

can u make this a separate test and parmetrize

[pwsiegel]

Yes, I think I can handle that here - working on it now.

[pwsiegel]

I wrote a new test called test_rolling_quantile and parametrized the interpolation argument as requested, but unfortunately it looks as though most of the tests in this file have broken since I merged in master. To test, I checked out the upstream master branch, rebuilt my conda environment from scratch, and ran the tests there. 10 of the tests failed, including test_rolling (stack trace below). Is this a known issue?

self = <pandas.tests.window.test_grouper.TestGrouperGrouping object at 0x116432208>

    def test_rolling(self):
        g = self.frame.groupby("A")
        r = g.rolling(window=4)

        for f in ["sum", "mean", "min", "max", "count", "kurt", "skew"]:

>           result = getattr(r, f)()

test_grouper.py:64:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../core/window/rolling.py:1870: in sum
    return super().sum(*args, **kwargs)
../../core/window/rolling.py:1245: in sum
    return self._apply("roll_sum", "sum", **kwargs)
../../core/window/common.py:81: in _apply
    return self._groupby.apply(f)
../../core/groupby/groupby.py:718: in apply
    result = self._python_apply_general(f)
../../core/groupby/groupby.py:734: in _python_apply_general
    keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis)
../../core/groupby/ops.py:171: in apply
    result_values, mutated = splitter.fast_apply(f, group_keys)
../../core/groupby/ops.py:904: in fast_apply
    return libreduction.apply_frame_axis0(sdata, f, names, starts, ends)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

>   ???
E   AttributeError: 'Int64Index' object has no attribute '_has_complex_internals'

pandas/_libs/reduction.pyx:519: AttributeError

[pwsiegel]

Oops, @WillAyd's suggestion to rebuild the Cython extensions was correct. The tests are now passing; I'm going to push my changes shortly.

[pwsiegel]

The change is pushed - let me know if that's different from what you had in mind!

[jreback]

this is including lots of other changes, can you merge master

[pwsiegel]

Oops, I think I rebased by accident instead of merging, give me a moment to fix!

[pwsiegel]

@jreback OK, that looks better I think. Should I try to squash the merge commit in with 352445c?

[jreback]

no need to squash ; we do on merge

[jreback]

i don’t think this will render, can u see how we reference rolling in other notes

[pwsiegel]

Sure, I'll find another example in the notes and imitate it.

[pwsiegel]

I changed the reference to DataFrameGroupBy.rolling().quantile(), similarly to the reference to DataFrameGroupBy.rolling().apply() here: https://pandas.pydata.org/pandas-docs/version/0.23.4/whatsnew.html#groupby-resample-rolling. Hopefully that will work better?

[jreback]

can u make this a separate test and parmetrize

[WillAyd]

Try rebuilding the Cython extensions via python setup.py build_ext —inplace

…

Sent from my iPhone

On Nov 17, 2019, at 8:52 AM, Paul Siegel ***@***.***> wrote:  @pwsiegel commented on this pull request. In pandas/tests/window/test_grouper.py: > @@ -69,9 +68,10 @@ def test_rolling(self): expected = g.apply(lambda x: getattr(x.rolling(4), f)(ddof=1)) tm.assert_frame_equal(result, expected) - result = r.quantile(0.5) - expected = g.apply(lambda x: x.rolling(4).quantile(0.5)) - tm.assert_frame_equal(result, expected) + for f in ["linear", "lower", "higher", "midpoint", "nearest"]: I wrote a new test called test_rolling_quantile and parametrized the interpolation argument as requested, but unfortunately it looks as though most of the tests in this file have broken since I merged in master. To test, I checked out the upstream master branch, rebuilt my conda environment from scratch, and ran the tests there. 10 of the tests failed, including test_rolling (stack trace below). Is this a known issue? `self = <pandas.tests.window.test_grouper.TestGrouperGrouping object at 0x116432208> def test_rolling(self): g = self.frame.groupby("A") r = g.rolling(window=4) for f in ["sum", "mean", "min", "max", "count", "kurt", "skew"]: result = getattr(r, f)() test_grouper.py:64: ../../core/window/rolling.py:1870: in sum return super().sum(*args, **kwargs) ../../core/window/rolling.py:1245: in sum return self._apply("roll_sum", "sum", **kwargs) ../../core/window/common.py:81: in _apply return self._groupby.apply(f) ../../core/groupby/groupby.py:718: in apply result = self._python_apply_general(f) ../../core/groupby/groupby.py:734: in _python_apply_general keys, values, mutated = self.grouper.apply(f, self._selected_obj, self.axis) ../../core/groupby/ops.py:171: in apply result_values, mutated = splitter.fast_apply(f, group_keys) ../../core/groupby/ops.py:904: in fast_apply return libreduction.apply_frame_axis0(sdata, f, names, starts, ends) ??? E AttributeError: 'Int64Index' object has no attribute '_has_complex_internals' pandas/_libs/reduction.pyx:519: AttributeError ` — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or unsubscribe.

[jreback]

thanks @pwsiegel