ENH: add quantile method to resample #22304

discort · 2018-08-13T07:28:02Z

closes ENH: add quantile method to resample #15023
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

codecov · 2018-08-13T08:19:13Z

Codecov Report

❗ No coverage uploaded for pull request base (master@9346e79). Click here to learn what that means.
The diff coverage is 100%.

@@            Coverage Diff            @@
##             master   #22304   +/-   ##
=========================================
  Coverage          ?   92.05%           
=========================================
  Files             ?      169           
  Lines             ?    50716           
  Branches          ?        0           
=========================================
  Hits              ?    46685           
  Misses            ?     4031           
  Partials          ?        0

Flag	Coverage Δ
#multiple	`90.46% <100%> (?)`
#single	`42.25% <11.11%> (?)`

Impacted Files	Coverage Δ
pandas/core/resample.py	`96.09% <100%> (ø)`
pandas/core/groupby/groupby.py	`96.18% <100%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9346e79...92b5b2d. Read the comment docs.

jreback · 2018-08-13T11:37:37Z

doc/source/whatsnew/v0.24.0.txt

@@ -650,6 +650,7 @@ Groupby/Resample/Rolling
  ``SeriesGroupBy`` when the grouping variable only contains NaNs and numpy version < 1.13 (:issue:`21956`).
 - Multiple bugs in :func:`pandas.core.Rolling.min` with ``closed='left'` and a
  datetime-like index leading to incorrect results and also segfault. (:issue:`21704`)
+- Added :meth:`Resampler.quantile` (:issue:`15023`).


move to other enhancements

this needs to be added in api.rst as well

jreback · 2018-08-13T11:37:48Z

pandas/core/resample.py

+    def quantile(self, q=0.5, **kwargs):
+        """
+        Return value at the given quantile.
+


add a versionadded tag

jreback · 2018-08-13T11:39:26Z

pandas/tests/test_resample.py

@@ -41,7 +41,7 @@

 # The various methods we support
 downsample_methods = ['min', 'max', 'first', 'last', 'sum', 'mean', 'sem',
-                      'median', 'prod', 'var', 'ohlc']


can you add the test from the OP which compares 2 methods of getting the same result

pep8speaks · 2018-08-13T11:53:34Z

Hello @discort! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on August 22, 2018 at 10:44 Hours UTC

discort · 2018-08-13T13:25:08Z

@jreback

jreback · 2018-08-14T00:11:23Z

pandas/core/resample.py

+        Parameters
+        ----------
+        q : float or array-like, default 0.5 (50% quantile)
+


can you add some See Also and point to the Series / groupby versions

jreback · 2018-08-14T00:11:59Z

pandas/tests/test_resample.py

@@ -2165,6 +2165,13 @@ def test_resample_datetime_values(self):
        res = df['timestamp'].resample('2D').first()
        tm.assert_series_equal(res, exp)

+    def test_resample_quantile(self):


can you test on timedelta / period as well. you might want to move to the Base class to make this easier.

jreback · 2018-08-14T00:13:14Z

doc/source/whatsnew/v0.24.0.txt

@@ -182,6 +182,7 @@ Other Enhancements
 - :func:`to_timedelta` now supports iso-formated timedelta strings (:issue:`21877`)
 - :class:`Series` and :class:`DataFrame` now support :class:`Iterable` in constructor (:issue:`2193`)
 - :class:`DatetimeIndex` gained :attr:`DatetimeIndex.timetz` attribute. Returns local time with timezone information. (:issue:`21358`)
+- Added :meth:`Resampler.quantile` (:issue:`15023`).


can you say that Series.resample and DataFrame.resample have gained the Resample.quantile() method (use references for .resample)

jreback · 2018-08-14T10:15:42Z

pandas/core/resample.py

@@ -1053,9 +1059,15 @@ def _downsample(self, how, **kwargs):
        how = self._is_cython_func(how) or how
        ax = self.ax

+        # Empty groupby objs are accepted only callable funcs


what triggers this?

@jreback

under periodindex scenario, test_resample_empty_dataframe is failed on this line

hmm this is odd. can you remove this fix, and instead: create a new issue which demonstrates this (is this only with this method?) and xfail the test. I think something bigger is going on here. We can merge this then come back with a battery of tests for this.

@jreback

to more precisely, this block doesn't pass the check and raises exception.
It became to fail after adding kwargs to _groupby_and_aggregate

hmm, this is old logic. can you try calling

def _try_aggregate_string_function(self, arg, *args, **kwargs) (its defined in the base class) and see if this works?

No, it doesn't

.../pandas/pandas/core/groupby/generic.py in _cython_agg_general(self, how, alt, numeric_only, min_count) 74 min_count=-1): 75 new_items, new_blocks = self._cython_agg_blocks( ---> 76 how, alt=alt, numeric_only=numeric_only, min_count=min_count) 77 return self._wrap_agged_blocks(new_items, new_blocks) 78 .../pandas/pandas/core/groupby/generic.py in _cython_agg_blocks(self, how, alt, numeric_only, min_count) 147 148 if len(new_blocks) == 0: --> 149 raise DataError('No numeric types to aggregate') 150 151 # reset the locs in the blocks to correspond to our DataError: No numeric types to aggregate

hmm, ok, then let's just xfail this test and pls open another issue. I think this is a bigger issues and needs to be addressed separately.

@jreback

Here is the issue - #22339

jreback · 2018-08-14T10:24:55Z

pandas/core/resample.py

@@ -1053,9 +1059,15 @@ def _downsample(self, how, **kwargs):
        how = self._is_cython_func(how) or how
        ax = self.ax

+        # Empty groupby objs are accepted only callable funcs


hmm this is odd. can you remove this fix, and instead: create a new issue which demonstrates this (is this only with this method?) and xfail the test. I think something bigger is going on here. We can merge this then come back with a battery of tests for this.

jreback · 2018-08-16T10:20:33Z

pandas/core/resample.py

@@ -1041,9 +1059,15 @@ def _downsample(self, how, **kwargs):
        how = self._is_cython_func(how) or how
        ax = self.ax

+        # Empty groupby objs are accepted only callable funcs
+        # with arg or kwargs currently


ok so remove this fix here

jreback · 2018-08-16T10:21:11Z

pandas/tests/test_resample.py

@@ -771,6 +771,15 @@ def test_apply_to_empty_series(self):

            assert_series_equal(result, expected, check_dtype=False)



then xfail this test for Period, in your followup you can unxfail it.

OR you can do everything here is ok too. (close the new PR / issue). I guess this was simpler that it looked.

The second way looks simpler. So I closed new PR/issue

discort · 2018-08-22T03:22:45Z

@jreback

jreback

minor comment on the doc-string, ping on green.

jreback · 2018-08-22T10:34:06Z

pandas/core/resample.py

+        DataFrame.quantile
+        DataFrameGroupBy.quantile
+
+        .. versionadded:: 0.24.0


the tag goes right below the summary line (the first line of the doc-string, with a blank line before and after)

jreback · 2018-08-22T12:27:10Z

thanks @discort keep em coming!

jreback requested changes Aug 13, 2018

View reviewed changes

jreback added Enhancement Resample resample method labels Aug 13, 2018

discort force-pushed the resampler_quantile branch from c0e8601 to b64bd34 Compare August 13, 2018 11:53

discort force-pushed the resampler_quantile branch from b64bd34 to 07ed27f Compare August 13, 2018 11:55

jreback requested changes Aug 14, 2018

View reviewed changes

This was referenced Aug 14, 2018

TestPeriodIndex.test_resample_empty_dataframe is failed when I pass kwargs to Resampler._groupby_and_aggregate #22339

Closed

TestPeriodIndex.test_resample_empty_dataframe is failed when I pass kwargs to Resampler._groupby_and_aggregate #22364

Closed

jreback requested changes Aug 16, 2018

View reviewed changes

discort force-pushed the resampler_quantile branch 2 times, most recently from 3f831f2 to 6f4f96f Compare August 17, 2018 08:05

jreback added this to the 0.24.0 milestone Aug 22, 2018

jreback requested changes Aug 22, 2018

View reviewed changes

discort added 2 commits August 22, 2018 13:38

Added quantile method to Resampler

f4605a4

addressed Jeff's comments

92b5b2d

discort force-pushed the resampler_quantile branch from 6f4f96f to 92b5b2d Compare August 22, 2018 10:44

jreback approved these changes Aug 22, 2018

View reviewed changes

jreback merged commit d83f533 into pandas-dev:master Aug 22, 2018

Sup3rGeo pushed a commit to Sup3rGeo/pandas that referenced this pull request Oct 1, 2018

ENH: add quantile method to resample (pandas-dev#22304)

f4d8052

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add quantile method to resample #22304

ENH: add quantile method to resample #22304

discort commented Aug 13, 2018

codecov bot commented Aug 13, 2018 •

edited

Loading

jreback Aug 13, 2018

jreback Aug 13, 2018

jreback Aug 13, 2018

jreback Aug 13, 2018

pep8speaks commented Aug 13, 2018 •

edited

Loading

discort commented Aug 13, 2018

jreback Aug 14, 2018

jreback Aug 14, 2018

jreback Aug 14, 2018

jreback Aug 14, 2018

discort Aug 14, 2018

jreback Aug 14, 2018

discort Aug 14, 2018

jreback Aug 14, 2018

discort Aug 14, 2018

jreback Aug 14, 2018

discort Aug 14, 2018

jreback Aug 14, 2018

jreback Aug 16, 2018

jreback Aug 16, 2018

discort Aug 17, 2018

discort commented Aug 22, 2018

jreback left a comment

jreback Aug 22, 2018

jreback commented Aug 22, 2018

		@@ -771,6 +771,15 @@ def test_apply_to_empty_series(self):

		assert_series_equal(result, expected, check_dtype=False)

ENH: add quantile method to resample #22304

ENH: add quantile method to resample #22304

Conversation

discort commented Aug 13, 2018

codecov bot commented Aug 13, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pep8speaks commented Aug 13, 2018 • edited Loading

Comment last updated on August 22, 2018 at 10:44 Hours UTC

discort commented Aug 13, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

discort commented Aug 22, 2018

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Aug 22, 2018

codecov bot commented Aug 13, 2018 •

edited

Loading

pep8speaks commented Aug 13, 2018 •

edited

Loading