ENH: add end and end_day origin for resample #38408

GYHHAHA · 2020-12-11T06:22:06Z

closes ENH: Add 'end' option in resample's origin #37804
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

GYHHAHA · 2020-12-11T06:25:03Z

I have revert the backward arg and only add the end and end_day.
Could u take a look at this? Thanks! @jreback

pandas/core/resample.py

arw2019 · 2020-12-11T06:34:11Z

doc/source/whatsnew/v1.2.0.rst

@@ -292,6 +292,7 @@ Other enhancements
 - Improved error reporting for subsetting columns of a :class:`.DataFrameGroupBy` with ``axis=1`` (:issue:`37725`)
 - Implement method ``cross`` for :meth:`DataFrame.merge` and :meth:`DataFrame.join` (:issue:`5401`)
 - When :func:`read_csv/sas/json` are called with ``chuncksize``/``iterator`` they can be used in a ``with`` statement as they return context-managers (:issue:`38225`)
+- Added ``end`` and ``end_day`` options for ``origin`` in :meth:`DataFrame.resample` (:issue:`37804`)


Not sure this will make it in for 1.2 - might need to move to 1.3

@jreback keep this in 1.2 or remove to 1.3 whatsnew?

Agreed that this should target 1.3

arw2019 · 2020-12-11T06:35:33Z

pandas/tests/resample/test_resample_api.py

@@ -611,3 +611,76 @@ def test_resample_agg_readonly():

    result = rs.agg("min")
    tm.assert_series_equal(result, expected)
+


is it possible to parametrize these tests?

arw2019 · 2020-12-11T06:35:47Z

pandas/tests/resample/test_resample_api.py

+    ts = Series(np.arange(len(rng)) * 3, index=rng)
+
+    res = ts.resample("17min", origin="end").sum().astype("int64")
+    data = [0, 18, 27, 63]


i'd stick that in the constructor

mroeschke said constructing ts in each test was better.

mroeschke · 2020-12-11T19:53:48Z

pandas/core/resample.py

            self.origin = origin
        else:
            try:
                self.origin = Timestamp(origin)
            except Exception as e:
                raise ValueError(
-                    "'origin' should be equal to 'epoch', 'start', 'start_day' or "
+                    "'origin' should be equal to 'epoch', 'start', 'start_day', "
+                    "'end' or 'end_day' "


"'end', 'end_day', or "

pandas/core/generic.py

mroeschke · 2020-12-12T04:14:39Z

pandas/core/groupby/grouper.py

@@ -93,6 +94,11 @@ class Grouper:

        .. versionadded:: 1.1.0

+        - 'end': `origin` is the last value of the timeseries
+        - 'end_day': `origin` is the ceiling midnight of the last day


Same comment/question here.

mroeschke

Two comments otherwise LGTM

GYHHAHA · 2020-12-12T04:24:50Z

Do u think it's worthwhile to explain more about these two backward resample method in ENH of whatsnew (maybe two examples added)?

mroeschke · 2020-12-12T04:28:14Z

Probably good to add to timeseries.rst#use-origin-or-offset-to-adjust-the-start-of-the-bins explaining the new options

GYHHAHA · 2020-12-12T04:30:07Z

ok

pandas/core/generic.py

pandas/core/resample.py

GYHHAHA · 2020-12-14T07:41:55Z

failures unrelated

jreback

small comment, pls merge master and ping on green.

cc @mroeschke if any other comments

jreback · 2020-12-22T00:57:00Z

pandas/core/generic.py

@@ -8027,7 +8027,8 @@ def resample(
        level : str or int, optional
            For a MultiIndex, level (name or number) to use for
            resampling. `level` must be datetime-like.
-        origin : {'epoch', 'start', 'start_day'}, Timestamp or str, default 'start_day'
+        origin : {'epoch', 'start', 'start_day', 'end', 'end_day'}, Timestamp \
+        or str, default 'start_day'


i think you have to indent this 2nd line

Here has a line-continuation at the end of 1st line, will this break the display if I indent the 'or str, default "start_day"' with 4 spaces?

yeah saw that but we don't usually use continuation like that

it will render if you indent on 2nd line

GYHHAHA · 2020-12-22T05:00:00Z

Now I change it to the following format.

origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp
    or str, default 'start_day'
    The timestamp on which to adjust the grouping. The timezone of origin
    must match the timezone of the index.

Will this be wrongly rendered to this?

origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp
or str, default 'start_day' The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index.

Instead of:

origin : {{'epoch', 'start', 'start_day', 'end', 'end_day'}}, Timestamp or str, default 'start_day'
The timestamp on which to adjust the grouping. The timezone of origin must match the timezone of the index.

@jreback

jreback · 2020-12-22T14:02:52Z

thanks @GYHHAHA

arw2019 reviewed Dec 11, 2020

View reviewed changes

mroeschke reviewed Dec 11, 2020

View reviewed changes

GYHHAHA added 7 commits December 12, 2020 07:55

add end and end_day origin for resample

5e8f034

Delete test.ipynb

8dce4e3

fix

f316aea

fix

20e15e4

update

11e1252

Update v1.3.0.rst

edfe52a

update version

f47b6ea

mroeschke reviewed Dec 12, 2020

View reviewed changes

pandas/core/generic.py Show resolved Hide resolved

mroeschke reviewed Dec 12, 2020

View reviewed changes

GYHHAHA added 3 commits December 13, 2020 10:42

Update timeseries.rst

0cc22cd

Update timeseries.rst

e6ae1a5

Update timeseries.rst

a2e0a57

jreback requested changes Dec 13, 2020

View reviewed changes

pandas/core/generic.py Show resolved Hide resolved

pandas/core/resample.py Show resolved Hide resolved

jreback added Enhancement Resample resample method labels Dec 13, 2020

jreback added this to the 1.3 milestone Dec 13, 2020

GYHHAHA added 3 commits December 14, 2020 10:23

Merge branch 'master' into add-end

8fcc8f0

Update resample.py

2ffce3c

Update generic.py

a554cad

GYHHAHA requested a review from jreback December 14, 2020 07:41

jreback requested changes Dec 22, 2020

View reviewed changes

Merge branch 'master' into add-end

546d647

GYHHAHA added 2 commits December 22, 2020 10:04

Update generic.py

ab473ce

Update grouper.py

8331a8e

mroeschke approved these changes Dec 22, 2020

View reviewed changes

GYHHAHA requested a review from jreback December 22, 2020 05:06

jreback approved these changes Dec 22, 2020

View reviewed changes

jreback merged commit 15b5946 into pandas-dev:master Dec 22, 2020

GYHHAHA deleted the add-end branch December 22, 2020 14:07

luckyvs1 pushed a commit to luckyvs1/pandas that referenced this pull request Jan 20, 2021

ENH: add end and end_day origin for resample (pandas-dev#38408)

2a98c09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: add end and end_day origin for resample #38408

ENH: add end and end_day origin for resample #38408

GYHHAHA commented Dec 11, 2020 •

edited

Loading

GYHHAHA commented Dec 11, 2020

arw2019 Dec 11, 2020

GYHHAHA Dec 11, 2020

mroeschke Dec 11, 2020

GYHHAHA Dec 12, 2020

arw2019 Dec 11, 2020

GYHHAHA Dec 11, 2020

arw2019 Dec 11, 2020

GYHHAHA Dec 11, 2020

mroeschke Dec 11, 2020

mroeschke Dec 12, 2020

mroeschke left a comment

GYHHAHA commented Dec 12, 2020

mroeschke commented Dec 12, 2020 •

edited

Loading

GYHHAHA commented Dec 12, 2020

GYHHAHA commented Dec 14, 2020

jreback left a comment

jreback Dec 22, 2020

GYHHAHA Dec 22, 2020

jreback Dec 22, 2020

GYHHAHA Dec 22, 2020

GYHHAHA commented Dec 22, 2020 •

edited

Loading

jreback commented Dec 22, 2020

		@@ -611,3 +611,76 @@ def test_resample_agg_readonly():

		result = rs.agg("min")
		tm.assert_series_equal(result, expected)

ENH: add end and end_day origin for resample #38408

ENH: add end and end_day origin for resample #38408

Conversation

GYHHAHA commented Dec 11, 2020 • edited Loading

GYHHAHA commented Dec 11, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mroeschke left a comment

Choose a reason for hiding this comment

GYHHAHA commented Dec 12, 2020

mroeschke commented Dec 12, 2020 • edited Loading

GYHHAHA commented Dec 12, 2020

GYHHAHA commented Dec 14, 2020

jreback left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GYHHAHA commented Dec 22, 2020 • edited Loading

jreback commented Dec 22, 2020

GYHHAHA commented Dec 11, 2020 •

edited

Loading

mroeschke commented Dec 12, 2020 •

edited

Loading

GYHHAHA commented Dec 22, 2020 •

edited

Loading