Skip to content

ENH: .shift optionally takes multiple periods #54115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 32 commits into from
Jul 28, 2023
Merged

ENH: .shift optionally takes multiple periods #54115

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
4e1a84e
init
jona-sassenhagen Jul 13, 2023
db9cd03
precommit
jona-sassenhagen Jul 13, 2023
9def045
slightly update test
jona-sassenhagen Jul 13, 2023
b7ea297
Fix groupby API tests
jona-sassenhagen Jul 13, 2023
299927a
mostly types, also exclude groupby
jona-sassenhagen Jul 14, 2023
6cf632e
fix test
jona-sassenhagen Jul 14, 2023
23d9142
mypy
jona-sassenhagen Jul 14, 2023
b78d7d6
fix docstring
jona-sassenhagen Jul 14, 2023
cabd28c
change how futurewarning is handled in the test
jona-sassenhagen Jul 15, 2023
5017721
fix docstring
jona-sassenhagen Jul 15, 2023
6f3ec9b
remove debug statement
jona-sassenhagen Jul 15, 2023
8d085f3
address comments
jona-sassenhagen Jul 16, 2023
6e66a19
refactor
jona-sassenhagen Jul 16, 2023
f7ea7a3
handle default
jona-sassenhagen Jul 17, 2023
f8e29d9
pylint
jona-sassenhagen Jul 17, 2023
e54ba33
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 17, 2023
715c397
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 18, 2023
21e8e70
merge conflicts and mypy
jona-sassenhagen Jul 18, 2023
0e78963
split tests, remove checking for None default
jona-sassenhagen Jul 18, 2023
2c602e2
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 18, 2023
d9bf54f
address comments
jona-sassenhagen Jul 19, 2023
778b7e9
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 19, 2023
ad49861
Merge branch 'jona/44424_shift' of github.com:jona-sassenhagen/pandas…
jona-sassenhagen Jul 19, 2023
28469db
mypy
jona-sassenhagen Jul 19, 2023
c8e5bad
mypy again
jona-sassenhagen Jul 19, 2023
cb4013d
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 19, 2023
226fa6f
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 21, 2023
cb49cac
black
jona-sassenhagen Jul 21, 2023
44b0866
address comments
jona-sassenhagen Jul 25, 2023
5257292
Merge branch 'main' into jona/44424_shift
jona-sassenhagen Jul 25, 2023
eff4ed2
mypy
jona-sassenhagen Jul 25, 2023
04d4513
Merge branch 'jona/44424_shift' of github.com:jona-sassenhagen/pandas…
jona-sassenhagen Jul 25, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pandas/core/generic.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10550,7 +10550,7 @@ def shift(
axis: Axis = 0,
fill_value: Hashable = lib.no_default,
suffix: str | None = None,
) -> Self:
) -> Self | DataFrame:
"""
Shift index by desired number of periods with an optional time `freq`.

Expand Down
2 changes: 1 addition & 1 deletion pandas/core/series.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3002,7 +3002,7 @@ def autocorr(self, lag: int = 1) -> float:
>>> s.autocorr()
nan
"""
return self.corr(self.shift(lag))
return self.corr(cast(Series, self.shift(lag)))
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is necessary because Series.shift can now return a dataframe if there are multiple periods. Mypy complains at this line because it wants Series and not Series | Dataframe, but because there is only one lag, it's always Series, so we can just cast.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could add type-overloads to handle this. Sometimes they can get gnarly (see e.g. concat), but I think this one should be relatively straight forward.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer not to if that's ok? There is no Series.shift to overload so I feel like it would be confusing to newcomers like me.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you insist I will try!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise I think this is done?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @rhshadrach , do you have time for another round/sign this off?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with opening an issue regarding this once it's merged for someone to followup on.


def dot(self, other: AnyArrayLike) -> Series | np.ndarray:
"""
Expand Down
4 changes: 0 additions & 4 deletions pandas/tests/groupby/test_groupby_shift_diff.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,6 @@ def test_shift_disallow_suffix_if_periods_is_int():
df.groupby("b").shift(1, suffix="fails")


@pytest.mark.filterwarnings("ignore:The 'axis' keyword in")
def test_group_shift_with_multiple_periods():
# GH#44424
df = DataFrame({"a": [1, 2, 3, 3, 2], "b": [True, True, False, False, True]})
Expand All @@ -200,7 +199,6 @@ def test_group_shift_with_multiple_periods():
tm.assert_frame_equal(shifted_series, expected_df)


@pytest.mark.filterwarnings("ignore:The 'axis' keyword in")
def test_group_shift_with_multiple_periods_and_freq():
# GH#44424
df = DataFrame(
Expand Down Expand Up @@ -228,7 +226,6 @@ def test_group_shift_with_multiple_periods_and_freq():
tm.assert_frame_equal(shifted_df, expected_df)


@pytest.mark.filterwarnings("ignore:The 'axis' keyword in")
def test_group_shift_with_multiple_periods_and_fill_value():
# GH#44424
df = DataFrame(
Expand All @@ -241,7 +238,6 @@ def test_group_shift_with_multiple_periods_and_fill_value():
tm.assert_frame_equal(shifted_df, expected_df)


@pytest.mark.filterwarnings("ignore:The 'axis' keyword in")
def test_group_shift_with_multiple_periods_and_both_fill_and_freq_fails():
# GH#44424
df = DataFrame(
Expand Down