ENH: new .agg for list-likes #43736

rhshadrach · 2021-09-24T21:25:23Z

tests added / passed
Ensure all linting tests pass, see here for how to run them

Replaces the list-like implementation for .agg used by Series, DataFrame, SeriesGroupBy, DataFrameGroupBy, Window, and Resample. This will transpose agg results, and swap multiindex levels as well as the order from within the levels. Adds the new implementation behind the option "new_udf_methods".

Docs: future_udf_behavior.zip

attack68 · 2021-09-25T05:54:33Z

just a comment - I have always tried to avoid names like "new_routine" because at some point, it won't be, and you wished you never called it that in the first place! :)

rhshadrach · 2021-09-25T15:51:58Z

@attack68 - The plan is to put all usage of these methods behind the option "new_udf_methods"; when the old are deprecated and then removed, they will be renamed by dropping all the "new_" prefixes. If you have any suggestions for different names, please do make them!

attack68 · 2021-09-25T16:55:01Z

no, no suggestions. But, and I kid you not, in my line of work I have seen swap_pricer(), new_swap_pricer(), new_swap_pricer2() and new_new_swap_pricer() - imagine explaining that to new recruits!

mroeschke · 2021-09-25T17:22:30Z

Agreed, with @attack68. Helpful to have a slightly clearer naming even during the transition period. Maybe shared_list_like if the plan is for method to be agg, transform, etc?

rhshadrach · 2021-09-25T17:25:41Z

@mroeschke and @attack68 - is there opposition to the "new_udf_methods" option as well, or is it just on the method names themselves? I am not a particular fan of "new" myself, but I could not think of anything concise and better. Regardless of the name that is picked, I would like the relationship between the option name and the methods that are behind that option name to be clear.

Edit: could go with "experimental" and prefix the methods with that?

Edit: "experimental" is a bad choice because there will be a period where the default for this option is True and thus it is no longer experimental.

mroeschke · 2021-09-25T18:54:05Z

I would like the relationship between the option name and the methods that are behind that option name to be clear

+1 I would hope these are consistent.

Maybe future_udf_behavior if the idea is that this will be the default some day?

rhshadrach · 2021-09-25T19:09:37Z

Thanks @mroeschke - I like "future".

…w_udfs_list_agg � Conflicts: � pandas/tests/resample/test_resample_api.py

jbrockmendel · 2021-09-29T02:02:46Z

pandas/core/apply.py

@@ -408,6 +414,77 @@ def agg_list_like(self) -> DataFrame | Series:
            )
            return concatenated.reindex(full_ordered_index, copy=False)

+    def future_list_single_arg(


same request from the other PR re naming; "future" won't be very helpful for a reader a year from now

The intention is to have "future_" methods alongside the current methods; all with the same prefix so they are easy to identify. Any such method is behind the option "future_udf_behavior" meaning they will only be called when set to True. Assuming we do end up going forward with this new (experimental) behavior, once it is in a good place we deprecate the option and then remove the option.

A year from now, we will still have the option "future_udf_behavior", and in my opinion, the "future_" prefix is meaningful and helpful - namely in its connection to this option. It is also the (intended) future behavior of the methods. When the option is removed, the "old" methods are removed and the "future_" methods are renamed by removing the prefix (none are public).

you've clearly given this more thought than i have so im going to stop complaining about this, will instead grumble to myself

The name is still up for improvements and suggestions are most welcome, but wanted to explain why I felt "future" was appropriate.

…w_udfs_list_agg � Conflicts: � pandas/core/groupby/generic.py � pandas/core/groupby/groupby.py � pandas/tests/apply/test_frame_apply.py � pandas/tests/groupby/test_groupby.py

…ev#43873)

* Fix dtypes for read_json * Address comments * Add whatsnew entry * Update doc/source/whatsnew/v1.4.0.rst Co-authored-by: Matthew Zeitlin <[email protected]> * Linting Co-authored-by: Matthew Zeitlin <[email protected]>

…andas-dev#43674)

…as-dev#43787) (pandas-dev#43787)

…ev#43882)

…subclass itself (pandas-dev#43897)

…th `None` (pandas-dev#43893)

pandas-dev#40653)

…ev#43928)

…w_udfs_list_agg

rhshadrach · 2021-10-18T20:16:12Z

I've added docs (built attachment in OP) summarizing the changes here.

…w_udfs_list_agg � Conflicts: � pandas/core/groupby/generic.py � pandas/core/groupby/groupby.py � pandas/tests/apply/test_frame_apply.py � pandas/tests/groupby/test_groupby.py

into new_udfs_list_agg � Conflicts: � pandas/tests/apply/test_frame_apply.py � pandas/tests/groupby/aggregate/test_aggregate.py � pandas/tests/groupby/aggregate/test_other.py � pandas/tests/groupby/test_groupby.py � pandas/tests/resample/test_resample_api.py

github-actions · 2021-12-08T00:18:39Z

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

rhshadrach added 3 commits September 24, 2021 17:22

ENH: new .agg for list-likes

3dfb779

Refactor single arg computation, test fixup

9ef1eb0

Revert change to GroupBy.agg

1974e07

Merge branch 'master' of https://github.com/pandas-dev/pandas into ne…

af18184

…w_udfs_list_agg � Conflicts: � pandas/tests/resample/test_resample_api.py

rhshadrach mentioned this pull request Sep 25, 2021

POC: New UDF methods option #43678

Closed

3 tasks

Rename option and methods

d7b6c7f

jbrockmendel reviewed Sep 29, 2021

View reviewed changes

rhshadrach and others added 16 commits October 9, 2021 16:04

Merge branch 'master' of https://github.com/pandas-dev/pandas into ne…

af002fc

…w_udfs_list_agg � Conflicts: � pandas/core/groupby/generic.py � pandas/core/groupby/groupby.py � pandas/tests/apply/test_frame_apply.py � pandas/tests/groupby/test_groupby.py

Merge fixups

d412b4f

BUG/ERR: sparse array cmp methods mismatched len (pandas-dev#43863)

0cea15b

Add deprecation tag for passing a string for ewm(times=...) (pandas-d…

665b304

…ev#43873)

Make components of Suffixes Optional (pandas-dev#42544)

214ba4a

BUG: Fix dtypes for read_json (pandas-dev#42819)

9d6da6d

* Fix dtypes for read_json * Address comments * Add whatsnew entry * Update doc/source/whatsnew/v1.4.0.rst Co-authored-by: Matthew Zeitlin <[email protected]> * Linting Co-authored-by: Matthew Zeitlin <[email protected]>

TST: dropping of nuisance columns for groupby ops pandas-dev#38815 (p…

005598c

…andas-dev#43674)

BUG: retain EA dtypes in DataFrame __pos__, __neg__ (pandas-dev#43883)

7afb062

TST: Test Series' settitem with Interval and NaN (pandas-dev#43844)

195f9cf

PERF: tighter cython declarations, faster __iter__ (pandas-dev#43872)

6021c06

PERF: read_csv with memory_map=True when file encoding is UTF-8 (pand…

aa0a1d6

…as-dev#43787) (pandas-dev#43787)

TYP: enable reportMissingImports (pandas-dev#43790)

ef35a19

Don't suppress exception chaining for optional dependencies (pandas-d…

eefd0f0

…ev#43882)

BUG: DataFrame arithmetic with subclass where constructor is not the …

d3f5a44

…subclass itself (pandas-dev#43897)

REF: remove _get_attributes_dict (pandas-dev#43895)

1146215

Annotates indexers/utils.py functions that don't return anything wi…

58ff02d

…th `None` (pandas-dev#43893)

realead and others added 18 commits October 10, 2021 00:40

[ENH] introducing IntpHashMap and making unique_label_indices use intp (

2688ca8

pandas-dev#40653)

ENH: implement Index.__array_ufunc__ (pandas-dev#43904)

5fe8d7d

TST/REF: share/split index tests (pandas-dev#43905)

a49977c

TST/REF: misplaced Index.putmask tests (pandas-dev#43906)

4779171

Add clarifications to the docs regarding to_feather (pandas-dev#43866)

d4ae657

TST/REF: collect/de-dup index tests (pandas-dev#43914)

bde9b11

BENCH: indexing_engines (pandas-dev#43916)

ecab3a2

TST: avoid re-running tests 14 times (pandas-dev#43922)

adef17c

CLN: unnecessary warning-catching (pandas-dev#43919)

d5716c7

TST/REF: fixturize (pandas-dev#43918)

505ed3f

BUG: NumericIndex.insert (pandas-dev#43933)

9ee956b

TST: Skip leaky test on Python 3.10 (pandas-dev#43910)

1e370aa

ENH: EA.tolist (pandas-dev#43920)

acb7650

fixed rolling for a decreasing index, added a test for that (pandas-d…

e12643e

…ev#43928)

Merge branch 'master' of https://github.com/pandas-dev/pandas into ne…

8a454e0

…w_udfs_list_agg

Added docs

3bba371

Merge branch 'master' of https://github.com/pandas-dev/pandas into ne…

4a69adf

…w_udfs_list_agg

Make quotes consistent

7abdff9

jreback added the Enhancement label Oct 18, 2021

Fixup docs

a72a5eb

rhshadrach requested a review from jreback October 20, 2021 23:27

rhshadrach added 3 commits November 7, 2021 10:35

Merge branch 'master' of https://github.com/pandas-dev/pandas into ne…

f8aa318

…w_udfs_list_agg � Conflicts: � pandas/core/groupby/generic.py � pandas/core/groupby/groupby.py � pandas/tests/apply/test_frame_apply.py � pandas/tests/groupby/test_groupby.py

Merge cleanup

afc27ba

github-actions bot added the Stale label Dec 8, 2021

rhshadrach closed this Dec 11, 2021

rhshadrach mentioned this pull request Jan 22, 2022

ENH: Experimental Higher Order Methods API #45557

Closed

2 tasks

rhshadrach deleted the new_udfs_list_agg branch September 10, 2022 16:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: new .agg for list-likes #43736

ENH: new .agg for list-likes #43736

rhshadrach commented Sep 24, 2021 •

edited

Loading

attack68 commented Sep 25, 2021

rhshadrach commented Sep 25, 2021 •

edited

Loading

attack68 commented Sep 25, 2021

mroeschke commented Sep 25, 2021

rhshadrach commented Sep 25, 2021 •

edited

Loading

mroeschke commented Sep 25, 2021

rhshadrach commented Sep 25, 2021

jbrockmendel Sep 29, 2021

rhshadrach Sep 29, 2021 •

edited

Loading

jbrockmendel Sep 29, 2021

rhshadrach Sep 29, 2021

rhshadrach commented Oct 18, 2021 •

edited

Loading

github-actions bot commented Dec 8, 2021

ENH: new .agg for list-likes #43736

ENH: new .agg for list-likes #43736

Conversation

rhshadrach commented Sep 24, 2021 • edited Loading

attack68 commented Sep 25, 2021

rhshadrach commented Sep 25, 2021 • edited Loading

attack68 commented Sep 25, 2021

mroeschke commented Sep 25, 2021

rhshadrach commented Sep 25, 2021 • edited Loading

mroeschke commented Sep 25, 2021

rhshadrach commented Sep 25, 2021

jbrockmendel Sep 29, 2021

Choose a reason for hiding this comment

rhshadrach Sep 29, 2021 • edited Loading

Choose a reason for hiding this comment

jbrockmendel Sep 29, 2021

Choose a reason for hiding this comment

rhshadrach Sep 29, 2021

Choose a reason for hiding this comment

rhshadrach commented Oct 18, 2021 • edited Loading

github-actions bot commented Dec 8, 2021

rhshadrach commented Sep 24, 2021 •

edited

Loading

rhshadrach commented Sep 25, 2021 •

edited

Loading

rhshadrach commented Sep 25, 2021 •

edited

Loading

rhshadrach Sep 29, 2021 •

edited

Loading

rhshadrach commented Oct 18, 2021 •

edited

Loading