Can't use a lambda function in named aggregation

[rafaelcf0]

Code Sample, a copy-pastable example if possible

animals = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],
                            'height': [9.1, 6.0, 9.5, 34.0],
                            'weight': [7.9, 7.5, 9.9, 198.0]})

# Yields an error
animals.groupby('kind').agg(mean_height=('height', 'mean'),
                            perc90=('height', lambda s: np.percentile(s, q=0.90)))

def myfunc(s): return np.percentile(s, q=0.90)

# Works just fine
animals.groupby('kind').agg(mean_height=('height', 'mean'),
                            perc90=('height', myfunc))

Problem description

When I groupby+agg with a named function, the named aggregation works perfectly. However, when done with a lambda function instead, the following error is raised:

KeyError: "[('height', '<lambda>')] not in index"

Notice that his is not error 7186 because there are no more than one lambda here.

I've noticed that this error only happens when I mix lambda functions along with str-mapped functions, i.e. just

animals.groupby('kind').agg(perc90=('height', lambda s: np.percentile(s, q=0.90)))

works just fine.

Expected Output

      mean_height  perc95
kind
cat           9.3  9.1038
dog          20.0  6.2660

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-1036-gcp
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.17.2
pytz : 2019.2
dateutil : 2.8.0
pip : 19.0.3
setuptools : 40.8.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : None
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
[paste the output of pd.show_versions() here below this line]

[randomstuff]

Works fine for me (python 3.7.4 and pandas 0.25.1).

[rafaelcf0]

An easy way to try the error out is through this shared repl.it console.

[TomAugspurger]

Reproduced on 0.25.1, but not on master FWIW.

…

On Mon, Sep 16, 2019 at 2:37 PM Rafael Ferreira ***@***.***> wrote: An easy way to try the error out is through this shared repl.it console <https://repl.it/repls/OlivedrabLikelyDemo>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#28467?email_source=notifications&email_token=AAKAOIWUGM6GPWWAPTZKUMLQJ7N6RA5CNFSM4IXFVIC2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD62IXCY#issuecomment-531925899>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAKAOIQZYH656IIOKC2FM7DQJ7N6RANCNFSM4IXFVICQ> .

[robertmuil]

This occurs for me in 0.25.2

[robertmuil]

It occurs when you use more than one unnamed function on the same column: so it is the tuple of (<column>, lambda) that cannot be duplicated.

A workaround is using named functions (which is a pain).

So, this fails with KeyError: "[('height', '<lambda>')] not in index"

animals.groupby("kind").agg(
    min_height=pd.NamedAgg(column='height', aggfunc=lambda x: x.min()),
    max_height=pd.NamedAgg(column='height', aggfunc=lambda x: x.max()),
    average_weight=pd.NamedAgg(column='weight', aggfunc='mean')
)

While this works:

def maxfunc(x):
    return x.max()
def minfunc(x):
    return x.min()

animals.groupby("kind").agg(
    min_height=pd.NamedAgg(column='height', aggfunc=minfunc),
    max_height=pd.NamedAgg(column='height', aggfunc=maxfunc),
    average_weight=pd.NamedAgg(column='weight', aggfunc='mean')
)

[mroeschke]

Both work fine on master for me. Could use a regression test.

In [1]: animals = pd.DataFrame({'kind': ['cat', 'dog', 'cat', 'dog'],
   ...:                             'height': [9.1, 6.0, 9.5, 34.0],
   ...:                             'weight': [7.9, 7.5, 9.9, 198.0]})
   ...:
   ...: # Yields an error
   ...: animals.groupby('kind').agg(mean_height=('height', 'mean'),
   ...:                             perc90=('height', lambda s: np.percentile(s, q=0.90)))
Out[1]:
      mean_height  perc90
kind
cat           9.3  9.1036
dog          20.0  6.2520

In [2]: pd.__version__
Out[2]: '0.26.0.dev0+684.g953757a3e'

In [4]: def myfunc(s): return np.percentile(s, q=0.90)
   ...:
   ...: # Works just fine
   ...: animals.groupby('kind').agg(mean_height=('height', 'mean'),
   ...:                             perc90=('height', myfunc))
Out[4]:
      mean_height  perc90
kind
cat           9.3  9.1036
dog          20.0  6.2520

[robertmuil]

@mroeschke exactly your code yields the following error for me (from the first agg):
KeyError: "[('height', '<lambda>')] not in index"

$ pd.__version__
'0.25.2'

[robertmuil]

Not sure that this issue should be closed: the referenced merged PR only contains testing functions: excellent that this is now covered, but the failure will remain...

[jreback]

@robertmuil the reason the PR only contains testing functions is the issue was previously fixed
if you have a reproducible example on master open a new issue

[robertmuil]

understood