BUG: lambda with positional references in apply after groupby on empty dataframe errors

[dataist]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> import pandas as pd
>>> pd.DataFrame([],columns=['a','b','c']).groupby('a').b.apply(sum)
Series([], Name: b, dtype: object)
>>> pd.Series([],dtype='float64').apply(sum)
Series([], dtype: float64)
>>> pd.Series([],dtype='float64').apply(lambda x: x[0]-x[1])
Series([], dtype: float64)
>>> pd.DataFrame([],columns=['a','b','c']).groupby('a').b.apply(lambda x: x[0]-x[1])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/Jesse/repo/frame/dev-virtualenv-frame/lib/python3.9/site-packages/pandas/core/groupby/generic.py", line 244, in apply
    return super().apply(func, *args, **kwargs)
  File "/Users/Jesse/repo/frame/dev-virtualenv-frame/lib/python3.9/site-packages/pandas/core/groupby/groupby.py", line 1414, in apply
    result = self._python_apply_general(f, self._selected_obj)
  File "/Users/Jesse/repo/frame/dev-virtualenv-frame/lib/python3.9/site-packages/pandas/core/groupby/groupby.py", line 1455, in _python_apply_general
    values, mutated = self.grouper.apply(f, data, self.axis)
  File "/Users/Jesse/repo/frame/dev-virtualenv-frame/lib/python3.9/site-packages/pandas/core/groupby/ops.py", line 776, in apply
    f(data.iloc[:0])
  File "<stdin>", line 1, in <lambda>
  File "/Users/Jesse/repo/frame/dev-virtualenv-frame/lib/python3.9/site-packages/pandas/core/series.py", line 955, in __getitem__
    return self._values[key]
IndexError: index 0 is out of bounds for axis 0 with size 0

Issue Description

apply on empty series after groupby attempts to evaluate and errors

Expected Behavior

returns empty series

Installed Versions

{
"date": "2022-02-12T09:04:11+0000",
"dirty": false,
"error": null,
"full-revisionid": "06d230151e6f18fdb8139d09abf539867a8cd481",
"version": "1.4.1"
}

[rhshadrach]

Thanks for the report!

[simonjayhawkins]

Expected Behavior

returns empty series

returns empty series with object dtype in pandas-1.3.5

first bad commit: [884d00f] TST: fix groupby-empty xfails (#44092)

#44092 has a code change despite the PR title and no release note about the change in behavior.

cc @jbrockmendel

NOTE: both pd.DataFrame([],columns=['a','b','c']).groupby('a').b.apply(sum) and pd.DataFrame([],columns=['a','b','c']).groupby('a').b.apply(lambda x: x[0]-x[1]) return an empty series with float64 dtype in pandas-1.2.5

[simonjayhawkins]

will perhaps revert #44092 later today to get this fixed for released pandas.

[simonjayhawkins]

will perhaps revert #44092 later today to get this fixed for released pandas.

moving to 1.4.4 to allow time to investigate alternatives properly.

[rhshadrach]

This is an instance of a common issue with apply/agg/transform with a UDF on an empty frame/series. If we don't ever call the function, we can't infer the shape of the output. If we do call the function, it can fail as it does here. As a band-aid, I'm thinking we should wrap the f(data.iloc[:0]) attempt in a try-except and fall back to returning an empty Series when anything other than a TypeError is raised.

However, I think we should align on a solution to how empty objects are handled across the apply/agg/transform implementations and document it for users. I plan to put up a tracking issue on this.

[simonjayhawkins]

removing from 1.4.x milestone.