result from groupby / nlargest with data frame with one row does not include the groupby key in the resulting index

[joshuastorck]

Code Sample, a copy-pastable example if possible

In [1]: df = pandas.DataFrame([["Dog", 1], ["Dog", 2]], columns=["animal", "value"])

In [2]: df.groupby("animal").value.nlargest(5)
Out[2]: 
animal   
Dog     1    2
        0    1
Name: value, dtype: int64
In [3]: df = pandas.DataFrame([["Dog", 1]], columns=["animal", "value"])
In [4]: df.groupby("animal").value.nlargest(5)
Out[4]: 
0    1
Name: value, dtype: int64

Problem description

Expected Output

In [3]: df = pandas.DataFrame([["Dog", 1]], columns=["animal", "value"])
In [4]: df.groupby("animal").value.nlargest(5)
Out[4]: 
animal
Dog      0    1
Name: value, dtype: int64

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.25.2
numpy: 1.12.1
scipy: None
statsmodels: None
xarray: None
IPython: 6.0.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.5.3
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.9.6
boto: None
pandas_datareader: None

[jreback]

this is a filtering method, like head, so this is expected.

In [5]: df.groupby('animal').head(2)
Out[5]: 
  animal  value
0    Dog      1
1    Dog      2

[jreback]

though it should have the group-and column though.

[jreback]

actually, this appears to work.

In [1]: df = pandas.DataFrame([["Dog", 1], ["Dog", 2]], columns=["animal", "value"])
   ...: 

In [2]: df.groupby('animal').value.nlargest(2)
   ...: 
Out[2]: 
animal   
Dog     1    2
        0    1
Name: value, dtype: int64

[joshuastorck]

The problem occurs when the data frame contains only one row (see example above). In that case, the "animal" column disappears.

[RhysU]

A slight generalization on the recreate also exhibits the "group dropping" behavior despite having multiple rows in the output. Worth confirming any fix covers this situation too.

Observed:

In [1]: import pandas

In [2]: df = pandas.DataFrame([["A", 1.0], ["B", 0.99], ["B", 0.1]], columns=['group', 'value'])

In [3]: df
Out[3]: 
  group  value
0     A   1.00
1     B   0.99
2     B   0.10

In [4]: df.groupby("group").value.nlargest(5)
Out[4]: 
0    1.00
1    0.99
2    0.10
Name: value, dtype: float64

In [5]: pandas.__version__
Out[5]: '0.19.2.post+ts6'

Expected: group column is retained in Out[4].

[phofl]

Hi, @jreback @mroeschke

I looked into this a bit. I found the reason, why the column is dropped sometimes and sometimes not. For functions like nlarges and apply (not head) the column is always dropped, if the input DataFrame equals the output DataFrame (sorting too!). It is checked, if the Index was changed. Depending on the result, different code is executed. While I tried to fix this, I ran into some issues with existing unittests.

If we do something like:

``
df = pd.DataFrame({"key": ["b"] * 10, "value": 2})

actual = df.groupby("key")["value"].cumprod()

``

Should the resulting Series have the column b in the Index or is

0    2
1    4
2    8
3    16
4    32
5    64
6    128
7    256
8    512
9    1024

Name: value, dtype: int64

the desired output?

Similar question: If we execute

``
base_df = pd.DataFrame({"A": [1, 1, 1, 1, 2, 2, 2, 2], "B": [np.nan] * 8})

expected = pd.DataFrame({"B": [np.nan] * 8})
result = base_df.groupby("A").cummax()

``

should the output Series look like

B
0 NaN
1 NaN
2 NaN
3 NaN
4 NaN
5 NaN
6 NaN
7 NaN

or should the column A be part of the Index?

I would really appreciate an answert about the output format of these two functions. Depending on that I may have found a way to fix this issue and the issues related with this (#29129 for example). If the columns should not be part of the Index, the solution is more complex.

Thanks very much.

[rhshadrach]

@phofl - transformers should return the index of the original DataFrame.