groupby().first() skips NaN values

[jwkvam]

I do this

>>> df = pd.DataFrame([[1,np.nan,0],[1,1,1],[2,2,2],[2,3,3]], columns=list('abc'))
>>> print(df)
   a   b  c
0  1 NaN  0
1  1   1  1
2  2   2  2
3  2   3  3

[4 rows x 3 columns]
>>> print(df.groupby('a').first())
   b  c
a      
1  1  0
2  2  2

[2 rows x 2 columns]

but I expected this

>>> print(df.groupby('a').first())
     b  c
a      
1  NaN  0
2    2  2

[2 rows x 2 columns]

Is it possible to achieve my expected output? I get the same output in master and 0.13.1.

[hayd]

Yup, you can use nth (for the moment skipping NaN is a feature of first/last):

In [5]: df.groupby('a').nth(0)
Out[5]:
   a   b  c
0  1 NaN  0
2  2   2  2

@jreback Now, I recall why I haven't changed first/last yet, I need make these have a way to get this old (weird?) behaviour, similar to Series nth.

[jwkvam]

@hayd Thanks, if this is intended behavior then you can close this if you like. I'll leave it up to you.

[jreback]

@hayd I think first/last should pass dropna=False, so it is really the first/last (and presevers backward compat)

In [4]: df.groupby('a').nth(0,dropna=False)
Out[4]: 
   a   b  c
0  1 NaN  0
2  2   2  2

[2 rows x 3 columns]

In [5]: df.groupby('a').first()
Out[5]: 
   b  c
a      
1  1  0
2  2  2

[2 rows x 2 columns]

[jreback]

@hayd we need to change first/last to use nth right? (and will fix the perf issue as well)
see here: http://stackoverflow.com/questions/22845856/performance-issues-with-groupbys-last-in-pandas/22846215?noredirect=1#comment34855219_22846215

[hayd]

Yeah, perf of the dropping NaN will still be an issue. (Easiest soln is just to leave current implementation "as is" when using dropna=True / default).

[jreback]

@hayd you are covering this one I believe as well?

[hayd]

yes

[jreback]

@hayd ping!

[jreback]

@hayd

might want to add dropna kw to first/last to emulate this behavior (but can put that off to 0.14.1 if you want).

However, I think that _set_selection_from_grouper needs to be called in nth, right?
(as this should have 'a' as the index, right?

In [13]: df.groupby('a').nth(0,dropna=False)
Out[13]: 
   a   b  c
0  1 NaN  0
2  2   2  2

[2 rows x 3 columns]

In [8]: df.groupby('a').nth(0,dropna='any')
Out[8]: 
   b  c
a      
1  1  0
2  2  2

[2 rows x 2 columns]

In [4]: df.groupby('a').nth(0)
Out[4]: 
   a   b  c
0  1 NaN  0
2  2   2  2

[2 rows x 3 columns]

[alvarouc]

I can confirm first keeps ignoring NaNs in pandas 0.25

[Rabeez]

I can confirm first keeps ignoring NaNs in pandas 1.0.1

[maximlt]

If it's really how it works (which was a nice surprise to me) then it could be mentioned in the docs :)