BUG: GroupBy.first does not skip missing values in string-valued columns

[amalcgcg]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Create a DataFrame with numbers and missing entries, first() on groupby works as expected,
# skipping the missing leading entry in column b
>>> df2=pd.DataFrame({"a": [4,4,4,4,4,4], "b": [None,3,None,3,3,3]})
>>> df2.groupby("a").first()
     b
a
4  3.0

# Create a similar DataFrame, but with strings and missing entries, first() on groupby 
# does not work as expected, it doesn't skip the missing leading entry
df2=pd.DataFrame({"a": [4,4,4,4,4,4], "b": [None,"foo",None,"foo","foo","foo"]})
>>> df2.groupby("a").first()
      b
a
4  None

Problem description

One utility of the GroupBy.first() function is its coalescing behavior--it skips missing values within groups. This behavior does not seem to happen when a column has string values, which creates two problems: 1) type-dependent behavior of an otherwise generic method, and 2) I'm not able to find a workaround to get the behavior I'd like (returning the first non-missing string in a group.

Expected Output

I would expect the output of the second example above to be "foo", not None. If the behavior above is the expected behavior of Pandas, then please provide the correct code to use in order to extract the first non-missing string in a groupby column.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : 67a3d42
python : 3.8.5.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Thu Oct 29 22:56:45 PDT 2020; root:xnu-6153.141.2.2~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.1.4
numpy : 1.19.4
pytz : 2020.4
dateutil : 2.8.1
pip : 20.2.3
setuptools : 50.2.0
Cython : None
pytest : 6.1.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : None
odfpy : None
openpyxl : 3.0.5
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[amalcgcg]

Not ideal, but a potential workaround is:

df2.groupby("a").apply(lambda grp: grp.iloc[:,1:].apply(lambda col: next((val for val in col if val), None)))
     b
a
4  foo

Is this really the simplest way to get the first non-null string in each column of a DataFrame group?

[rhshadrach]

Thanks for the report! This is indeed a bug. Looking into this, I am wondering what should be done about pathological cases like:

df = pd.DataFrame({'a': 1, 'b': [None, np.nan]})
df.groupby('a').last()

One might expect that the result here would be np.nan. However, this opens up various consistency issues with nth. If there are n null-type values then it would make sense that we would return the nth one, but if there are less than n do we return None or the last one?

All of this complexity is not worth it IMO and we should just declare that if the input dtype is object and the group does not have sufficient non-null-types, then the result is None.

[rhshadrach]

This is a regression from 1.0.x to 1.1.x

[amalcgcg]

We just upgraded to 1.1.5 this morning, and it works great! Thank you for the blazing fast discussion, decision, fix, merge, and backport!