REGR: Groupby first/last/nth treats None as an observation

[rhshadrach]

• closes BUG: GroupBy.first does not skip missing values in string-valued columns #38286
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

This is a regression from 1.0.x to 1.1.x, introduced by #33462. Assuming it doesn't get into the 1.2 rc, not sure if it should go into 1.2 during the rc phase or wait until 1.3.

Had to decide on some edge-case behaviors in odd situations with missing values, see #38286 (comment)

[jreback]

pls find out when this was originally changed

this was on purpose

[rhshadrach]

@jreback This was originally changed in #33462 so that

pd.DataFrame({"id": ["a"], "value": [None]}, dtype=object).groupby("id").first()

returns None rather than np.nan. That behavior is retained here.

[jreback]

so you think that the original issues are wrong? this is deleting that test case and reverting

[simonjayhawkins]

This is a regression from 1.0.x to 1.1.x

if this is the case, should go in 1.1.5.

[rhshadrach]

@jreback I believe this is reverting the fix in #33462 and solving the issue there in the proper manner, along with the regression it caused. The original test case still exists, is the first parameter to test_first_last_with_none

[rhshadrach]

I can restore the original test and add the new one for the regression if that's preferable.

[jreback]

yes if this also doesn't change the original issue then leave the tests

ok for 1.1.5

[rhshadrach]

@jreback test restored, added to whatsnew 1.5

[jreback]

this doesn't affect nth right?

[rhshadrach]

Indeed, thanks. Fixed.

[jreback]

can u run some asvs / perf for this
i am worried we are converting to object type

[rhshadrach]

@jreback asv run with asv continuous -f 1.1 upstream/master HEAD -b ^groupby resulted in BENCHMARKS NOT SIGNIFICANTLY CHANGED.

Linux py39 failure is unrelated:

FAILED pandas/tests/io/pytables/test_store.py::TestHDFStore::test_append_with_data_columns
= 1 failed, 143282 passed, 5482 skipped, 1091 xfailed, 4 warnings in 1132.31s (0:18:52) =

[jreback]

thanks for the quick patch @rhshadrach

[simonjayhawkins]

@meeseeksdev backport 1.1.x

[amalcgcg]

We just upgraded to 1.1.5 this morning, and it works great! Thank you for the blazing fast discussion, decision, fix, merge, and backport!