Skip to content

BUG: GroupBy apply loses NaN groups #43227

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
misantroop opened this issue Aug 26, 2021 · 2 comments
Closed
2 of 3 tasks

BUG: GroupBy apply loses NaN groups #43227

misantroop opened this issue Aug 26, 2021 · 2 comments
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member

Comments

@misantroop
Copy link

misantroop commented Aug 26, 2021

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Here I'm expecting GroupBy to return an identical structure but NaN groups get removed even with dropna=False. Providing the level argument as a keyword works as expected.

    mux = pd.MultiIndex.from_arrays([['a', 'a', np.nan, 'b', 'b'], ['t', 'u', np.nan, 'w', 'y']],
                                    names=['level1', 'level2'])
    df = pd.DataFrame({'col': [0, np.nan, np.nan, 3, 4]}, mux)

    	       	    col
    level1  level2
    a	    t	    0.0
            u	    NaN
    NaN	    NaN	    NaN
    b	    w	    3.0
            y	    4.0

    df = df.groupby(by='level2', dropna=False).apply(lambda x: x)

    	       	   col
    level1  level2
    a	    t	   0.0
            u	   NaN
    b	    w	   3.0
            y	   4.0

    df = df.groupby(level='level2', dropna=False).apply(lambda x: x)

    	       	    col
    level1  level2
    a	    t	    0.0
            u	    NaN
    NaN	    NaN	    NaN
    b	    w	    3.0
            y	    4.0
@misantroop misantroop added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 26, 2021
@phofl
Copy link
Member

phofl commented Aug 26, 2021

Hi, have you checked #43205? Looks quite similar

@misantroop misantroop changed the title BUG: GroupBy apply loses NaN groups when multiindex level passed as a non-keyword BUG: GroupBy apply loses NaN groups Aug 26, 2021
@misantroop
Copy link
Author

Likely related if not identical.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Needs Triage Issue that has not been reviewed by a pandas team member
Projects
None yet
Development

No branches or pull requests

2 participants