BUG: Added test cases to check loc on multiindex with NaNs #29751

[kasim95]

• closes Inconsistent .loc slicing behaviour with NaNs in MultiIndex dataframe #29751
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Added test cases to check loc on multiindex containing NaN values using np.nan, pd.NA, and None

[phofl]

Could you try to parametrize with slices?

[kasim95]

I have parametrized the slices in the recent commit

[phofl]

There is a fixture for all NaNs

[jreback]

nulls_fixture

[kasim95]

I fixed the test case to use nulls_fixture in the most recent commit

[jreback]

use dtype='int64' to avoid the 32-bit failures

[kasim95]

Changed dtype to int64 in the recent commit

[jreback]

can you assert each of the 4 sliced results in the OP. also pls construct the expected by hard coding as much as possible (clearly the nan needs to come from the fixtures), but hard code the actual values.

[kasim95]

The original post contains the first two sliced results (a.loc) as the reference for the next two sliced results (b.loc). I have replaced the previous tests with the two relevant sliced tests from the original post.

I also had to remove the parameterized test cases because the mypy type checker fails when using a tuple to slice a dataframe. The code I used is as follows:

@pytest.mark.parametrize(
    "indexer,expected_arr",
    [
        (
            lambda df, null: df.loc[:(21, null)],
            lambda null: [[11, null, 13], [21, null, 23]]
        ),
        (
            lambda df, null: df.loc[(21, null):],
            lambda null: [[21, null, 23], [31, null, 33], [41, null, 43]]
        ),
        (
            lambda df, null: df.loc[(21, null):(31, null)],
            lambda null: [[21, null, 23], [31, null, 33]]
        ),
    ],
)
def test_frame_getitem_nan_multiindex(nulls_fixture, indexer, expected_arr):
    # GH#29751
    # loc on a multiindex containing nan values
    arr = [
        [11, nulls_fixture, 13],
        [21, nulls_fixture, 23],
        [31, nulls_fixture, 33],
        [41, nulls_fixture, 43]
    ]
    cols = ["a", "b", "c"]
    df = DataFrame(arr, columns=cols, dtype="int64").set_index(["a", "b"])

    result = indexer(df, nulls_fixture)
    arr1 = expected_arr(nulls_fixture)
    expected = DataFrame(arr1, columns=cols, dtype="int64").set_index(
        ["a", "b"]
    )
    tm.assert_frame_equal(result, expected)

The mypy type checker failed on lines 5, 9, & 13.
I would love to hear any workarounds for multiindex slicing to parameterize the tests.
Instead, I assigned the tuple to a variable and used it to slice the dataframe which passed the mypy type check.

Also, I hardcoded the expected array and the index tuple as requested.

[jreback]

thanks @kasim95