Bug in loc not raising KeyError with MultiIndex containing no longer used levels

[phofl]

• closes BUG: df.loc[[x], :] fails if df has zero rows #41170
• closes BUG: reindex on empty Series raises ValueError: zero-size array to reduction operation #40235
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

stumbled across a bug in MultiIndex.reindex, which caused the ValueError from the op

[attack68]

Not sure if this is relevant to this PR but wanted to raise to the point that KeyErrors with MultiIndexes I think are dangerous because a MultiIndex cannot be easily reindexed, to permit a valid key.

For example if you wanted to reindex a MultiIndex and insert a single key, a, into one level of a multilevel index then it is ambiguous how we might do that, and in the exhaustive case of allowing a to all possible combinations of values of keys within the other levels you will quickly exhaust the memory due to the number of combinations.

I previously raised an issue where I thought the opposite (basing my opinion solely on the case of consistency with single indexes) but revised later after I realised it could be unworkable.

[phofl]

Yes, I think I remember this discussion, but this does not really apply here, since "b" is not in the Index at all, the same as

df = pd.DataFrame({"A": [12,23,34,45]}, index = [list("aabb"), [0,1,2,3]])
df.loc[["c"], :]

I think the discussion was about something like:

df.loc[("a", 3), :]

where both elements are in the index, but the combination of both is not?

[jreback]

this was user facing in a Series.reindex right? can you add the user facing part of this test as well.

[phofl]

Done, but is essentially the same

[attack68]

Yes, I think I remember this discussion, but this does not really apply here, since "b" is not in the Index at all, the same as

df = pd.DataFrame({"A": [12,23,34,45]}, index = [list("aabb"), [0,1,2,3]])
df.loc[["c"], :]

I think the discussion was about something like:

df.loc[("a", 3), :]

where both elements are in the index, but the combination of both is not?

But my point in this context is that if a user has some value he want to slice over that is not in df.index for a single index he has two options:

• the index can reindexed (to include "c") which is the suggested route given the introduction of a KeyError.
• the indexer arg must be altered to explicitly exclude missing values from df.index (i.e. remove "c")

Since MultiIndexes cannot be reindexed without ambiguity or without necessarily expanding the number of level combinations to fill memory (in this case adding "c" would return (c,0) (c,1) (c,2) (c,3) which is exponentially bad) then the user is restricted, in the case of MultiIndexes to:

• the slice must be altered.

Which means that for each level the user must compare their indexer level args and exclude those that are not present.

I think this level of restriction will be more problematic than simply not returning values that don't exist?

[phofl]

Ah ok, did not remember this. I think this is a discussion we should have, but the pr is as you mentioned above probably not the right place, since this makes behavior more consistent and does not really change anything. Maybe ping on the other issue were the initial discussion was?

[attack68]

ok take a look here: #41358 sorry: #39424

[jreback]

thanks @phofl very nice.

@attack68 thanks for all of the comments & notes on MI indexing. Let's have that discussion in the issue you pointed. #39424