PERF: tighten _should_compare for MultiIndex

[jbrockmendel]

• closes #xxxx
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[jreback]

do tests hit this? e.g. as you didn't change anything

[jbrockmendel]

One of the affected cases was not tested; just added a test for that case.

[jreback]

is there a specific benchmark that this improves?

[jbrockmendel]

is there a specific benchmark that this improves?

No. The motivation is yak-shaving that traces back to getting rid of _convert_list_indexer

[jbrockmendel]

@jreback gentle ping (a whole mess of MultiIndex PRs yak-shaving inconsistencies)

[bashtage]

@jbrockmendel This PR seems to have introduced a bug. I verified this with the code below and a bisect.

import pandas as pd
mi = pd.MultiIndex.from_product([["a","b","c"],[1,2,3],["z","y","x"]])
df = pd.DataFrame(index=mi,dtype=float)
mi2 = pd.MultiIndex.from_product([["a","b","c"],[1,2,3]])
s = pd.Series(index=mi2, dtype=float)
s.iloc[:]=3.14
df["new"] = s
print(df)

now returns

        new
a 1 z  NaN
    y  NaN
    x  NaN
  2 z  NaN
    y  NaN
    x  NaN
  3 z  NaN
    y  NaN
    x  NaN

Before this patch is did a broadcast assignment to the remaining MultiIndex levels, i.e..

        new
a 1 z  3.14
    y  3.14
    x  3.14
  2 z  3.14
    y  3.14
    x  3.14
  3 z  3.14
    y  3.14
    x  3.14

[bashtage]

xref #40186

[bashtage]

This is the change that is breaking MultiIndex broadcasting. If one has 3 levels and the other has 2, then this is False. Previously these were comparable and so would be compared and expanded.

[jbrockmendel]

thanks. do you know what the calling method is in the problematic case?

[bashtage]

Walking back, previous is

pandas/pandas/core/indexes/base.py

Line 3481 in ddd90b0

if not self._should_compare(target) and not self._should_partial_index(target):

then

pandas/pandas/core/indexes/base.py

Line 3887 in ddd90b0

indexer = self.get_indexer(

Here self is the Series with 2 levels and other is the DataFrame with 3.

[bashtage]

The big change is driven by the return difference of self._should_compare(target). Before this patch it returned True, so the if not ... block was skipped. It now returns False, and so it incorrectly shortcuts and fills with an NA value.

[jbrockmendel]

OK, i think ive got a handle on whats going on here. The long-term fix will be in MultiIndex.get_indexer, but for now this should just be reverted.