BUG: Fix aligning a DataFrame with a Series with MultiIndex #46001

[johannes-mueller]

When aligning a DataFrame to a Series we are using Series.reindex() to
broadcast the Series data to the new index. That introduces NaNs when the new
index rows are not identical to the existing ones, which is not the case when
we introduce a new MultiIndex level.

In this patch we use the same approach as for aligning a Series to another
Series. That means that we have to replicate a part of Series._reindex_indexer
as DataFrame does not have it.

• closes BUG: alignment of MultiIndexed DataFrame to a Series with a common index level #46001
  and BUG: alignment in operations with multi-index #43321
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added an entry in the latest doc/source/whatsnew/v1.5.0.rst.

[jreback]

cc @phofl @jbrockmendel if you can have a look

[jreback]

small comment

cc @phofl @jbrockmendel if comments

[jbrockmendel]

is there a reason not to use the BlockManager method here?

[johannes-mueller]

I am not sure what exactly you mean by BlockManager method. The way I came to my solution was that I tried to replicate what Series._reindex_indexer does. So what I am doing here is inspired by this.

EDIT: The CI has already bitten me with an BlockManager ArrayManager issue (see for example here).

[jbrockmendel]

I am not sure what exactly you mean by BlockManager method.

I was thinking Manager.take (definitely only use public Manager methods)

EDIT: The CI has already bitten me with an BlockManager ArrayManager issue (see for example here).

yah, ArrayManager is a PITA.

[johannes-mueller]

I tried

left = self.take(lidx)
left.index = join_index

but that makes frame/test_align.py::test_align_series_combinations and a lot of tests in frame/test_arithmetic.py fail. The issue is, that when other.index has elements which are not in self.index. Then lidx contains -1 and so .take() will put in the last element instead of np.nan.

Consider the following example:

import pandas as pd

df = pd.DataFrame({'x': [1., 2., 3., 44.], 'y': [5., 6., 7., 88.]}, index=pd.Index([1, 2, 8, 9]))
series = pd.Series([10., 20., 30., 40.], index=pd.Index([1, 2, 3, 4]))

join_index, lidx, rindx = df.index.join(series.index, how="outer", return_indexers=True)

with_algos_take_nd = pd.DataFrame(
    pd.core.algorithms.take_nd(df.values, lidx), columns=['x', 'y'], index=join_index
)
with_take = df.take(lidx)
with_take.index = join_index

print("with take_nd")
print(with_algos_take_nd)
print("with take")
print(with_take)

gives

with take_nd
      x     y
1   1.0   5.0
2   2.0   6.0
3   NaN   NaN
4   NaN   NaN
8   3.0   7.0
9  44.0  88.0
with take
      x     y
1   1.0   5.0
2   2.0   6.0
3  44.0  88.0
4  44.0  88.0
8   3.0   7.0
9  44.0  88.0

[jbrockmendel]

I tried [...] left = self.take(lidx)

The name overlap can be confusing: by "Manager.takei meantself._mgr.take, which behaves differently from self.take`.

[jbrockmendel]

this will cause a problem if columns are not unique. is that case ruled out here?

[johannes-mueller]

It is not. Haven't thought of that. I have added a test for that case and also implemented a solution for it. However, I am not sure if the BlockManager method that you are referring to is better.

[johannes-mueller]

Made some further simplifications and added some more tests

[jreback]

thanks @johannes-mueller

[jbrockmendel]

self.values can be expensive (and lossy) if the dataframe is not a single block.

[jreback]

@johannes-mueller sorry would u mind doing a follow up address comments? thanks