BUG: Setting multiple values via .loc produces NaNs with MultiIndex

[LustigerKernSpalt]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
i = pd.MultiIndex.from_product([(0, 1), (2, 3)])
s = pd.Series([True]*4, index=i)
s.loc[0,:] = s.loc[0,:] 
print(s)

Issue Description

Executing the script above on pandas 1.4.2 produces NaNs:

0  2     NaN
   3     NaN
1  2    True
   3    True

Expected Behavior

Behavior before 1.4.2 (expected):

0  2    True
   3    True
1  2    True
   3    True

Installed Versions

INSTALLED VERSIONS

commit : 4bfe3d0
python : 3.9.6.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.4.2
numpy : 1.22.3
pytz : 2022.1
dateutil : 2.8.2
pip : 21.1.3
setuptools : 57.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
markupsafe : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
zstandard : None

[simonjayhawkins]

Thanks @LustigerKernSpalt for the report.

Behavior before 1.4.2 (expected):

first bad commit: [35b338e] BUG: .loc failing to drop first level (#42435)

cc @jbrockmendel

[simonjayhawkins]

moving to 1.4.4

[jorisvandenbossche]

The reason this is now failing is similar as some of the cases that are discussed in #46704

Te getitem operation on the right side has changed result:

In [12]: import pandas as pd
    ...: i = pd.MultiIndex.from_product([(0, 1), (2, 3)])
    ...: s = pd.Series([True]*4, index=i)

In [13]: s.loc[0,:]
Out[13]: 
2    True
3    True
dtype: bool

Selecting 0 now dropped that level, returning a Series without MultiIndex. And as a result of that change, now setting with that Series no longer works.
I think one can argue that setting those values still should work, as also in the setitem operation you are selecting the first level, so that should have a similar effect. But that is something that also didn't work in the past:

In [1]: pd.__version__
Out[1]: '1.3.5'

In [2]: import pandas as pd
   ...: i = pd.MultiIndex.from_product([(0, 1), (2, 3)])
   ...: s = pd.Series([True]*4, index=i)

In [3]: s.loc[0, :].droplevel(0)   # <-- the equivalent of what now is returned directly by s.loc[0, :]
Out[3]: 
2    True
3    True
dtype: bool

In [4]: s.loc[0, :] = s.loc[0, :].droplevel(0)

In [5]: s
Out[5]: 
0  2     NaN
   3     NaN
1  2    True
   3    True
dtype: object

[jbrockmendel]

Looks like in Loc._convert_to_indexer we return labels.get_locs(key) which correctly gives us [0, 1], but needs to somehow return info that this indexing is level-dropping, so that when we later call align_series, we take that into account

[simonjayhawkins]

But that is something that also didn't work in the past:

ok, let's not change that this late in the 1.4.x release cycle. removing milestone.