.loc with partial key flattens MultiIndex on index

[toobaz]

Code Sample, a copy-pastable example if possible

In [2]: df = pd.DataFrame([[1,2], [3,4]], index=pd.MultiIndex.from_product([['a', 'b'], ['c']]))

In [3]: df.loc['all'] = [5,6]

In [4]: df
Out[4]: 
        0  1
(a, c)  1  2
(b, c)  3  4
all     5  6

Problem description

From #16599 (comment)

In general, scalar labels are accepted as valid keys for MultiIndexes and automatically transformed to (key, ''). Compare to

In [2]: df = pd.DataFrame([[1,2], [3,4]], columns=pd.MultiIndex.from_product([['a', 'b'], ['c']]))

In [3]: df['all'] = [5,6]

In [4]: df
Out[4]: 
   a  b all
   c  c    
0  1  2   5
1  3  4   6

Expected Output

In [4]: df
Out[4]: 
       0  1
a   c  1  2
b   c  3  4
all    5  6

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 3.5.3.final.0 python-bits: 64 OS: Linux OS-release: 4.9.0-3-amd64 machine: x86_64 processor: byteorder: little LC_ALL: None LANG: it_IT.UTF-8 LOCALE: it_IT.UTF-8

pandas: 0.21.0.dev+265.g9e7666dae
pytest: 3.0.6
pip: 9.0.1
setuptools: None
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
xarray: None
IPython: 5.1.0.dev
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: 3.7.1
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: 0.2.1

[toobaz]

(also exhibited in https://stackoverflow.com/questions/44949953/how-to-add-a-row-to-a-pandas-dataframe-without-flattening-the-multiindex )

[gfyoung]

@toobaz : Agreed, that behavior doesn't look right. PR is welcome!

[phofl]

The conversion happens in concat. Is this expected? Meaning that concat converts MultiIndex and Index to Index with tuples?

[toobaz]

The conversion happens in concat. Is this expected?

I would have probably had a preference for concat transforming the flat index in the first level of a MultiIndex... but definitely not strong enough to change the current behavior of concat, so I would take another path to fix the above (maybe the one followed by the column case, which works?)