BUG: errors with .loc[(slice(...), ), ] when modifying a subset of rows in a pandas dataframe/series in 1.1.4

[taozuoqiao]

I often use .loc[(slice(...),),] for selection and modifying when dealing with both series and dataframe: see
advanced-indexing-with-hierarchical-index

In [43]: df.loc['bar']
Out[43]:
A B C
second
one 0.895717 0.410835 -1.413681
two 0.805244 0.813850 1.607920

This is a shortcut for the slightly more verbose notation df.loc[('bar',),] (equivalent to df.loc['bar',] in this example).

Every thinkg is OK in 0.25.4, but in 1.1.4 (also 1.0.3 and 1.1.3), only selection is valid and modifying will raise errors:

import pandas as pd
s = pd.Series([1,2,3], index=pd.MultiIndex.from_tuples([('a','A'),  ('b','B'), ('c', 'C')]))
df = s.to_frame()
print(s.loc[('a', ), ])
print(df.loc[('a', ), ])

but when I try to modify a subset of rows using .loc[(slice(...),),]

In [2]: df.loc[('a', ), ] = 0
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-2-480fc1763dde> in <module>
----> 1 df.loc[('a', ), ] = 0

~/opt/anaconda3/envs/edge/lib/python3.8/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)                                                                    
    664         else:
    665             key = com.apply_if_callable(key, self.obj)
--> 666         indexer = self._get_setitem_indexer(key)
    667         self._has_valid_setitem_indexer(key)
    668

~/opt/anaconda3/envs/edge/lib/python3.8/site-packages/pandas/core/indexing.py in _get_setitem_indexer(self, key)                                                                  
    591         """
    592         if self.name == "loc":
--> 593             self._ensure_listlike_indexer(key)
    594
    595         if self.axis is not None:

~/opt/anaconda3/envs/edge/lib/python3.8/site-packages/pandas/core/indexing.py in _ensure_listlike_indexer(self, key, axis)                                                        
    645             # key may be a tuple if we are .loc
    646             # in that case, set key to the column part of key
--> 647             key = key[column_axis]
    648             axis = column_axis
    649

IndexError: tuple index out of range

and

In [3]: s.loc[('a', ), ] = 0
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-3-c3a2e38a0417> in <module>
----> 1 s.loc[('a', ), ] = 0

~/opt/anaconda3/envs/edge/lib/python3.8/site-packages/pandas/core/indexing.py in __setitem__(self, key, value)
    668 
    669         iloc = self if self.name == "iloc" else self.obj.iloc
--> 670         iloc._setitem_with_indexer(indexer, value)
    671 
    672     def _validate_key(self, key, axis: int):

~/opt/anaconda3/envs/edge/lib/python3.8/site-packages/pandas/core/indexing.py in _setitem_with_indexer(self, indexer, value)
   1633         if take_split_path:
   1634             # Above we only set take_split_path to True for 2D cases
-> 1635             assert self.ndim == 2
   1636             assert info_axis == 1
   1637 

AssertionError:

Note that “partial” indexing .loc[slice(...)] is still valid in ``1.1.4`:

In [4]: df.loc['a'] = 0

In [5]: s.loc['a'] = 0

In [6]: df
Out[6]: 
     0
a A  0
b B  2
c C  3

In [7]: s
Out[7]: 
a  A    0
b  B    2
c  C    3
dtype: int64

Of course, .loc[('a', )] is valid for series and .loc[('a', ), : ] is vaild for dataframe, but I have to use different codes for series and dataframe.

Update on 20201113:

• For series, s.loc[('a')] = 0, s.loc[('a', )] = 0 and s.loc[('a'), ] = 0 are vaild, but s.loc[('a', ), ] = 0 will raise similar AssertionError like above.
• For dataframe, df.loc[('a')] = 0, df.loc[('a'), :] = 0 and df.loc[('a',), :] = 0 are vaild, but df.loc[('a', )] = 0 , df.loc[('a'), ] = 0 and df.loc[('a', ), ] = 0 will raise similar IndexError like above.

[simonjayhawkins]

@taozuoqiao thanks for the report

first bad commit: [810a4e5] BUG: assignment to multiple columns when some column do not exist (#29334) cc @howsiwei

[taozuoqiao]

Thanks very much for the answering and fix. I just update my comment, adding more examples of setting values which could fail in 1.1.3. I am rather new to pandas, and I am not sure whether they will all be fixed by phofl's PR,

• For series, s.loc[('a')] = 0, s.loc[('a', )] = 0 and s.loc[('a'), ] = 0 are vaild, but s.loc[('a', ), ] = 0 will raise similar AssertionError like above.
• For dataframe, df.loc[('a')] = 0, df.loc[('a'), :] = 0 and df.loc[('a',), :] = 0 are vaild, but df.loc[('a', )] = 0 , df.loc[('a'), ] = 0 and df.loc[('a', ), ] = 0 will raise similar IndexError like above.

They are all valid when getting values.

[phofl]

I am not quite sure, if s.loc[('a', ), ] = 0 this should work, a Series has no columns. I think s.loc[('a'), ] = 0 works wrongly?

[taozuoqiao]

I agree with you, they are really very ambiguous practices.