BUG: Regression in 1.1.0: ValueError when assigning tuple

[zeeMonkeez]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd
d = pd.DataFrame({'a':[1,1,2,1], 'b':[(1,1,0)]*4})
d.loc[d['a']==1, 'b'] = [[(0, 0, 1)]]

Traceback (most recent call last):
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-7-9e18787827bf>", line 1, in <module>
    d.loc[d['a']==1, 'b'] = [[(0, 0, 1)]]
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/pandas/core/indexing.py", line 670, in __setitem__
    iloc._setitem_with_indexer(indexer, value)
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/pandas/core/indexing.py", line 1666, in _setitem_with_indexer
    raise ValueError(
ValueError: cannot set using a multi-index selection indexer with a different length than the value

And similarly:

import pandas as pd
d = pd.DataFrame({'a':[1,1,2,1], 'b':[(1,1,0)]*4})
ixs = [0,1,3]
vals = [[(0,0,1)]*len(ixs)]
d.loc[ixs,['b']] = vals

Traceback (most recent call last):
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3418, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-8-73385df8ecbc>", line 1, in <module>
    d.loc[ixs,['b']] = vals
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/pandas/core/indexing.py", line 670, in __setitem__
    iloc._setitem_with_indexer(indexer, value)
  File "/opt/anaconda3/envs/test/lib/python3.8/site-packages/pandas/core/indexing.py", line 1666, in _setitem_with_indexer
    raise ValueError(
ValueError: cannot set using a multi-index selection indexer with a different length than the value

Also see related SO question.

Problem description

The code posted above works until pandas version 1.0.5, and fails in 1.1.0. I checked deprecation notes for 1.1.0 and could not identify if this is expected behaviour now.

Expected Output

After both assignment operations, d should have the following contents:

>>> print(d)

   a          b
0  1  (0, 0, 1)
1  1  (0, 0, 1)
2  2  (1, 1, 0)
3  1  (0, 0, 1)

Output of pd.show_versions()

INSTALLED VERSIONS

commit : db08276
python : 3.8.5.final.0
python-bits : 64
OS : Darwin
OS-release : 19.6.0
Version : Darwin Kernel Version 19.6.0: Mon Aug 31 22:12:52 PDT 2020; root:xnu-6153.141.2~1/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : en_US.UTF-8
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.1.3
numpy : 1.19.4
pytz : 2020.1
dateutil : 2.8.1
pip : 20.2.4
setuptools : 50.3.0.post20201103
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.19.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.3.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
numba : None

[ssortman]

Hi. I'm new and looking to help out on this project! Is this an issue that I'd be able to pick up and work on?

[phofl]

cc @ssortman of course, feel free to have a look.

[phofl]

Labeling as a Regression, since it worked on 1.0.5

[simonjayhawkins]

first bad commit: [a12ab06] CLN: simplify _setitem_with_indexer (#31887) cc @jbrockmendel

[simonjayhawkins]

@zeeMonkeez This may have happened to work in previous versions, but as you point out in the SO conversation, how is the nested list interpreted?

If this is not a documented/expected behaviour, I'm inclined to remove the regression tag.

I think the following demonstrates that the nested list just happened to work for a specific case.

>>> pd.__version__
'1.0.5'
>>>
>>> df = pd.DataFrame({"a": [1, 1, 2, 1], "b": [(1, 1, 0)] * 4})
>>> df
   a          b
0  1  (1, 1, 0)
1  1  (1, 1, 0)
2  2  (1, 1, 0)
3  1  (1, 1, 0)
>>>
>>> df.loc[df["a"] == 2, "b"] = [[(0, 0, 1)]]
>>> df
   a            b
0  1    (1, 1, 0)
1  1    (1, 1, 0)
2  2  [(0, 0, 1)]
3  1    (1, 1, 0)
>>>
>>> df.loc[df["a"] == 2, "b"] = [(0, 0, 1)]
Traceback (most recent call last):
...
ValueError: Must have equal len keys and value when setting with an ndarray
>>>
>>> df.loc[df["a"] == 2, "b"] = (0, 0, 1)
Traceback (most recent call last):
...
ValueError: Must have equal len keys and value when setting with an iterable
>>>

[simonjayhawkins]

@jbrockmendel thoughts on how to proceed here?

[jbrockmendel]

1. d.loc[d['a']==1, 'b'] = [[(0, 0, 1)]] is sketchy, im not sure if it should work. The LHS is a Series, and the RHS is 2D-like.

If instead of a nested list we had a 2D ndarray[object], this would be pretty easy. To fix this via that route, we need a helper function we can call at the beginning of setitem_with_indexer (or possibly before?) that does this inference/casting.

I wouldn't mark this as a regression, but im also notoriously cranky when it comes to nested data.

[kevin-guimard-ext]

The problem doesn't only happen with nested lists. Here is a simpler case that triggers the error:

df = pd.DataFrame({"a":[1]})
df.at[0, "b"] = [2, 3]