BUG: KeyError when assigning to Series values after pop from DataFrame

[itssimon]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
series = df.pop("b")
series[[True, False, False]] = 9

Problem description

Throws a KeyError:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/.envs/sigma/lib/python3.8/site-packages/pandas/core/series.py in __setitem__(self, key, value)
   1061         try:
-> 1062             self._set_with_engine(key, value)
   1063         except (KeyError, ValueError):

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/series.py in _set_with_engine(self, key, value)
   1094         # fails with AttributeError for IntervalIndex
-> 1095         loc = self.index._engine.get_loc(key)
   1096         # error: Argument 1 to "validate_numeric_casting" has incompatible type

~/.envs/sigma/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

~/.envs/sigma/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

TypeError: '[True, False, False]' is an invalid key

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/.envs/sigma/lib/python3.8/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3360             try:
-> 3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:

~/.envs/sigma/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

~/.envs/sigma/lib/python3.8/site-packages/pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 'b'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
/tmp/ipykernel_29158/1487425742.py in <module>
      3 df = pd.DataFrame({"a": [1, 2, 3], "b": [4, 5, 6]})
      4 series = df.pop("b")
----> 5 series[[True, False, False]] = 9

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/series.py in __setitem__(self, key, value)
   1080                 key = np.asarray(key, dtype=bool)
   1081                 try:
-> 1082                     self._where(~key, value, inplace=True)
   1083                 except InvalidIndexError:
   1084                     self.iloc[key] = value

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/generic.py in _where(self, cond, other, inplace, axis, level, errors)
   8859             new_data = self._mgr.putmask(mask=cond, new=other, align=align)
   8860             result = self._constructor(new_data)
-> 8861             return self._update_inplace(result)
   8862 
   8863         else:

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/generic.py in _update_inplace(self, result, verify_is_copy)
   4228         self._clear_item_cache()
   4229         self._mgr = result._mgr
-> 4230         self._maybe_update_cacher(verify_is_copy=verify_is_copy)
   4231 
   4232     @final

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/series.py in _maybe_update_cacher(self, clear, verify_is_copy)
   1232                 if len(self) == len(ref):
   1233                     # otherwise, either self or ref has swapped in new arrays
-> 1234                     ref._maybe_cache_changed(cacher[0], self)
   1235                 else:
   1236                     # GH#33675 we have swapped in a new array, so parent

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/frame.py in _maybe_cache_changed(self, item, value)
   3896         The object has called back to us saying maybe it has changed.
   3897         """
-> 3898         loc = self._info_axis.get_loc(item)
   3899         arraylike = value._values
   3900         self._mgr.iset(loc, arraylike)

~/.envs/sigma/lib/python3.8/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:
-> 3363                 raise KeyError(key) from err
   3364 
   3365         if is_scalar(key) and isna(key) and not self.hasnans:

KeyError: 'b'

Note that when changing the line series = df.pop("b") to series = df.pop("b").copy() it works as expected.

Expected Output

No exception, first value in series set to 9.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : f00ed8f
python : 3.8.10.final.0
python-bits : 64
OS : Linux
OS-release : 5.4.117-58.216.amzn2.x86_64
Version : #1 SMP Tue May 11 20:50:07 UTC 2021
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : en_AU.UTF-8
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 1.3.0
numpy : 1.21.0
pytz : 2021.1
dateutil : 2.8.1
pip : 21.1.3
setuptools : 49.6.0.post20210108
Cython : None
pytest : 6.2.4
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.1 (dt dec pq3 ext lo64)
jinja2 : 3.0.1
IPython : 7.25.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 4.0.1
pyxlsb : None
s3fs : None
scipy : 1.7.0
sqlalchemy : 1.3.23
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : 0.53.1

[rhshadrach]

Thanks for the report! This appears to be a bug to me - using df["b"] instead gives the expected result. Investigations and PRs to fix are welcome.

[phofl]

the del self[item] call in pop messes up the _cacher attribute of the view created onto self causing the KeyError later on.

[debnathshoham]

take

[simonjayhawkins]

This is a regression from 1.2.5

first bad commit: [804e08e] REF: move Series-specific methods from NDFrame (#40776)

we should probably backport the fix #42644 (and move the release note) cc @jreback