BUG: FutureWarning appears when assigning new column through loc

[utilifeedagnes]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
df1 = pd.DataFrame({"a": [1]})
df2 = pd.DataFrame({"d": [True]})
df1.loc[df2.index, df2.columns] = df2

Issue Description

The code above gives
FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value '[ True]' has dtype incompatible with float64, please explicitly cast to a compatible dtype first. df1.loc[df2.index, df2.columns] = df
where I do not expect any warning at all. A new column is created without prior dtype defined.

Expected Behavior

No FutureWarning raised.

Installed Versions

INSTALLED VERSIONS

commit : a60ad39
python : 3.11.5.final.0
python-bits : 64
OS : Linux
OS-release : 5.15.0-87-generic
Version : #97~20.04.1-Ubuntu SMP Thu Oct 5 08:25:28 UTC 2023
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.1.2
numpy : 1.26.0
pytz : 2023.3.post1
dateutil : 2.8.2
setuptools : 65.5.0
pip : 23.2.1
Cython : None
pytest : 7.4.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : 1.3.7
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : 3.8.0
numba : None
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 13.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.11.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[lithomas1]

cc @MarcoGorelli.

[aureliobarbosa]

Small contribution. A minimal, reproducible example, giving an almost identical FutureWarning, could be:

df = pd.DataFrame({"a": [1]})
df.loc[0, "b"] = True

The only difference is that instead of indicating that '[ True]' has dtype incompatible with float64, this example captures True. Note that the problem is related to the dtype of the newly generated column.

print(df["b"].dtype)

Returns object.

[MarcoGorelli]

Thanks - indeed, the dtype of the column created here is object, not bool (as you might have expected):

In [9]: df = pd.DataFrame({"a": [1]})
   ...: df.loc[0, "b"] = True
<ipython-input-9-44fb3a4bf8be>:2: FutureWarning: Setting an item of incompatible dtype is deprecated and will raise in a future error of pandas. Value 'True' has dtype incompatible with float64, please explicitly cast to a compatible dtype first.
  df.loc[0, "b"] = True

In [10]: df.dtypes
Out[10]:
a     int64
b    object
dtype: object

You should instead do

In [11]: df = pd.DataFrame({"a": [1]})
    ...: df.loc[:, "b"] = True

In [12]: df.dtypes
Out[12]:
a    int64
b     bool
dtype: object

and then you'll get a bool column as expected

This is exactly what the deprecation warning is meant to address

---

Still, would be nice to be able to do this without the upcasting happening at all, just wondering how to do it without further spaghettifying logic

---

culprit is

pandas/pandas/core/indexing.py

Line 875 in 0e2277b

self.obj._mgr = self.obj._mgr.reindex_axis(keys, axis=0, only_slice=True)

which expands by setting a new column to float with nan

[MarcoGorelli]

To give a longer explanation - what's the difference between

• df1.loc[:, df2.columns] = df2
• df1.loc[df1.index, df2.columns] = df2
  ?

The difference is

pandas/pandas/core/indexing.py

Lines 860 to 876 in 0e2277b

	# e.g. if we are doing df.loc[:, ["A", "B"]] = 7 and "B"
	# is a new column, add the new columns with dtype=np.void
	# so that later when we go through setitem_single_column
	# we will use isetitem. Without this, the reindex_axis
	# below would create float64 columns in this example, which
	# would successfully hold 7, so we would end up with the wrong
	# dtype.
	indexer = np.arange(len(keys), dtype=np.intp)
	indexer[len(self.obj.columns) :] = -1
	new_mgr = self.obj._mgr.reindex_indexer(
	keys, indexer=indexer, axis=0, only_slice=True, use_na_proxy=True
	)
	self.obj._mgr = new_mgr
	return
	self.obj._mgr = self.obj._mgr.reindex_axis(keys, axis=0, only_slice=True)

In the first case, BaseBlockManager.reindex_indexer is called with use_na_proxy=True, which creates a new block of dtype np.void, whereas in the second case it creates a new block of dtype 'float64'

Then, they get to

pandas/pandas/core/indexing.py

Lines 2101 to 2118 in 0e2277b

	if is_null_setter:
	# no-op, don't cast dtype later
	return
	elif is_full_setter:
	try:
	self.obj._mgr.column_setitem(
	loc, plane_indexer, value, inplace_only=True
	)
	except (ValueError, TypeError, LossySetitemError):
	# If we're setting an entire column and we can't do it inplace,
	# then we can use value's dtype (or inferred dtype)
	# instead of object
	self.obj.isetitem(loc, value)
	else:
	# set value into the column (first attempting to operate inplace, then
	# falling back to casting if necessary)
	self.obj._mgr.column_setitem(loc, plane_indexer, value)

The first case is a full setter, the second one isn't. So, they both attempt self.obj._mgr.column_setitem(, but:

• the first one does so with inplace_only=True, so it then raises here:
  

  pandas/pandas/core/internals/base.py

  Line 326 in 192a4e3

  value = np_can_hold_element(arr.dtype, value)

  The LossySetItemError is raised, so it goes to self.obj.isetitem(loc, value) instead

• the second one doesn't, so it ends up here:
  

  pandas/pandas/core/internals/blocks.py

  Lines 1280 to 1285 in 5f88eb1

  	try:
  	casted = np_can_hold_element(values.dtype, value)
  	except LossySetitemError:
  	# current dtype cannot store value, coerce to common dtype
  	nb = self.coerce_to_target_dtype(value, warn_on_upcast=True)
  	return nb.setitem(indexer, value)

  There, the LossySetItemError is handled, and the warning is raised

So, in the first place, the same warning should probably be raised as well

As for what to do about not warning when expanding, I don't know yet

[aureliobarbosa]

Thanks for the explanation @MarcoGorelli. df.loc[0,'b'] follows through a different path! This helps us understand how a new dataframe column is created.

[MarcoGorelli]

Problem is, we can't just raise within the except LossySetItemError block, because

        df = DataFrame({"a": [1, 2], "b": [3, 4]})
        df.loc[:, "a"] = {1: 100, 0: 200}

also goes there

This extreme flexibility comes at a cost unfortunately, will see what can be done

---

Also, came across #56024 whilst investigating