BUG: DataFrame.__setitem__ creates object-dtype array for extension type scalars

[justinessert]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• (optional) I have confirmed this bug exists on the master branch of pandas.

---

Code Sample, a copy-pastable example

import pandas as pd

df = pd.DataFrame({
    'a': [1, 2]
})

df['b'] = pd.Period('2020-01')
df['c'] = df['b'] + 0
df['d'] = df['a'] + df['b']

df.dtypes

Problem description

Not sure if this is best classified as a bug, but wasn't sure how to classify it since it certainly defies my expectations of how the Pandas package would work and would like to understand what's happening.

Essentially the issue is that, in the code snippet above, df['b']'s is considered an object rather than a period[M]. Stranger to me still is that both df['c'] and df['d'] are correctly set to period[M] columns. This seems very odd to me, that when you combine an object column with a int, you get a period[M] column.

Can you help me understand what the logic is here? Is it possible to update Pandas so that when someone does df['b'] = pd.Period('2020-01') the resulting column is a period[M] type?

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Darwin
OS-release : 19.4.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.4
numpy : 1.18.5
pytz : 2019.1
dateutil : 2.8.0
pip : 19.1.1
setuptools : 41.0.1
Cython : None
pytest : 4.4.1
hypothesis : None
sphinx : 2.4.2
blosc : None
feather : None
xlsxwriter : 1.2.8
lxml.etree : 4.4.0
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10.1
IPython : 7.5.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.4.0
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 4.4.1
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 1.2.0
xlwt : None
xlsxwriter : 1.2.8
numba : None

[TomAugspurger]

Slightly related to #31763.

Bug is somewhere in

pandas/pandas/core/frame.py

Lines 3731 to 3734 in 5fdd6f5

	infer_dtype, _ = infer_dtype_from_scalar(value, pandas_dtype=True)
	# upcast
	value = cast_scalar_to_array(len(self.index), value)

/ cast_scalar_to_array

pandas/pandas/core/dtypes/cast.py

Line 1491 in 5fdd6f5

def cast_scalar_to_array(shape, value, dtype: Optional[DtypeObj] = None) -> np.ndarray:

.

(Pdb) value
Period('2020-01', 'M')
(Pdb) pp infer_dtype
period[M]
(Pdb) pp cast_scalar_to_array(len(self.index), value)
array([Period('2020-01', 'M'), Period('2020-01', 'M')], dtype=object)

Ideally we would recognize that as an extension dtype and create the desired array with

(Pdb) pp infer_dtype.construct_array_type()._from_sequence([value] * len(self.index))
<PeriodArray>
['2020-01', '2020-01']
Length: 2, dtype: period[M]

Further investigation / PRs to fix this would be very welcome.

Further investigation

[jreback]

pls try on master

[TomAugspurger]

Confirmed that's the same behavior on master.

[justinessert]

Thank you @TomAugspurger for the lead!

Looking into this issue a little further I found two notable issues that cause this (not sure how much I'm just repeating @TomAugspurger):

Issue 1: cast_scalar_to_array(len(self.index), value) determines the data type of a variable calls infer_dtype_from_scalar(value). But the latter function will not check for a pandas DType unless it is called like infer_dtype_from_scalar(value, pandas_dtype=True). Calling the function this way will correctly identify that the value is a pd.Period

Issue 2: The cast_scalar_to_array(len(self.index), value) returns a Numpy Array, but a Numpy Array obviously can't be a Pandas data type. So even if Issue 1 is fixed via setting pandas_dtype=True, the function will fail.

To correctly handle this bug, the function would have to return a Period Array (is this what you are referring to as an extension array?) via the code @TomAugspurger provided above. Is this an acceptable way of solving this issue?

I believe this fix would also require some code do auto-detect whether a scalar value is a Period type, so that the function knows whether to return an extension array or a numpy array. Is that something that exists currently?

[TomAugspurger]

To correctly handle this bug, the function would have to return a Period Array (is this what you are referring to as an extension array?

Yeah. So cast_scalar_to_array(10, pd.Period("2000")) should return a length-10 PeriodArray.

[simonjayhawkins]

Is it possible to update Pandas so that when someone does df['b'] = pd.Period('2020-01') the resulting column is a period[M] type?

>>> df = pd.DataFrame({
...     'a': [1, 2]
... })
>>>
>>> df['b'] = pd.Period('2020-01')
>>> df['c'] = df['b'] + 0
>>> df['d'] = df['a'] + df['b']
>>>
>>> df.dtypes
a        int64
b    period[M]
c    period[M]
d    period[M]
dtype: object
>>> pd.__version__
'1.2.0.dev0+317.g6aa311db41'
>>>

@justinessert #34875 fixed this and added tests. This issue can be closed?