BUG: Setting data of same dtype as existing column in shallow copy DF causes change to propagate to original DF

[Finndersen]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

# Setting column data with different dtype on shallow copy causes original to be unchanged
original = pd.DataFrame({'a': ['initial string data']})
shallow = original.copy(deep=False)
shallow['a'] = 5   # Set column to integer dtype
original['a'][0]
>> 'initial string data'

# Whereas if new data has same dtype as existing, original DF is updated too
original = pd.DataFrame({'a': ['initial string data']})
shallow = original.copy(deep=False)
shallow['a'] = 'new string data'  # Set column to different str/object value
original['a'][0]
>> 'new string data'

Issue Description

Setting data of same dtype as existing column in shallow DF copy causes change to propagate to original DF. However this does not happen if new data has different dtype.

Expected Behavior

Assigning column data in DF shallow copy should not also cause change in original DF (regardless of dtype)

Installed Versions

INSTALLED VERSIONS

commit : f2c8480
python : 3.9.1.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.18362
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_Australia.1252

pandas : 1.2.3
numpy : 1.20.3
pytz : 2021.1
dateutil : 2.8.1
pip : 20.2.3
setuptools : 49.2.1
Cython : 0.29.24
pytest : 6.2.2
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.25.0
pandas_datareader: None
bs4 : 4.9.3
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.3
numexpr : None
odfpy : None
openpyxl : 3.0.6
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.54.0

[phofl]

This behaves exactly as documented:

When deep=False, a new object will be created without copying the calling object’s data or index (only references to the data and index are copied). Any changes to the data of the original will be reflected in the shallow copy (and vice versa).

[Finndersen]

I understand the intended behaviour of a shallow copy. If the underlying data of the original / shallow copy is mutated, it will be reflected in both.
This is not what is happening. In this case, the reference of a column is changed to a different pieces of data (new Series) and this change is reflected across both shallow copy and original which is not expected.
Can you explain why behavior differs if dtype of newly referenced Series is different?

[phofl]

Setitem operates inplace if dtype is the same and makes a copy if not.

There is an open pr addressing this. It should always make a copy

[Finndersen]

Yes exactly, thats the issue I'm raising
Didnt see the PR: #43406

[phofl]

Yeah but this is independent of shallow copy then. Was/is a setitem bug, but should be fixed in 1.4 then

[Finndersen]

I guess shallow copy brings the behaviour to light, otherwise with a standalone DF it is not obvious to see/know whether the setitem is changing data in place or changing reference to new data

[phofl]

You can do something like this:

df = pd.DataFrame({"a": ["test"]})
expected = df.copy()
view = df[:]
df["a"] = "new"
tm.assert_frame_equal(view, expected)

The statement df[:] creates a view which should be unchanged, but is not in these cases.