BUG: transpose not respecting CoW

[phofl]

• xref ENH / CoW: Use the "lazy copy" (with Copy-on-Write) optimization in more methods where appropriate #49473 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

This is the same solution as in iterrows, I work on a solution on the BlockManager level now

[phofl]

Updated for new block manager method

[jorisvandenbossche]

@phofl there are some relevant failures here with the usage of add_reference in the CoW build and a failure in the ArrayManager build

[phofl]

Is it possible to get views when splitting the block and converting the dtype? We got the errors when converting object to a proper dtype which splitted the block

[jorisvandenbossche]

That sounds impossible, unless one of the split blocks kept its object dtype?

[phofl]

Ok then my train of though was correct. No there is no reason to split and keep object

[jorisvandenbossche]

I also don't understand how the case in transpose can ever run into this? Because we should only get here in case of self._can_fast_transpose, and then by definition there should only be 1 block?

[phofl]

df = DataFrame({"a": [pd.Timedelta("1 days"), pd.NaT, pd.Timedelta("2 days")]}, dtype="object")

df.T

the constructor infers this to date time after transposing the array. We get 1 Timedelta block back and another NaT block We could consider this a bug though...

[jorisvandenbossche]

Ah, yes, I seem to recall that we have had discussions about this corner case before (IMO we should indeed change that)

I tried locally with object dtype with ints to test, but so this corner case is only for time like, I suppose.

[phofl]

Yeah we are calling maybe_infer_datetimelike under the hood which causes this.

[phofl]

Should we deprecate this first?

[jorisvandenbossche]

I seem to recall that we have had discussions about this corner case before (IMO we should indeed change that)

And note that this is not only about transpose, it's about constructors in general. So the question is also if we want to change this:

In [10]: df = DataFrame({"a": np.array([pd.Timedelta("1 days"), pd.NaT, pd.Timedelta("2 days")], dtype=object)})

In [11]: df
Out[11]: 
       a
0 1 days
1    NaT
2 2 days

In [12]: df.dtypes
Out[12]: 
a    timedelta64[ns]
dtype: object

(but that's for another issue)

Although of course we could fix this locally in transpose to specifically fix transpose to preserve the dtype.

[phofl]

I think the constructor would have to be deprecated. Transpose is more of a bug fix imo because that we are using the inference rules of the constructor under the hood is more an implementation detail.

[jorisvandenbossche]

One related issue: #39117

It's also not clear we necessarily want to deprecate this for the constructors. But for transpose() I would certainly consider it a bug / something to change. And for transpose() I am not sure a deprecation is needed

[phofl]

Yeah constructors makes a bit of sense.

I'll try what's failing locally when changing this.

[phofl]

We have tests covering exactly this behavior :(

FAILED pandas/tests/frame/methods/test_transpose.py::TestTranspose::test_transpose_tzaware_2col_mixed_tz - AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="A") are different
FAILED pandas/tests/frame/methods/test_transpose.py::TestTranspose::test_transpose_object_to_tzaware_mixed_tz - assert False

#26285

So lets merge this without tackling this case

[phofl]

Merging, I'll tackle the transpose case separately