REF: avoid broadcasting of Series to DataFrame in ops for ArrayManager

[jorisvandenbossche]

xref #39772

Currently, we broadcast a Series to a DataFrame for elementwise operations in the "op(df, series)" case. This is done (I think) to be able to perform the operation block-wise (which is only implemented for block/block case, and not for block/array).
But, when using the ArrayManager, this broadcasting is 1) not necessary (since we perform column-by-column anyway) and 2) costly.

[jbrockmendel]

BTW i think the array manager build can be simplified with something like pytest pandas/tests/[abcdfglpstuw]*. Getting close!

[jorisvandenbossche]

I am running into the following issue:

In [10]: np.array([1, 2]) * pd.NaT
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-0274fc287e43> in <module>
----> 1 np.array([1, 2]) * pd.NaT

TypeError: unsupported operand type(s) for *: 'numpy.ndarray' and 'NaTType'

while

In [12]: np.array([1, 2]) * np.timedelta64("NaT", "ns")
Out[12]: array(['NaT', 'NaT'], dtype='timedelta64[ns]')

This happens if you do eg df * ser where the DataFrame has numerical columns, and the Series has timedelta64 dtype with NaTs.
I suppose this will also be a problem for the untyped pd.NA

(from pandas/tests/frame/test_arithmetic.py::TestFrameArithmetic::test_td64_op_nat_casting)

[jbrockmendel]

1. this is AM-only right?
2. wont the array_op call just re-wrap this?
3. if not, will something like DataFrame[int] + Series[timedelta64ns] fail to raise because of this?

[jbrockmendel]

4. if this is a hack as the commit message suggests, then a comment would be worthwhile

[jorisvandenbossche]

See #40482 (comment) for the general explanation of in which case this happens. It was not my intention to keep this, I am hoping to find a better solution.

[jbrockmendel]

Yah I saw that, thought about commenting, but decided that "last time i handled that by reshaping to 2D" wouldn't be something you'd find that helpful.

[jbrockmendel]

this is just an optimization, independent of the broadcasting thing?

[jbrockmendel]

if we're doing this both here and on L293, then might as well do it just once after L289

Do we have cases that get here with x not object dtype?

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[mroeschke]

Appears this PR has been dormant for a while so closing. Feel free to reopen when you have time to work on this PR.

[jreback]

@jorisvandenbossche if you can rebase

[simonjayhawkins]

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.