REF: hold PeriodArray in NDArrayBackedExtensionBlock

[jbrockmendel]

• closes #xxxx
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

After this, Categorical will be the only NDArrayBackedExtensionArray not held in NDarrayBackedExtensionBlock.

(this also fixes a Series.where bug for PeriodDtype, the general fix (and tests) are blocked by #44514)

[jreback]

these warnings tested?

[jbrockmendel]

not the new one; need to revert it since ive xfailed the cases that get here

[jorisvandenbossche]

@jbrockmendel what's the direct advantage of using NDArrayBackedExtensionBlock instead of ExtensionBlock for PeriodArray?

Because this broke the pyarrow integration for period dtype up to pyarrow 3.0 (released just less than a year ago)

[jbrockmendel]

what's the direct advantage of using NDArrayBackedExtensionBlock instead of ExtensionBlock for PeriodArray?

2D support!

[jorisvandenbossche]

And can you do that without breaking pyarrow <= 3.0?

[jbrockmendel]

And can you do that without breaking pyarrow <= 3.0?

I dont know what private APIs it uses. Might be able to put in a shim.

[jorisvandenbossche]

They are already using the pd.core.internals.make_block function, but so in older versions pyarrow is also passing the block class (https://github.com/apache/arrow/blob/d613aa68789288d3503dfbd8376a41f2d28b6c9d/python/pyarrow/pandas_compat.py#L752-L753):

Letting it infer it works fine (what happens on the latest pyarrow releases):

In [10]: pd.core.internals.make_block(arr, placement=slice(0, 1, 1))
Out[10]: NDArrayBackedExtensionBlock: slice(0, 1, 1), 1 x 3, dtype: period[D]

but explicitly asking for an ExtensionBlock gives a mangled block with a 2D array under the hood (and for that reason it thinks it is shape (1, 1), because the underlying array is (1, 3)):

In [11]: pd.core.internals.make_block(
    ...:     arr, placement=slice(0, 1, 1), klass=pd.core.internals.ExtensionBlock
    ...: )
Out[11]: ExtensionBlock: slice(0, 1, 1), 1 x 1, dtype: period[D]

In [12]: _.shape
Out[12]: (1, 1)

[jbrockmendel]

but explicitly asking for an ExtensionBlock gives a mangled block

yah it shouldn't do that. we could put a shim in ExtensionBlock.__init__ to check for that and raise, maybe telling users to upgrade pyarrow.

[jorisvandenbossche]

And it already doesn't do that anymore.

we could put a shim in ExtensionBlock.init to check for that and raise, maybe telling users to upgrade pyarrow.

Can't we handle that inside make_block? If you pass a period array, and specify ExtensionBlock, we can return the proper block instead?

[jbrockmendel]

Can't we handle that inside make_block? If you pass a period array, and specify ExtensionBlock, we can return the proper block instead?

makes sense