COMPAT: add back block manager constructors to pandas.core.internals API

[jorisvandenbossche]

Those were removed in a for the rest unrelated PR (#33100, cc @jbrockmendel)

But at least create_block_manager_from_blocks is used by dask (in partd: https://github.com/dask/partd/blob/master/partd/pandas.py), so dask's tests are failing with pandas master (cc @TomAugspurger dask isn't testing with pandas master?). But so to be safe just added back both.

[TomAugspurger]

Dask does test against pandas master, but it's allowed to fail. Looking into it this week.

…

On Thu, Apr 30, 2020 at 3:44 AM Joris Van den Bossche < ***@***.***> wrote: Those were removed in a for the rest unrelated PR (#33100 <#33100>, cc @jbrockmendel <https://github.com/jbrockmendel>) But at least create_block_manager_from_blocks is used by dask (in partd: https://github.com/dask/partd/blob/master/partd/pandas.py), so dask's tests are failing with pandas master (cc @TomAugspurger <https://github.com/TomAugspurger> dask isn't testing with pandas master?). But so to be safe just added back both. ------------------------------ You can view, comment on, or merge this pull request online at: #33892 Commit Summary - COMPAT: add back block manager constructors to pandas.core.internals API File Changes - *M* pandas/core/internals/__init__.py <https://github.com/pandas-dev/pandas/pull/33892/files#diff-cf58b8e0abc60541619ae243fe0ff2ce> (9) Patch Links: - https://github.com/pandas-dev/pandas/pull/33892.patch - https://github.com/pandas-dev/pandas/pull/33892.diff — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#33892>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOITFMAAACTU7FSHIV2LRPE26JANCNFSM4MVJZ63Q> .

[jorisvandenbossche]

I only catched it in the parquet tests, so the pandas-master build might need to have pyarrow/fastparquet installed to catch it

[jbrockmendel]

Is weaning downstream packages off of these a viable option?

[jbrockmendel]

can you add a comment about why these are needed here

[jorisvandenbossche]

Is weaning downstream packages off of these a viable option?

That is certainly something to look at, but apart from that, we still need to add it back for compat with released versions IMO (and we could also deprecate it first before removing again)

[jbrockmendel]

CI fails unrelated; LGTM

[TomAugspurger]

It'd be a bit hard for partd to move off this. The purpose is to (de)serialize a DataFrame as quickly as possible, which essentially requires working at the block level.

[jorisvandenbossche]

It'd be a bit hard for partd to move off this. The purpose is to (de)serialize a DataFrame as quickly as possible, which essentially requires working at the block level.

Yes, that's exactly the same reason that pyarrow also interacts with the BlockManager.

Looking at pyarrow, there we are using the main constructor: BlockManager(reconstructed_blocks, axes), which seems basically what create_block_manager_from_blocks does (the function additionally calls _consolidate_inplace()).
But, I am personally not sure it is necessarily "better" to use BlockManager(..) over create_block_manager_from_blocks(..). Although in theory it would be nice that all external projects use the same method to create BlockManager.

[TomAugspurger]

If we want, we could create a restricted "publich-ish but not necessarily stable" dev API for projects to interact with the block manager. That way we at least know what projects are using.

Dunno if that's worth the effort though. The current setup of downstream projects testing against master seems to work out OK.

[jorisvandenbossche]

For me that "dev API" to interact with blocks is basically what is exposed in pandas.core.internals now? (maybe with the exception of concatenate_block_managers)

[TomAugspurger]

Haven't really looked, but that sounds fine :)

[jbrockmendel]

@jorisvandenbossche before i make a PR removing _safe_reshape from internals.__init__, are you aware of any downstream packages that need it there?

[jorisvandenbossche]

I am not aware of any _-prefixed functions being used

[jbrockmendel]

@phofl does dask still need create_block_manager_from_blocks?

[phofl]

yeah that is still in use