PERF: skip non-consolidatable blocks when checking consolidation

[jorisvandenbossche]

This skips the non-consolidatable blocks to determine if the blocks are consolidated. So meaning that if you have multiple EA columns with the same dtype, we do not do an unnecessary consolidation.

From investigating #32196 (comment)

@rth this should give another speed-up to your benchmark

[jorisvandenbossche]

With:

arrays = [pd.arrays.SparseArray(np.random.randint(0, 2, 1000), dtype="float64") for _ in range(10000)]
index = pd.Index(range(len(arrays[0])))  
columns = pd.Index(range(len(arrays)))    

it gives

In [4]: %timeit pd.DataFrame._from_arrays(arrays, index=index, columns=columns)  
122 ms ± 2.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)   <-- PR
249 ms ± 8.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)   <-- master

[rth]

Very nice @jorisvandenbossche , thanks!

I can confirm that when cherry-picked on top of #32825 this gives another 2x speed-up,

[100.00%] ··· sparse.SparseDataFrameConstructor.time_from_scipy                                                               115±0.5ms
       before           after         ratio
     [dbd7a5d3]       [c2476380]
     <master>         <tmp-sparse>
-       115±0.5ms       14.1±0.2ms     0.12  sparse.SparseDataFrameConstructor.time_from_scipy

So around 14x speed-up for these 3 PRs.

[jbrockmendel]

Looks like a nice speedup. Couple of questions, non-blockers:

• are the improvements specific to SparseArray?
• if we exclude non-consolidateable, can we check dtype instead of ftype?
• does caching ftype make a difference?

[jorisvandenbossche]

Not specific to SparseArray, it's for all non-consolidatable blocks (which is now just ExtensionBlock I think?)
It's just that creating a DataFrame from sparse matrix is a case where you only have extension blocks, and thus were this is relatively more costly.

I suppose that we can get rid of ftype entirely, as indeed, that was only used in the past to differentiate dense and sparse dtypes, but now sparse dtypes are ExtensionDtypes, and not numpy dtypes, so we don't need that anymore.

[jorisvandenbossche]

@jbrockmendel updated with removing ftype, thanks for the questions!

[WillAyd]

lgtm - nice cleanup

[jreback]

thanks @jorisvandenbossche

yeah we had removed ftypes in 1.0, so this was a leftover i think.