REGR: DataFrame.transpose resulting in not contiguous data on nullable EAs

[rhshadrach]

• closes BUG: "ValueError: ndarray is not C-contiguous" when aggregating concatenated dataframe #57315 (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[phofl]

    masked_arrays = list(masked_arrays)
    dtype = masked_arrays[0].dtype

    values = [arr._data.reshape(1, -1) for arr in masked_arrays]
    transposed_values = np.concatenate(values, axis=0, out=np.empty((len(masked_arrays), len(masked_arrays[0])), order="F", dtype=dtype.numpy_dtype))

    masks = [arr._mask.reshape(1, -1) for arr in masked_arrays]
    transposed_masks = np.concatenate(masks, axis=0, out=np.empty_like(transposed_values, dtype=np.bool))

    arr_type = dtype.construct_array_type()
    transposed_arrays: list[BaseMaskedArray] = []
    for i in range(transposed_values.shape[1]):
        transposed_arr = arr_type(
            transposed_values[:, i], mask=transposed_masks[:, i]
        )
        transposed_arrays.append(transposed_arr)

    return transposed_arrays

this should be faster (avoids the second copy, but not sure if I can come up with a meaningful benchmark...)

arr = [
    pd.array(np.random.randint(1, 1_000_000, (100, )), dtype="Int64"),
    pd.array(np.random.randint(1, 1_000_000, (100, )), dtype="Int64"),
    pd.array(np.random.randint(1, 1_000_000, (100, )), dtype="Int64"),
    pd.array(np.random.randint(1, 1_000_000, (100, )), dtype="Int64"),
]

arr = arr * 10_000

transpose_homogeneous_masked_arrays(arr)

[rhshadrach]

Nice!

size = 100_000
df = pd.DataFrame(
    {
        "a": pd.array([0]*size, dtype="Int8"),
        "b": pd.array([0]*size, dtype="Int8"),
    }
)
%timeit df.T

# main
416 ms ± 1.43 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# orig PR
477 ms ± 4.38 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

# phofl
417 ms ± 1.98 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

[phofl]

Ah nice that we keep consistent on the Python dominated benchmark as well

[phofl]

thx @rhshadrach

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

git checkout 2.2.x
git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

git cherry-pick -x -m1 abc3efb63f02814047a1b291ac2b1ee3cd70252f

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

git commit -am 'Backport PR #57474: REGR: DataFrame.transpose resulting in not contiguous data on nullable EAs'

4. Push to a named branch:

git push YOURFORK 2.2.x:auto-backport-of-pr-57474-on-2.2.x

5. Create a PR against branch 2.2.x, I would have named this PR:

"Backport PR #57474 on branch 2.2.x (REGR: DataFrame.transpose resulting in not contiguous data on nullable EAs)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the Still Needs Manual Backport label once the PR gets merged.

If these instructions are inaccurate, feel free to suggest an improvement.