PERF: reindex default fill_value dtype

[lukemanley]

• closes PERF: reindex unnecessarily introduces block with new dtype, preventing consolidation #45857
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.

Default fill_value (np.nan) to match interleaved_dtype when possible to avoid upcasting and allow for block consolidation.

Using the example from the OP:

import pandas as pd
import numpy as np

df = pd.concat([
    pd.DataFrame(np.zeros((1000, 1000), dtype='f4')),
], axis=1).reindex(columns=np.arange(5, 1005))
print(df._data.nblocks) 

df.values
print(df._data.nblocks)  # <- now consolidating

%timeit df.values

main:

2
2
789 µs ± 135 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops 

PR:

2
1
2.01 µs ± 48.8 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops

[lukemanley]

@jbrockmendel - any suggestions here? The test added in this PR is failing for certain builds. Unfortunately, I'm unable to reproduce the failures locally so I'm not sure where to start.

[jbrockmendel]

@lukemanley do you happen to be using a windows machine or 32bit build locally?

[jbrockmendel]

is there a risk of getting e.g. Float64Dtype here?

[lukemanley]

I don't think there is a risk in a DataFrame of only float32 values.

[jbrockmendel]

right, im talking about a case where the DataFrame isnt all-float32, but includes at least one Float64Dtype

[jbrockmendel]

There's a comment in NDFrame._reindex_with_indexers # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi) that i think might be useful in these cases. Could that be helpful?

[lukemanley]

do you happen to be using a windows machine or 32bit build locally?

Nope. I'm unable to reproduce those errors on mac or 64bit windows.

There's a comment in NDFrame._reindex_with_indexers # TODO: speed up on homogeneous DataFrame objects (see _reindex_multi) that i think might be useful in these cases. Could that be helpful?

I'll take a look, thanks.

[lukemanley]

I’m going to close this PR. I cannot reproduce the failures locally. I tried a few different things but was unsuccessful in identifying where the float64 dtype comes from. I do not see that behavior locally.