PERF: avoid copies in lib.infer_dtype

[jbrockmendel]

• closes API: infer_dtype with skipna=True only skip valid-for-dtype NAs #45022
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

In the skipna=True case, instead of doing values = values[~isnaobj(values)], just pass skipna to the relevant validator functions. This causes trouble because those validator functions have different rules about what NA values they allow (analogous to is_valid_na_for_dtype).

I'm down to 1 test case failing locally, fixing it will require changing the StringArray constructor to allow None and np.nan (xref #40839) in addition to just pd.NA (AFAICT pd.array and pd.Series(... dtype="string") will not be affected). I'm fine with that, but someone more involved with the string code might want to weigh in cc @simonjayhawkins

In [1]: import numpy as np

In [2]: arr = np.arange(10**5).astype(object)

In [3]: from pandas._libs import lib

In [4]: %timeit lib.infer_dtype(arr, skipna=True)
962 µs ± 77 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
3.81 ms ± 33 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- master


In [7]: %load_ext memory_profiler

In [16]: arr = arr.repeat(100)

In [18]: %memit lib.infer_dtype(arr, skipna=True)
peak memory: 169.24 MiB, increment: 0.00 MiB  # <- PR
peak memory: 265.39 MiB, increment: 95.38 MiB  # <- master

[jbrockmendel]

Bad news @mroeschke:

    def test_dialect_conflict_except_delimiter(all_parsers, custom_dialect, arg, value):
[...]
E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed file <_io.BufferedRandom name=10>'), 'D:\\a\\1\\s\\pandas\\core\\indexes\\base.py', 665)]

[mroeschke]

Bad news @mroeschke:

    def test_dialect_conflict_except_delimiter(all_parsers, custom_dialect, arg, value):
[...]
E           AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed file <_io.BufferedRandom name=10>'), 'D:\\a\\1\\s\\pandas\\core\\indexes\\base.py', 665)]

:(

This looks like it occurred on Windows? The check I added doesn't work there since idk the equivalent lsof check.

[jreback]

wow, looks pretty good.

[lithomas1]

Quick question:
If I do pd.arrays.StringArray(np.array(["a", np.nan], dtype=object)), won't this result in a StringArray with np.nan in it? (something like this? #30966 (comment))

[jbrockmendel]

won't this result in a StringArray with np.nan in it?

yep. probably ought to do something about that. have something min mind?

[lithomas1]

I'm trying to fix this in #41412. This PR is probably stuck in the meantime.

[jbrockmendel]

@lithomas1 is this un-blocked?

[jreback]

this WIP?

[jbrockmendel]

this WIP?

@lithomas1 says this needs to wait on #41412

[jreback]

nice!