Skip to content

API/BUG: isna_old on EAs? #33209

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jbrockmendel opened this issue Apr 1, 2020 · 2 comments
Closed

API/BUG: isna_old on EAs? #33209

jbrockmendel opened this issue Apr 1, 2020 · 2 comments
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Milestone

Comments

@jbrockmendel
Copy link
Member

isna_new has a code path that handles EAs that isna_old doesnt, so the following will raise:

pd.set_option("mode.use_inf_as_na", True)
pd.isna(my_decimal_array)

Most of this can be fixed by adding to isna_ndarray_old:

if is_extension_array_dtype(values.dtype):
    result = values.isna()

Then we can add into the interface tests:

    def test_isna_old_extension_array(self, data_missing):
        # TODO: do we need EA.isna to handle `inf`?
        expected = data_missing.isna()
        with cf.option_context("mode.use_inf_as_na", True):
            result = pd.isna(data_missing)

        tm.assert_equal(result, expected)

The two problems here are 1) as in the comment, this may be wrong/ambiguous if the EA has a concept of "inf" and 2) we still get a test failure on StringArray([<NA>, 'A'])

@simonjayhawkins simonjayhawkins added Bug ExtensionArray Extending pandas with custom dtypes or arrays. Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Apr 20, 2020
@jreback jreback added this to the 1.1 milestone Apr 29, 2020
@dsaxton
Copy link
Member

dsaxton commented May 10, 2020

@jbrockmendel I think this is mostly fixed after #33656 (wasn't positive at the time so didn't mark it in the PR)? The "custom EA with infinity" thing may still be an open question.

@jbrockmendel
Copy link
Member Author

Great, thanks for taking point on this. I think you're right that use_inf_as_na should be deprecated longer-term too

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug ExtensionArray Extending pandas with custom dtypes or arrays. Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

No branches or pull requests

4 participants