BUG: Categorical.remove_categories(np.nan) fails when underlying dtype is float

[evanpw]

Fixes GH #10156. This also makes different null values indistinguishable inside of remove_categories, but they're already indistinguishable in most other contexts:

>>> pd.Categorical([], categories=[np.nan, None])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/categorical.py", line 289, in __init__
    categories = self._validate_categories(categories)
  File "pandas/core/categorical.py", line 447, in _validate_categories
    raise ValueError('Categorical categories must be unique')
ValueError: Categorical categories must be unique

[jreback]

can you add a test for this as well >>> pd.Categorical([], categories=[np.nan, None])
also add in using None in the .remove_categories. Pls test on an object dtype as well as floating-point.
Bonus points if you can make this work for datetimelike (e.g. using pd.NaT.

[jreback]

can you update

[evanpw]

I think these tests are what you were asking for. Let me know if otherwise.

[jreback]

yep, ping when green.

[evanpw]

It turns out that this already works for datetimelike categoricals. I added some tests and squashed.

[jreback]

awesome @evanpw ! keep em coming

[wikiped]

Thanks very much for fixing this. I am not sure if I got this right but would not it make sense to reorder the code slightly to avoid trying removing categories twice when nan is present? So use this code:

    # GH 10156
    if any(isnull(removals)):
        not_included = [x for x in not_included if notnull(x)]
        new_categories = [x for x in new_categories if notnull(x)]
    else:
        removal_set = set(list(removals))
        not_included = removal_set - set(self._categories)
        new_categories = [ c for c in self._categories if c not in removal_set ]

Instead of this:

    removal_set = set(list(removals))
    not_included = removal_set - set(self._categories)
    new_categories = [ c for c in self._categories if c not in removal_set ]

    # GH 10156
    if any(isnull(removals)):
        not_included = [x for x in not_included if notnull(x)]
        new_categories = [x for x in new_categories if notnull(x)]

[evanpw]

In your if branch, what's the original definition of not_included and new_categories? It looks like they're defined in terms of themselves.

[wikiped]

Fair point ;)
Thanks again for the fix.