REGR: Categorical with np.str_ categories

[jbrockmendel]

• closes Pandas 1.0 no longer handles numpy.str_s as catgories #31499
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[TomAugspurger]

LGTM. Is this something we want to support long-term?

[jreback]

actually, I think we should be santizing these inputs on Index construction; we already do this for Series IIRC (deep in block manager i think). Can you see if that's better (as its a more maintainable long term soln). Not averse to your change, but that's after we already saved the inputs.

[jreback]

i would prefer to actually fix this rather than work around np.str_ which is not used anywhere else.

[TomAugspurger]

What's the proposed fix? Convert these to regular strings early on?

IMO, we should deprecate the old behavior first if it isn't too costly. This PR doesn't seem too bad, though I don't really understand all its implications.

[jorisvandenbossche]

I think we should be santizing these inputs on Index construction; we already do this for Series IIRC

I thought as well, but that doesn't seem to be the case:

In [1]: s = pd.Series([np.str_("0"), np.str_("1")])  

In [2]: s[0]        
Out[2]: '0'

In [3]: type(s[0])  
Out[3]: numpy.str_

I would keep a possible sanitation for a separate issue / discussion. This PRs seems an easy fix to address the regression.

[jreback]

this is just an unsupportable bandaid. this is handling a leak of np.str_ into the internals, which is really bad. I don't think its worth trying to fix this for 1.0.1 like this, rather address a systematic real fix.

The reason this is unsupportable is that now this hides this issue in one particular place, rather than actually handling it (by converting np.str_ to str on construction).

[jorisvandenbossche]

this is just an unsupportable bandaid. this is handling a leak of np.str_ into the internals, which is really bad.

It is what we did before for years. So I don't think it is that unsupportable.

Since we can still store numpy strings in a Series, and since we supported converting those to a Categorical before, I think this is a good fix.

For 1.1, we can discuss further if we want to keep supporting this, or want to deprecate it, or want to sanitize on input (to avoid needing to support it).

[TomAugspurger]

For 1.1, we can discuss further if we want to keep supporting this, or want to deprecate it, or want to sanitize on input (to avoid needing to support it).

Agreed.

[jreback]

these last minute patches are just causing more and more issues. do what you will this.

[TomAugspurger]

@jbrockmendel are you aware of any maintenance burdens or ambiguities this might cause?

At least on 0.25.3, we seem to treat str and np.str_ equivalently in methods like get_loc and unique

[jbrockmendel]

are you aware of any maintenance burdens or ambiguities this might cause?

Not especially. If we want to root np.str_ out entirely, that would be a pretty big endeavor.

[TomAugspurger]

OK. Naively, I agree that rooting out np.str_ objects in our constructors sounds difficult. At the least, it would require a scan over the values and a couple isinstance checks on each value, which I'd like to avoid if it's not causing problems elsewhere. I'm sure we'll get reports if it is.

[TomAugspurger]

Thanks @jbrockmendel.