Categorical[idx].codes != Categorical.codes[idx]

[jseabold]

I'm unable to isolate this, but maybe this rings a bell for someone. It took me a long time to find the source of error in my code. I don't think this is intentional, and if so, is pretty dangerous.

[~/]
[22]: y[idx].codes[:15]
[22]: array([4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 1, 4], dtype=int8)

[~/]
[23]: y.codes[idx][:15]
[23]: array([4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4, 4], dtype=int8)

[~/]
[24]: idx[:15]
[24]:
array([ 74172,  53885, 128953,  28125, 136548, 136700,  93633,  61147,
        56535,  90577, 115719,  58038, 111711,  53399,  77475])

[~/]
[25]: type(y)
[25]: pandas.core.categorical.Categorical

[~/]
[27]: pd.version.version
[27]: '0.15.2'

[jseabold]

It seems that Categorical._maybe_coerce_indexer is transforming the int64 index to int8.

[jreback]

The indexer is coerced to the same dtype as the codes (int8) in this case. This is necessary to have the indexing work properly (otherwise numpy complains). But I think if the indexer is out of bounds it will wrap around which is prob not what is expected.

can you show a construction for y?

[jreback]

needless to say this is prob not tested well. and prob not used very much as this is almost always used from a Series. (this is somewhat legacy code)

[jseabold]

I'm trying to make a MWE.

[jseabold]

np.random.seed(1)
c = pd.Categorical(np.random.randint(0, 5, size=150000).astype(np.int8))
c.codes[np.array([100000]).astype(np.int64)]
c[np.array([100000]).astype(np.int64)].codes

[jseabold]

In my case, y was created from some real data with 5 categories and the idx was created from sklearn cross-validation functions. Given the size of my y, int64s.

[jseabold]

Not obvious to me why numpy would care between dtype of indexer and the dtype of the array. AFAIK, it just needs to be any integer type or boolean, no?

[jseabold]

Removing the casting everything seems to work fine for me.

[4]: np.version.version
[4]: '1.10.0.dev+4ed1587'

[jreback]

fixed in #9470 (just had to remove coercion), but didn't work for both setting and getting.

IIRC this was a problem on windows as it complained when trying to index (but maybe I am remembering incorrectly). I agree it should work.

If you can test on windows would be great, otherwise I can do next week.

[jseabold]

I'll see if I can't find a windows box around. If not, can look in to it one night. Any min. numpy version?

[jreback]

numpy 1.7 is our min as of 0.15.0

[jorisvandenbossche]

I tested it on Windows, and at least with Windows 7, 64bit, python 2.7 and numpy 1.9 this works (and the bug is fixed)

[jseabold]

Fantastic. Thanks @jorisvandenbossche

Is this worth a 0.15.3? This is pretty serious IMO.

[jreback]

well considering it wasn't noticed till 3 mo after 0,15.2 no

0.16.0 is schedule somewhat soon anyhow

[jseabold]

Well, I lost about 2 weeks of work bc. I was silently working with garbage.