BUG: Avoid casting Int to object in Categorical.from_codes #31794

dsaxton · 2020-02-07T21:37:33Z

closes Categorical.from_codes fails for the (new nullable) Int64 dtype #31779
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

WillAyd · 2020-02-08T00:53:43Z

pandas/core/arrays/categorical.py

@@ -644,7 +644,11 @@ def from_codes(cls, codes, categories=None, ordered=None, dtype=None):
            )
            raise ValueError(msg)

-        codes = np.asarray(codes)  # #21767
+        if is_extension_array_dtype(codes) and is_integer_dtype(codes):


Hmm does this universally work if we just call to_numpy? I suppose could copy where undesired but maybe better than if...else branching

Yeah the if / else isn't too pretty but I think we have to worry if codes is a numpy array already or possibly a bare list. Ideally it'd be nice to just do pd.array(codes) instead of np.asarray(codes), but it looks like min and max haven't been implemented yet for IntegerArray so we get an error soon after.

In principle we could do a pd.array(codes).to_numpy("int64")
But a problem with that is that it will convert an already integer array of potentially lower bitsize than int64

Has numpy a concept of "any integer dtype"?
Because then we could also simply do codes = np.asarray(codes, dtype="any int")

Another option could be this (to avoid converting an already int ndarray to int64)::

if not isinstance(codes, np.ndarray): codes = np.asarray(codes, dtype="int64")

which seems simpler as what you have now

Another option could be this (to avoid converting an already int ndarray to int64)::

if not isinstance(codes, np.ndarray): codes = np.asarray(codes, dtype="int64")

which seems simpler as what you have now

I think we want it to raise if codes is something like [0.0, 1.0, 2.0] but this would make it a valid input

I think we want it to raise if codes is something like [0.0, 1.0, 2.0] but this would make it a valid input

Ah, yes indeed...

(we should probably try to gather some of those situations we run into handling array-like input including nullable dtypes, to see if we can extract some useful helpers for those cases)

WillAyd · 2020-02-08T00:53:56Z

Can you merge master? CI failure should be resolved on master

jorisvandenbossche · 2020-02-08T10:43:48Z

pandas/core/arrays/categorical.py

@@ -644,7 +644,11 @@ def from_codes(cls, codes, categories=None, ordered=None, dtype=None):
            )
            raise ValueError(msg)

-        codes = np.asarray(codes)  # #21767
+        if is_extension_array_dtype(codes) and is_integer_dtype(codes):


In principle we could do a pd.array(codes).to_numpy("int64")
But a problem with that is that it will convert an already integer array of potentially lower bitsize than int64

jorisvandenbossche · 2020-02-08T10:44:17Z

pandas/core/arrays/categorical.py

-        codes = np.asarray(codes)  # #21767
+        if is_extension_array_dtype(codes) and is_integer_dtype(codes):
+            # Avoid the implicit conversion of Int to object
+            codes = codes.to_numpy(dtype=np.int64, na_value=np.nan)


Codes shouldn't contain missing values (and the na_value=np.nan will not work anyway for a int64 dtype)

pandas/tests/arrays/categorical/test_constructors.py

jorisvandenbossche · 2020-02-08T10:48:44Z

pandas/core/arrays/categorical.py

@@ -644,7 +644,11 @@ def from_codes(cls, codes, categories=None, ordered=None, dtype=None):
            )
            raise ValueError(msg)

-        codes = np.asarray(codes)  # #21767
+        if is_extension_array_dtype(codes) and is_integer_dtype(codes):


Has numpy a concept of "any integer dtype"?
Because then we could also simply do codes = np.asarray(codes, dtype="any int")

Another option could be this (to avoid converting an already int ndarray to int64)::

if not isinstance(codes, np.ndarray): codes = np.asarray(codes, dtype="int64")

which seems simpler as what you have now

Co-Authored-By: Joris Van den Bossche <[email protected]>

jreback · 2020-02-09T17:38:43Z

pandas/core/arrays/categorical.py

+            # Avoid the implicit conversion of Int to object
+            codes = codes.to_numpy(dtype=np.int64)
+        else:
+            codes = np.asarray(codes)  # #21767


you don't need to add the issue number in code generally, unless you add a full expl (so here just remove i)

jreback · 2020-02-09T17:39:41Z

pandas/tests/arrays/categorical/test_constructors.py

@@ -560,6 +560,15 @@ def test_from_codes_neither(self):
        with pytest.raises(ValueError, match=msg):
            Categorical.from_codes([0, 1])

+    def test_from_codes_with_nullable_int(self):


can you also test with a NaN in the codes and Int64 (should raise)

Done. The error message that ends up getting raised is maybe not the best: ValueError: cannot convert to '<class 'numpy.int64'>'-dtype NumPy array with missing values. Specify an appropriate 'na_value' for this dtype.. Would you recommend creating one specific to this method when we have NA values?

Yes, I think we should probably catch the error to raise a better one (because the error message now would be really confusing for users)

jreback · 2020-02-12T12:36:55Z

thanks @dsaxton

jorisvandenbossche · 2020-02-12T12:41:19Z

@meeseeksdev backport to 1.0.x

…egorical.from_codes

…rom_codes (#31921) Co-authored-by: Daniel Saxton <[email protected]>

Daniel Saxton added 3 commits February 7, 2020 15:28

Add test

416a869

Don't cast EA to object

4bd7588

Add whatsnew

6c2edda

WillAyd reviewed Feb 8, 2020

View reviewed changes

WillAyd added the Categorical Categorical Data Type label Feb 8, 2020

Merge branch 'master' into cat-from-int-codes

916998f

jorisvandenbossche reviewed Feb 8, 2020

View reviewed changes

dsaxton and others added 3 commits February 8, 2020 08:52

Update pandas/tests/arrays/categorical/test_constructors.py

270199a

Co-Authored-By: Joris Van den Bossche <[email protected]>

No na_value

75e25d6

Merge branch 'master' into cat-from-int-codes

acbadaa

jreback requested changes Feb 9, 2020

View reviewed changes

jreback added this to the 1.0.2 milestone Feb 9, 2020

jreback added the Regression Functionality that used to work in a prior pandas version label Feb 9, 2020

Daniel Saxton added 2 commits February 9, 2020 11:46

Remove issue number comment

6ec804e

Add test with NA code

c5508dc

jorisvandenbossche removed the Regression Functionality that used to work in a prior pandas version label Feb 11, 2020

Daniel Saxton added 2 commits February 11, 2020 13:42

Raise a better error message

63b7c1c

Merge branch 'master' into cat-from-int-codes

e9edcfb

jorisvandenbossche approved these changes Feb 11, 2020

View reviewed changes

jreback added the Constructors Series/DataFrame/Index/pd.array Constructors label Feb 12, 2020

jreback approved these changes Feb 12, 2020

View reviewed changes

jreback merged commit 16684f2 into pandas-dev:master Feb 12, 2020

jreback added the Still Needs Manual Backport label Feb 12, 2020

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Feb 12, 2020

Backport PR pandas-dev#31794: BUG: Avoid casting Int to object in Cat…

719c85a

…egorical.from_codes

meeseeksmachine mentioned this pull request Feb 12, 2020

Backport PR #31794 on branch 1.0.x (BUG: Avoid casting Int to object in Categorical.from_codes) #31921

Merged

jreback removed the Still Needs Manual Backport label Feb 12, 2020

jorisvandenbossche pushed a commit that referenced this pull request Feb 12, 2020

Backport PR #31794: BUG: Avoid casting Int to object in Categorical.f…

affd898

…rom_codes (#31921) Co-authored-by: Daniel Saxton <[email protected]>

dsaxton deleted the cat-from-int-codes branch February 12, 2020 15:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Avoid casting Int to object in Categorical.from_codes #31794

BUG: Avoid casting Int to object in Categorical.from_codes #31794

dsaxton commented Feb 7, 2020

WillAyd Feb 8, 2020

dsaxton Feb 8, 2020

jorisvandenbossche Feb 8, 2020

jorisvandenbossche Feb 8, 2020

dsaxton Feb 8, 2020

jorisvandenbossche Feb 11, 2020

WillAyd commented Feb 8, 2020

jorisvandenbossche Feb 8, 2020

jorisvandenbossche Feb 8, 2020

jorisvandenbossche Feb 8, 2020

jreback Feb 9, 2020

jreback Feb 9, 2020

dsaxton Feb 9, 2020

jorisvandenbossche Feb 11, 2020

jreback commented Feb 12, 2020

jorisvandenbossche commented Feb 12, 2020

BUG: Avoid casting Int to object in Categorical.from_codes #31794

BUG: Avoid casting Int to object in Categorical.from_codes #31794

Conversation

dsaxton commented Feb 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WillAyd commented Feb 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 12, 2020

jorisvandenbossche commented Feb 12, 2020