BUG-19214 int categoricals are formatted as ints #24494

JustinZhengBC · 2018-12-30T08:21:51Z

closes categorical series with null converts ints to float #19214
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This PR alters the ExtensionArrayFormatter such that if it receives a categorical series with only integers and NaN, it will print the integers as integers instead of converting to floats.

>>> # previous behaviour
>>> pd.Series([1,2, np.nan], dtype="object").astype("category")
0    1.0
1    2.0
2    NaN
dtype: category
Categories (2, int64): [1, 2]
>>> # new behaviour
>>> pd.Series([1,2, np.nan], dtype="object").astype("category")
0      1
1      2
2    NaN
dtype: category
Categories (2, int64): [1, 2]

codecov · 2018-12-30T08:57:47Z

Codecov Report

Merging #24494 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #24494      +/-   ##
==========================================
+ Coverage   92.31%   92.31%   +<.01%     
==========================================
  Files         166      166              
  Lines       52412    52414       +2     
==========================================
+ Hits        48382    48384       +2     
  Misses       4030     4030

Flag	Coverage Δ
#multiple	`90.73% <100%> (ø)`	⬆️
#single	`43.05% <50%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/io/formats/format.py	`97.98% <100%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1ebfd8a...a2593f0. Read the comment docs.

codecov · 2018-12-30T08:57:49Z

Codecov Report

Merging #24494 into master will increase coverage by 0.05%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #24494      +/-   ##
==========================================
+ Coverage   92.32%   92.38%   +0.05%     
==========================================
  Files         166      166              
  Lines       52440    52379      -61     
==========================================
- Hits        48417    48388      -29     
+ Misses       4023     3991      -32

Flag	Coverage Δ
#multiple	`90.8% <ø> (+0.05%)`	⬆️
#single	`43.01% <ø> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/arrays/categorical.py	`95.43% <ø> (-0.05%)`	⬇️
pandas/core/internals/construction.py	`95.93% <0%> (-0.73%)`	⬇️
pandas/core/nanops.py	`94.36% <0%> (-0.55%)`	⬇️
pandas/core/dtypes/concat.py	`96.6% <0%> (-0.46%)`	⬇️
pandas/core/internals/concat.py	`96.45% <0%> (-0.39%)`	⬇️
pandas/core/algorithms.py	`94.77% <0%> (-0.34%)`	⬇️
pandas/core/indexes/period.py	`92.06% <0%> (-0.32%)`	⬇️
pandas/core/indexes/timedeltas.py	`90.22% <0%> (-0.26%)`	⬇️
pandas/core/arrays/datetimes.py	`97.84% <0%> (-0.08%)`	⬇️
pandas/core/indexes/base.py	`96.29% <0%> (-0.07%)`	⬇️
... and 44 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cf92230...7cc02c1. Read the comment docs.

pandas/io/formats/format.py

pandas/tests/arrays/categorical/test_repr.py

JustinZhengBC · 2018-12-30T21:34:45Z

@jreback I looked into it and the CategoricalFormatter is used for printing categoricals like this:

[1.0, 2.0, NaN]
Categories (2, int64): [1, 2]

instead of like a series. The PR now modifies the CategoricalFormatter in addition to the ExtensionArrayFormatter.

jreback

So the fix is in .get_values() do this

n [28]: c.categories.astype(object).take(c.codes, fill_value=np.nan)
Out[28]: Index([1, 2, nan], dtype='object')

if -1 in c.codes and is_integer_dtype(c.categories)

pandas/io/formats/format.py

jreback · 2018-12-30T23:08:14Z

@TomAugspurger any objections here?

doc/source/whatsnew/v0.24.0.rst

pandas/core/arrays/categorical.py

jreback

the warnings are not needed

jreback · 2018-12-31T23:27:52Z

doc/source/whatsnew/v0.24.0.rst

@@ -1323,6 +1323,7 @@ Categorical
 - Bug in many methods of the ``.str``-accessor, which always failed on calling the ``CategoricalIndex.str`` constructor (:issue:`23555`, :issue:`23556`)
 - Bug in :meth:`Series.where` losing the categorical dtype for categorical data (:issue:`24077`)
 - Bug in :meth:`Categorical.apply` where ``NaN`` values could be handled unpredictably. They now remain unchanged (:issue:`24241`)
+- Bug in :meth:`Categorical.get_values` where integers would be formatted as floats if ``NaN`` values were present (:issue:`19214`)


this is not the user facing note though, it is formtting of a Series/Categorical where this happens. .get_values() its just an implementation detail.

@jreback I'm a bit confused as to what a user-facing note is. Is it a specific section somewhere? Or does it mean to phrase the note differently, like "bug in the Categorical repr"?

sorry. what I mean is a user would want to know that the string repr of Series (or categorical dtype) and Categoricals for integer categories will now not be coerced to float. Its maybe worth making this a sub-section (e.g. show previous / new behavior). you can just in-line it here.

jreback · 2018-12-31T23:28:13Z

pandas/core/arrays/categorical.py

@@ -1520,6 +1520,11 @@ def get_values(self):
        # if we are a datetime and period index, return Index to keep metadata
        if is_datetimelike(self.categories):
            return self.categories.take(self._codes, fill_value=np.nan)
+        elif is_integer_dtype(self.categories) and -1 in self._codes:
+            warn("Integer values represented as objects to accomodate NaNs",


no need for a warning

jreback · 2018-12-31T23:28:30Z

pandas/tests/arrays/categorical/test_repr.py

@@ -240,6 +241,16 @@ def test_categorical_repr_datetime_ordered(self):

        assert repr(c) == exp

+    @pytest.mark.filterwarnings("ignore:Integer values:RuntimeWarning")


no warnings are needed

TomAugspurger · 2019-01-01T14:17:16Z

doc/source/whatsnew/v0.24.0.rst

@@ -1653,6 +1683,7 @@ Reshaping
 - :meth:`DataFrame.nlargest` and :meth:`DataFrame.nsmallest` now returns the correct n values when keep != 'all' also when tied on the first columns (:issue:`22752`)
 - Constructing a DataFrame with an index argument that wasn't already an instance of :class:`~pandas.core.Index` was broken (:issue:`22227`).
 - Bug in :class:`DataFrame` prevented list subclasses to be used to construction (:issue:`21226`)
+- Calling :func:`pandas.concat` on a ``Categorical`` of ints with NA values now causes them to be processed as objects (formerly coerced to floats) (:issue:`19214`)


This only applies when concating a categorical with a different dtype, right? If I concat two integer cats with the same dtype, it’s still categorical right?

TomAugspurger · 2019-01-01T14:18:22Z

pandas/tests/arrays/categorical/test_repr.py

+        assert repr(c) == c_exp
+
+        s = Series([1, 2, np.nan], dtype="object").astype("category")
+        s_exp = """0      1\n1      2\n2    NaN\ndtype: category\nCategories (2, int64): [1, 2]"""  # noqa


You can use parenthesis to wrap this line.

TomAugspurger

To me, the change in dtype for concat is the larger change. Just from the release notes it looks like the formatting change is larger.

jreback · 2019-01-01T16:18:52Z

To me, the change in dtype for concat is the larger change. Just from the release notes it looks like the formatting change is larger.

I don't think this is that big of a deal, do you want a sub-section for this?

jreback · 2019-01-03T00:57:27Z

@TomAugspurger you think need to highlite the change more?

TomAugspurger · 2019-01-03T12:20:50Z

The formatting change seems fine. I'm more concerned about the breaking change to get_values with no deprecation warning. Is there any way to deprecate the behavior, or would that require to many internal changes so that the warning doesn't show up when printing the categorical?

…

On Wed, Jan 2, 2019 at 6:57 PM Jeff Reback ***@***.***> wrote: @TomAugspurger <https://github.com/TomAugspurger> you think need to highlite the change more? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#24494 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABQHIvMndIhl1YZHo1Iye0ANfYxd0P88ks5u_VV8gaJpZM4Zk2sS> .

jreback · 2019-01-04T12:20:09Z

I'm more concerned about the breaking
change to get_values
with no deprecation warning. Is there any way to deprecate the behavior, or
would that require to

This is a very small case to be honest (and our current behavior for concatting any non-same types, so this is a bug IMHO).

@JustinZhengBC can you make a small sub-section about the changes in Categorical int-like concat, showing previous and current behavior. I think that is fine.

…nto BUG-19214

jreback · 2019-01-05T15:12:48Z

@TomAugspurger better?

TomAugspurger

Just doc comments.

doc/source/whatsnew/v0.24.0.rst

jreback · 2019-01-05T21:21:14Z

@TomAugspurger ?

jreback · 2019-01-05T22:15:24Z

thanks @JustinZhengBC keep em coming!

BUG-19214 int categoricals are formatted as ints

a2593f0

jreback requested changes Dec 30, 2018

View reviewed changes

pandas/io/formats/format.py Outdated Show resolved Hide resolved

pandas/tests/arrays/categorical/test_repr.py Outdated Show resolved Hide resolved

jreback added Output-Formatting __repr__ of pandas objects, to_string Categorical Categorical Data Type labels Dec 30, 2018

BUG-19214 alter categorical as well as categorical series

36e16e1

jreback requested changes Dec 30, 2018

View reviewed changes

pandas/io/formats/format.py Outdated Show resolved Hide resolved

JustinZhengBC added 2 commits December 30, 2018 14:21

BUG-19214 change get_values instead

ee22299

fix typo in whatsnew

e6ef56e

jreback added this to the 0.24.0 milestone Dec 30, 2018

jreback requested changes Dec 30, 2018

View reviewed changes

doc/source/whatsnew/v0.24.0.rst Outdated Show resolved Hide resolved

pandas/core/arrays/categorical.py Outdated Show resolved Hide resolved

JustinZhengBC added 4 commits December 30, 2018 20:04

BUG-19214 add warning and change check order

97472b3

merge conflict

6938616

BUG-19214 fix tests

ee5ab83

Merge branch 'master' into BUG-19214

bbd2ff0

jreback requested changes Dec 31, 2018

View reviewed changes

BUG-19214 add specific section in whatsnew

2771401

TomAugspurger reviewed Jan 1, 2019

View reviewed changes

JustinZhengBC added 2 commits January 1, 2019 16:26

BUG-19214 make requested changes

2bb35dc

Merge branch 'master' into BUG-19214

f8e67f9

BUG-19214 add whatsnew section about concat

4057932

Merge branch 'BUG-19214' of https://github.com/justinzhengbc/pandas i…

7b46fec

…nto BUG-19214

jreback approved these changes Jan 5, 2019

View reviewed changes

TomAugspurger reviewed Jan 5, 2019

View reviewed changes

BUG-19214 whatsnew changes

7cc02c1

TomAugspurger approved these changes Jan 5, 2019

View reviewed changes

jreback merged commit b9d2700 into pandas-dev:master Jan 5, 2019

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG-19214 int categoricals are formatted as ints (pandas-dev#24494)

2e59512

Pingviinituutti pushed a commit to Pingviinituutti/pandas that referenced this pull request Feb 28, 2019

BUG-19214 int categoricals are formatted as ints (pandas-dev#24494)

b109cec

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG-19214 int categoricals are formatted as ints #24494

BUG-19214 int categoricals are formatted as ints #24494

JustinZhengBC commented Dec 30, 2018

codecov bot commented Dec 30, 2018

codecov bot commented Dec 30, 2018 •

edited

Loading

JustinZhengBC commented Dec 30, 2018

jreback left a comment

jreback commented Dec 30, 2018

jreback left a comment

jreback Dec 31, 2018

JustinZhengBC Dec 31, 2018

jreback Dec 31, 2018

jreback Dec 31, 2018

jreback Dec 31, 2018

TomAugspurger Jan 1, 2019 •

edited

Loading

TomAugspurger Jan 1, 2019

TomAugspurger left a comment

jreback commented Jan 1, 2019

jreback commented Jan 3, 2019

TomAugspurger commented Jan 3, 2019 via email

jreback commented Jan 4, 2019

jreback commented Jan 5, 2019

TomAugspurger left a comment

jreback commented Jan 5, 2019

jreback commented Jan 5, 2019

		@@ -240,6 +241,16 @@ def test_categorical_repr_datetime_ordered(self):

		assert repr(c) == exp

		@pytest.mark.filterwarnings("ignore:Integer values:RuntimeWarning")

BUG-19214 int categoricals are formatted as ints #24494

BUG-19214 int categoricals are formatted as ints #24494

Conversation

JustinZhengBC commented Dec 30, 2018

codecov bot commented Dec 30, 2018

Codecov Report

codecov bot commented Dec 30, 2018 • edited Loading

Codecov Report

JustinZhengBC commented Dec 30, 2018

jreback left a comment

Choose a reason for hiding this comment

jreback commented Dec 30, 2018

jreback left a comment

Choose a reason for hiding this comment

jreback Dec 31, 2018

Choose a reason for hiding this comment

JustinZhengBC Dec 31, 2018

Choose a reason for hiding this comment

jreback Dec 31, 2018

Choose a reason for hiding this comment

jreback Dec 31, 2018

Choose a reason for hiding this comment

jreback Dec 31, 2018

Choose a reason for hiding this comment

TomAugspurger Jan 1, 2019 • edited Loading

Choose a reason for hiding this comment

TomAugspurger Jan 1, 2019

Choose a reason for hiding this comment

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback commented Jan 1, 2019

jreback commented Jan 3, 2019

TomAugspurger commented Jan 3, 2019 via email

jreback commented Jan 4, 2019

jreback commented Jan 5, 2019

TomAugspurger left a comment

Choose a reason for hiding this comment

jreback commented Jan 5, 2019

jreback commented Jan 5, 2019

codecov bot commented Dec 30, 2018 •

edited

Loading

TomAugspurger Jan 1, 2019 •

edited

Loading