BUG: repr of Categorical does not distinguish int and str. #34222

MarcoGorelli · 2020-05-17T10:42:59Z

closes BUG: repr of Categorical does not distinguish int and str. #33676
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

simonjayhawkins · 2020-05-17T15:44:08Z

pandas/io/formats/format.py

@@ -1237,6 +1237,8 @@ def _format(x):
                fmt_values.append(f" {_format(v)}")
            elif is_float_type[i]:
                fmt_values.append(float_format(v))
+            elif isinstance(v, str):


GenericArrayFormatter has a quoting parameter. format_array does not have this parameter to be able to pass it on.

_repr_categories in pandas\core\arrays\categorical.py uses format_array.

is it feasible to add quoting parameter to format_array instead of changing this?

@simonjayhawkins thanks for the suggestion!

So, in _repr_categories from pandas\core\arrays\categorical.py, format_array is called. This could be changed to, say,

category_strs = fmt.format_array(self.categories, None, quoting=csv.QUOTE_NONNUMERIC)

and then, inside format_array, pass quoting on to fmt_klass, so that (here) GenericArrayFormatter would be initialised with it.

It's unclear to me me what GenericArrayFormatter currently does with quoting, I don't see it being used. Should there be no need to change anything here in _format_strings?

It's unclear to me me what GenericArrayFormatter currently does with quoting, I don't see it being used. Should there be no need to change anything here in _format_strings?

that does seem strange, I must have assumed wrongly where the quoting parameter is used. will look again, but your suggestions sgtm.

instead of this you need to adjust _format, L1221; you maybe able to simply adjust the args to pprint_thing which can quote as needed.

Yes, thanks for pointing me towards that, that was useful!

So, pprint_thing takes a quote_string argument, which is boolean, while quoting is Optional[int], so I've tried doing this by setting

quote_strings = self.quoting is not None and self.quoting != QUOTE_NONE

and then passing quote_strings to pprint_thing

pandas/tests/arrays/categorical/test_repr.py

jorisvandenbossche · 2020-05-18T07:07:25Z

Does this have impact on the Series repr, or is only for the Categorical (the array) repr?

MarcoGorelli · 2020-05-18T07:21:59Z

Should only be for the categorical repr, will update accordingly

simonjayhawkins

Thanks @MarcoGorelli generally lgtm. some doctest failures.

simonjayhawkins · 2020-06-22T09:29:01Z

doc/source/whatsnew/v1.1.0.rst

@@ -997,6 +997,7 @@ I/O
 - Bug in :meth:`~SQLDatabase.execute` was raising a ``ProgrammingError`` for some DB-API drivers when the SQL statement contained the `%` character and no parameters were present (:issue:`34211`)
 - Bug in :meth:`~pandas.io.stata.StataReader` which resulted in categorical variables with difference dtypes when reading data using an iterator. (:issue:`31544`)
 - :meth:`HDFStore.keys` has now an optional `include` parameter that allows the retrieval of all native HDF5 table names (:issue:`29916`)
+- Repr of :class:`Categorical` was not distinguishing between int and str (:issue:`33676`)


The is probably more appropriate for the Categorical section to be consistent with the similar change for Sparse, see #34352. (NOTE: 1.1 does not yet have a Categorical section)

Also, I think safe to say this was a bug since the repr of the categories in the dtype repr included the quotes #34352 (comment)

simonjayhawkins · 2020-06-22T10:17:40Z

pandas/io/formats/format.py

+                lambda x: pprint_thing(
+                    x, escape_chars=("\t", "\r", "\n"), quote_strings=quote_strings
+                )
+            )


as the ternary expression is getting more complex, an if else may now be more readable. maybe use partial instead of lambda and maybe move quote_strings assignment inside the relevant if else block.

simonjayhawkins · 2020-06-22T10:24:24Z

pandas/core/arrays/categorical.py

@@ -1921,7 +1928,7 @@ def _get_repr(self, length=True, na_rep="NaN", footer=True) -> str:
        from pandas.io.formats import format as fmt

        formatter = fmt.CategoricalFormatter(
-            self, length=length, na_rep=na_rep, footer=footer
+            self, length=length, na_rep=na_rep, footer=footer, quoting=QUOTE_NONNUMERIC


maybe don't need to pass this through here, since CategoricalFormatter should always be QUOTE_NONNUMERIC

simonjayhawkins · 2020-06-22T10:24:48Z

pandas/io/formats/format.py

@@ -170,12 +171,14 @@ def __init__(
        length: bool = True,
        na_rep: str = "NaN",
        footer: bool = True,
+        quoting: Optional[int] = None,


maybe redundant see above comment.

simonjayhawkins · 2020-06-22T10:24:56Z

pandas/io/formats/format.py

    ):
        self.categorical = categorical
        self.buf = buf if buf is not None else StringIO("")
        self.na_rep = na_rep
        self.length = length
        self.footer = footer
+        self.quoting = quoting


simonjayhawkins · 2020-06-22T10:25:52Z

pandas/io/formats/format.py

@@ -200,6 +203,7 @@ def _get_formatted_values(self) -> List[str]:
            None,
            float_format=None,
            na_rep=self.na_rep,
+            quoting=self.quoting,


quoting always QUOTE_NONNUMERIC

simonjayhawkins · 2020-06-22T10:30:13Z

pandas/core/arrays/categorical.py

-            category_strs = fmt.format_array(self.categories, None)
+            category_strs = fmt.format_array(
+                self.categories, None, quoting=QUOTE_NONNUMERIC
+            )


maybe partial here to avoid duplication

…ical formatter

simonjayhawkins

Thanks @MarcoGorelli lgtm ex doctest failures

simonjayhawkins

Thanks @MarcoGorelli lgtm. @jreback

jreback · 2020-06-24T15:01:21Z

pandas/core/arrays/categorical.py

@@ -1872,13 +1874,16 @@ def _repr_categories(self):
        )
        from pandas.io.formats import format as fmt

+        format_array = partial(


since you changed this in pandas/io/format.py is it also necessary here?

Yes, it's necessary so that GenericArrayFormatter is initialised with it

jreback · 2020-06-24T15:02:00Z

lgtm ex a question.

jreback · 2020-06-24T22:40:27Z

thanks @MarcoGorelli very nice

…v#34222)

quote string elements

aaf4e42

MarcoGorelli requested review from simonjayhawkins and removed request for simonjayhawkins May 17, 2020 10:43

MarcoGorelli changed the title ~~quote string elements~~ BUG: repr of Categorical does not distinguish int and str. May 17, 2020

MarcoGorelli marked this pull request as draft May 17, 2020 10:49

wip

1f90a62

simonjayhawkins reviewed May 17, 2020

View reviewed changes

pandas/tests/arrays/categorical/test_repr.py Show resolved Hide resolved

wip

5a93a67

jorisvandenbossche mentioned this pull request May 26, 2020

Use repr of fill_value in SparseDtype repr #34352

Merged

jreback added Categorical Categorical Data Type Output-Formatting __repr__ of pandas objects, to_string labels Jun 14, 2020

MarcoGorelli added 9 commits June 18, 2020 09:30

Merge remote-tracking branch 'upstream/master' into cat-rep

4cef20c

wip

a3c6eda

remove vscode failures

9277d38

merge error

b978bf9

quotestring

3215936

only quote categorical

e6ce96f

fix some failing tests due to changes

197038b

whatsnew

f594fa1

set quote strings to boolean

4912ec3

MarcoGorelli marked this pull request as ready for review June 18, 2020 10:59

MarcoGorelli requested review from simonjayhawkins and jreback June 18, 2020 11:01

MarcoGorelli added 2 commits June 22, 2020 07:27

Merge remote-tracking branch 'upstream/master' into cat-rep

79dd24b

fix doctest

d57ae96

simonjayhawkins reviewed Jun 22, 2020

View reviewed changes

simonjayhawkins added this to the 1.1 milestone Jun 22, 2020

MarcoGorelli added 2 commits June 23, 2020 07:11

move whatsnew section, use partial, always quotenonnumeric in categor…

aa62a24

…ical formatter

docstrings

9562313

MarcoGorelli marked this pull request as draft June 23, 2020 07:35

simonjayhawkins approved these changes Jun 23, 2020

View reviewed changes

fix failing doctests

457abe3

MarcoGorelli marked this pull request as ready for review June 24, 2020 09:15

MarcoGorelli requested a review from simonjayhawkins June 24, 2020 09:15

simonjayhawkins approved these changes Jun 24, 2020

View reviewed changes

jreback reviewed Jun 24, 2020

View reviewed changes

jreback merged commit db48799 into pandas-dev:master Jun 24, 2020

MarcoGorelli deleted the cat-rep branch June 25, 2020 07:03

fangchenli pushed a commit to fangchenli/pandas that referenced this pull request Jun 27, 2020

BUG: repr of Categorical does not distinguish int and str. (pandas-de…

95c6be5

…v#34222)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: repr of Categorical does not distinguish int and str. #34222

BUG: repr of Categorical does not distinguish int and str. #34222

MarcoGorelli commented May 17, 2020 •

edited

Loading

simonjayhawkins May 17, 2020

MarcoGorelli May 21, 2020 •

edited

Loading

simonjayhawkins May 21, 2020

jreback Jun 14, 2020

MarcoGorelli Jun 18, 2020

jorisvandenbossche commented May 18, 2020

MarcoGorelli commented May 18, 2020

simonjayhawkins left a comment

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins Jun 22, 2020

simonjayhawkins left a comment

simonjayhawkins left a comment

jreback Jun 24, 2020

MarcoGorelli Jun 24, 2020

jreback commented Jun 24, 2020

jreback commented Jun 24, 2020

BUG: repr of Categorical does not distinguish int and str. #34222

BUG: repr of Categorical does not distinguish int and str. #34222

Conversation

MarcoGorelli commented May 17, 2020 • edited Loading

Choose a reason for hiding this comment

MarcoGorelli May 21, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jorisvandenbossche commented May 18, 2020

MarcoGorelli commented May 18, 2020

simonjayhawkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

simonjayhawkins left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Jun 24, 2020

jreback commented Jun 24, 2020

MarcoGorelli commented May 17, 2020 •

edited

Loading

MarcoGorelli May 21, 2020 •

edited

Loading