BUG: Always cast to Categorical in lexsort_indexer

[dsaxton]

• closes BUG: df.sort_values w/ key function fails with multiple sort columns and Categorical sorting #36383
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

[jbrockmendel]

Not sure if this is related, but I noticed a problem in Categorical.sort_values setting self._codes = foo instead of self._codes[:] = foo

[jreback]

can you try when the categorical is ordered=False as well (parameterize)

[dsaxton]

Yeah this is broken actually. Will need to be more careful above.

[dsaxton]

Or maybe my expectation is off about how this should behave when the categorical is unordered. This is odd (it seems to respect the order in which categories are given even when ordered=False):

[ins] In [1]: import pandas as pd
         ...:
         ...:
         ...: values = ["a", "b", "c"]
         ...:
         ...: cat = pd.Categorical(values, categories=["a", "b", "c"], ordered=False)
         ...: print(cat.sort_values())
         ...:
         ...: cat = pd.Categorical(values, categories=["c", "b", "a"], ordered=False)
         ...: print(cat.sort_values())
         ...:
         ...: print(pd.__version__)
         ...:
['a', 'b', 'c']
Categories (3, object): ['a', 'b', 'c']
['c', 'b', 'a']
Categories (3, object): ['c', 'b', 'a']
1.2.0.dev0+390.g595791b6f.dirty

Maybe sorting an unordered categorical should actually be raising.

[jreback]

Maybe sorting an unordered categorical should actually be raising.

had this discussion with @jorisvandenbossche a while back......

yeah sorting just gives back the same ordering as the categories that you have, they just don't mean anything.

so we do allow it.

[jreback]

what I think is broken is actually this

In [184]: pd.Categorical(pd.Categorical(values, categories=["a", "b", "c"], ordered=False), ordered=True)                                                       
Out[184]: 
[a, b, c]
Categories (3, object): [a < b < c]

[jreback]

I think this should raise, though we do allow this via .set_ordered() so maybe its ok

cc @TomAugspurger

[jreback]

since we aren't actually testing this likely i think prob ok to merge this and open an issue for discussion.

[dsaxton]

I would almost expect sorting an unordered categorical to simply return the original array (since maybe you could argue it's already "trivially ordered" in some sense) if it weren't to raise

[dsaxton]

Actually, R does the same thing as pandas interestingly enough:

> x <- factor(c("a", "b", "c", "a"), levels = c("c", "b", "a"), ordered = FALSE)
> x
[1] a b c a
Levels: c b a
> sort(x)
[1] c b a a
Levels: c b a

I guess because it's easiest just to always sort by the underlying codes.

[jorisvandenbossche]

I would almost expect sorting an unordered categorical to simply return the original array (since maybe you could argue it's already "trivially ordered" in some sense)

@dsaxton strings also don't necessarily have a meaningfull order, but we still sort them lexicographically. In the same way, we still sort an unordered categorical, using the order of the categories (which is the same as lexicographically sorted in most cases, unless you specified the categories manually in a certain order).

There are lots of reasons to allow sorting for an "unordered" categorical. One example is to get a deterministic order of your values, which can be useful regardless of the order of the categories having a meaning or not.

[jreback]

thanks @dsaxton

[simonjayhawkins]

@jreback you milestone 1.2. change note in 1.1.3. which should it be?

[simonjayhawkins]

@jreback ok to backport? see #36385 (comment)

[jreback]

yeah this is ok

[simonjayhawkins]

thanks

[simonjayhawkins]

@meeseeksdev backport 1.1.x