BUG: Fix Series nunique groupby with object dtype

[cpcloud]

closes #11077

[jreback]

cc @behzadnouri

or you want to use do a factorize?

[behzadnouri]

diff --git a/pandas/core/groupby.py b/pandas/core/groupby.py
index f34fd6e..39be706 100644
--- a/pandas/core/groupby.py
+++ b/pandas/core/groupby.py
@@ -2565,13 +2565,21 @@ class SeriesGroupBy(GroupBy):
         ids, _, _ = self.grouper.group_info
         val = self.obj.get_values()

-        sorter = np.lexsort((val, ids))
+        try:
+            sorter = np.lexsort((val, ids))
+        except TypeError:
+            val, _ = algos.factorize(val, sort=False)
+            sorter = np.lexsort((val, ids))
+            isnull = lambda a: a == -1
+        else:
+            isnull = com.isnull
+
         ids, val = ids[sorter], val[sorter]

         # group boundries are where group ids change
         # unique observations are where sorted values change
-        idx = com._ensure_int64(np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]])
-        inc = com._ensure_int64(np.r_[1, val[1:] != val[:-1]])
+        idx = np.r_[0, 1 + np.nonzero(ids[1:] != ids[:-1])[0]]
+        inc = np.r_[1, val[1:] != val[:-1]]

         # 1st item of each group is a new unique observation
         mask = isnull(val)

[cpcloud]

@behzadnouri If you submit that as a PR to my fork, I'll happily merge it in

[behzadnouri]

the error says TypeError: merge sort not available for item 0 even though numpy does have merge sort for objects (both np.argsort and np.sort). so it may be a bug on numpy side. so, it is better to implement try catch in case future numpy releases do implement lexsort for this case

once, values are factorized, original values are not needed and it is more efficient to work with integer factors, as long as isnull function is adjusted accordingly.

[cpcloud]

@jreback getting a somewhat strange travis failure here that seems unrelated to my change: https://travis-ci.org/cpcloud/pandas/jobs/80036566

[cpcloud]

any ideas what that might be?

[jreback]

hmm someone else reported that as well
try restarting it and see

[shoyer]

Note that this means object dtype? Maybe add an assert?

[cpcloud]

this is passing on my travis ci fork

[jreback]

add a comment here that this catches object dtypes

[jreback]

couple comments, squash and merge ok

[jreback]

move this to where the other nunique comment is (just put in in the list)

[jreback]

revert these 2 lines, these don't pass on windows ATM. this can possibly be changed, but on another PR/issue

[jreback]

@cpcloud can you update (e.g. revert those 2 lines). ok to merge on green after that.

[jreback]

ty sir!