nunique is slower than len(set(x.dropna())) for smaller Series.

[lexual]

This is biting me when applying nunique on some groupby operations.

In one benchmark, which I'll shortly share, nunique() is slower until we get to a length of 3000.

[lexual]

See a benchmark here:

https://gist.github.com/lexual/118084593a98aa472289

[lexual]

So I'm wondering if we can use len(set(x.dropna())) where the Series length is below a certain threshold.

e.g. pseudo code.

def nunique(s):
    threshhold = 3000
    if len(s) < threshold:
        return len(set(s.dropna()))
    else:
        return current_nunique_implementation(s)

[jreback]

actually this looks changed recently

time this with

len(unique())

this was replaced by len(value_counts())

iIIRC because that handles all dtypes properly (this is in core/base.py)

[lexual]

Yes len(s.unique()) looks faster across the board, is that what nunique() is now using?

[jreback]

I think it was
was changed somewhat recently

cc @sinhrks ?

[lexual]

https://github.com/pydata/pandas/blob/master/pandas/core/base.py#L295

?

[jreback]

u want I do a pr with a vbench and change to using len(unique)
(also need to handle dropna argument)
FYI this is used by series and index which doesn't have dropna (that may be why value_counts is used(

[jreback]

yep

[lexual]

Updated my gist benchmark thing:

https://gist.github.com/lexual/118084593a98aa472289

[sinhrks]

nunique had been used value_counts prior to #6734, thus I'm not sure the actual background. it may be a reason that unique and value_counts handles NaT differently (fixed in #7424).

I agree len(series.unique()) looks better.

[jreback]

@sinhrks yep the NaT treatment might be slightly different (so maybe need to fix that!!).

@lexual since you brought this up, wnt to do a PR to revert back to using unique, and see if anything pops up?

[lexual]

It's not completely straightforward, due to needing to handle dropna.

I think probably best approach is to add dropna to unique().

Then nunique becomes: return len(self.unique(dropna=dropna))

adding dropna to unique isn't completely trivial and I'm not sure best way to approach that.

I did initially try:
len(self.unique().dropna())

But that errored I think from a test case that uses an Index instead of a Series. Apparently Index doesn't have dropna method.

[jreback]

yes I would pass the dropna to unique, which can then simply dropna if indicated and then use unique1d which handles NA's, not sure why this change to using value_counts, as the unique machinery handles NA's properly.

[lexual]

I'm confused, unique doesn't currently accept a dropna parameter.
Neither does unique1d.
And I don't know much about what's going on in hashtable.

[jreback]

you can add the parameter to unique (dropna=False), then nunique becomes a pass thru

then unique starts like

def unique(self, dropna=False):
    if dropna:
        return self.value_counts(dropna=dropna).index

    # the original unique

I think will work. unique can be called with an Index or a Series. Since Index doesn't have dropna then you have to use value_counts which supports it.

I think unique1d is much faster than value_counts, so its worth it to use if you don't care about dropna.

[jreback]

@lexual hows't this coming?

[jreback]

closed by #9134

@lexual closing but If u have a further improvement in perf pls do a pull request!