Series.unique() dies with many NaNs

[kieranholland]

Series.unique() dies with many NaNs:

import time

def test_unique(obj):
    for n in range(6):
        objs = Series([obj] * 10 ** n)
        start = time.time()
        objs.unique()
        stop = time.time()
        print('%6.0f %s' % (len(objs), stop - start))

test_unique('a')

       1 4.98294830322e-05
      10 2.40802764893e-05
     100 4.10079956055e-05
    1000 6.91413879395e-05
   10000 0.000164985656738
  100000 0.0013279914856

test_unique(float('nan'))

       1 3.91006469727e-05
      10 3.60012054443e-05
     100 0.000331163406372
    1000 0.0283088684082
   10000 2.71325206757
  100000 Boom!     

[adamklein]

Interesting. Guessing because the hash back-end doesn't realize nan != nan

[wesm]

went a different route, added a float64 hash table with NA handling. getting this result now:

In [1]: paste
import time

def test_unique(obj):
    for n in range(6):
        objs = Series([obj] * 10 ** n)
        start = time.time()
        objs.unique()
        stop = time.time()
        print('%6.0f %s' % (len(objs), stop - start))

test_unique('a')
## -- End pasted text --
     1 7.48634338379e-05
    10 5.3882598877e-05
   100 5.31673431396e-05
  1000 7.41481781006e-05
 10000 0.000144004821777
100000 0.00133991241455

In [2]: test_unique(float('nan'))
     1 5.19752502441e-05
    10 3.40938568115e-05
   100 3.19480895996e-05
  1000 3.91006469727e-05
 10000 9.29832458496e-05
100000 0.000288009643555

[kieranholland]

Thanks for quick response.
I encountered an issue with the new version.
It only happens with multiple nan instances and other objects mixed.

import time

def test_unique_single_nan_instance_and_non_nans():
    for n in range(6):
        s = []
        nan = float('nan')
        for _ in range(10 ** n):
            s.append(nan)
            s.append('a')
        objs = Series(s)
        start = time.time()
        objs.unique()
        stop = time.time()
        print('%6.0f %s' % (len(objs), stop - start))

test_unique_single_nan_instance_and_non_nans() # fine

def test_unique_multiple_nan_instances():
    for n in range(6):
        s = []
        for _ in range(10 ** n):
            s.append(float('nan'))
            s.append(float('nan'))
        objs = Series(s)
        start = time.time()
        objs.unique()
        stop = time.time()
        print('%6.0f %s' % (len(objs), stop - start))

test_unique_multiple_nan_instances() # fine

def test_unique_multiple_nan_instances_and_non_nans():
    for n in range(6):
        s = []
        for _ in range(10 ** n):
            s.append(float('nan'))
            s.append('a')
        objs = Series(s)
        start = time.time()
        objs.unique()
        stop = time.time()
        print('%6.0f %s' % (len(objs), stop - start))

test_unique_multiple_nan_instances_and_non_nans() # not so fine

[wesm]

reopened the issue and will take a look

[wesm]

fixed this in master, let me know if you have any more issues