PERF: Sparse IntIndex.make_union / Numeric ops

[sinhrks]

• tests added / passed
• passes git diff upstream/master | flake8 --diff
• whatsnew entry

Replace repeated list.append with np.union1d in IntIndex.make_union. make_union is used in numeric ops.

NOTE: It is also possible to fix IntIndex.intersect to use np.intersect1d, but it doesn't increase the performance (because the length of the result is smaller).

The below microbench assumes array's 90% is sparse.

import numpy as np
import pandas as pd

np.random.seed(1)
N = 1000000
a = np.array([np.nan] * N)
b = np.array([np.nan] * N)

indexer_a = np.unique(np.random.randint(0, N, N / 10))
indexer_b = np.unique(np.random.randint(0, N, N / 10))
a[indexer_a] = np.random.randint(0, 100, len(indexer_a))
b[indexer_b] = np.random.randint(0, 100, len(indexer_b))

sa = pd.SparseArray(a)
sb = pd.SparseArray(b)

on current master

%timeit sa.sp_index.make_union(sb.sp_index)
#10 loops, best of 3: 52.7 ms per loop

%timeit sa + sb
10 loops, best of 3: 47.8 ms per loop

After this PR

%timeit sa.sp_index.make_union(sb.sp_index)
100 loops, best of 3: 11.6 ms per loop

%timeit sa + sb
100 loops, best of 3: 15.3 ms per loop

[codecov-io]

Current coverage is 84.06%

Merging #13036 into master will decrease coverage by -0.00%

@@             master     #13036   diff @@
==========================================
  Files           136        136          
  Lines         50005      49994    -11   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          42040      42027    -13   
- Misses         7965       7967     +2   
  Partials          0          0          

1. 2 files (not in diff) in pandas/tseries were modified. more
   • Misses +1
   • Hits -7
2. 2 files (not in diff) in pandas/io were modified. more
   • Misses +1
   • Hits -1
3. 1 files (not in diff) in pandas were modified. more
   • Hits -5

Powered by Codecov. Last updated by e9ef522...33104f5

[jreback]

this looks fine for 0.18.1. do you want to add some sparse benchmarks?

[sinhrks]

OK, added whatsnew and bench.

   before     after       ratio
  [286782da] [a48f05db]
-   18.15ms     8.20ms      0.45  sparse.sparse_arithmetic.time_sparse_addition_10percent
-   18.17ms     8.08ms      0.44  sparse.sparse_arithmetic.time_sparse_division_10percent
-   17.97ms     7.73ms      0.43  sparse.sparse_arithmetic.time_sparse_addition_10percent_zero
-   18.47ms     7.84ms      0.42  sparse.sparse_arithmetic.time_sparse_division_10percent_zero
-    2.06ms   810.82μs      0.39  sparse.sparse_arithmetic.time_sparse_division_1percent
-    2.05ms   804.12μs      0.39  sparse.sparse_arithmetic.time_sparse_addition_1percent

[jreback]

not that I am recommending it, because the numpy code is just so much shorter. But this could have been much faster if the array was pre-allocated (to a max len), then you slice it at the end. Rather than appending using a list.

[sinhrks]

Yeah will take suggested approach when I fix others using list.append.

[jreback]

ok ping on green.

[sinhrks]

Thanks, now green.

[jreback]

thanks @sinhrks