PERF: Sparse Series to scipy COO sparse matrix

[TLouf]

• closes ENH: sparse_series_to_coo performance #42880
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

I made the test more thorough, as it only checked the output's type, but not that the output matrix was actually the expected one.

Here is the output from the asv benchmark:

       before           after         ratio
     [e045034e]       [042dbbc3]
     <master>         <sseries-to-coo-perf>
-      12.6±0.6ms       2.35±0.1ms     0.19  sparse.ToCoo.time_sparse_series_to_coo_single_level(False)
-      19.9±0.6ms      3.68±0.05ms     0.19  sparse.ToCoo.time_sparse_series_to_coo(True)
-      19.6±0.4ms      3.51±0.09ms     0.18  sparse.ToCoo.time_sparse_series_to_coo(False)
-        12.5±1ms         220±10μs     0.02  sparse.ToCoo.time_sparse_series_to_coo_single_level(True)

As you can see the performance improvement is the most dramatic when there is only one row and column level, and for an output with sorted labels. This is simply due to the fact that the labels are sorted in the levels attribute of an Index, so I covered this case, which should be the most common, with a simpler (and faster) implementation. For this reason I was thinking maybe sort_labels could default to True, but that would probably require prior deprecation warnings and all. In any case I can add this information as a performance tip to the docs, let me know what you think.

[jreback]

can you add a whatsnew note for 1.4 perf section. will have to look again

[TLouf]

@jreback I think everything's ok now, the failing checks seem unrelated.

[jreback]

do we have an asv for sorted_labels=False but row_level/column_levels a single level? e.g. the case you mention in the doc-string?

[jreback]

where is this called? ideally prefer to return a dataclass / namedtuple if possible

[TLouf]

The only place it's called is in sparse_series_to_coo https://github.com/pandas-dev/pandas/pull/42925/files#diff-29d84e278af1528165388e964717fd13f9cabeb155887550b9d2613579d52b65L110, and the first three returns are fed directly to scipy.sparse.coo_matrix. Do you still think it's preferable to have this return a namedtuple?

[jreback]

its ok, makes the code hard to read though (but prob not worth changing)

[TLouf]

The one for sort_labels=False but row_level/column_levels a single level is the first one reported below:

       before           after         ratio
     [e045034e]       [621b1028]
     <master>         <sseries-to-coo-perf>
-      11.8±0.9ms      2.17±0.05ms     0.18  sparse.ToCoo.time_sparse_series_to_coo_single_level(False)
-        21.2±1ms       3.29±0.5ms     0.16  sparse.ToCoo.time_sparse_series_to_coo(True)
-      19.0±0.5ms       2.78±0.1ms     0.15  sparse.ToCoo.time_sparse_series_to_coo(False)
-      12.5±0.9ms         200±10μs     0.02  sparse.ToCoo.time_sparse_series_to_coo_single_level(True)

the one I mention in the docstring, which is the last one above and has better performance, is for sort_labels=True.

[jreback]

@TLouf lgtm.

@pandas-dev/pandas-core if any comments.

[jreback]

thanks @TLouf very nice!