PERF: increase performance of str_split when returning a frame

[cgevans]

Closes #10081.

This simply removes the unnecessary Series() in the creation of the DataFrame for the split values in str_split. It vastly improves performance (20ms vs 3s for a series of 20,000 strings as a test) while not changing current behavior.

[cgevans]

Hmm... this actually causes significant problems when dealing with NaNs.

[jreback]

@cgevans this code path has changed in master a bit (though issue still remains).

[jreback]

@cgevans can you rebase and see where this is?

[cgevans]

The change in the code path means this particular fix would need to be rewritten, but it seems this would need further changes to avoid changing behavior. I've written a bit about the problem in #10081.

One solution is to change the DataFrame constructor such that it handles lists of lists differently, making it consistent with lists of Series.

The other, somewhat hackish method is to take the array that comes from str_split, change all NaNs to [None], construct a DataFrame, and then change all Nones to NaNs.

[cgevans]

This new version now tries to handle NaNs in a bit of a roundabout way, but still in a manner that will be fast.

[jreback]

just redefine cons_row as a function. e.g. something like

def cons_row(x):
    if np.isnan(x):
         return [ x ]
    return x

you don't need to fill

[jreback]

use np.isnan(x)

[cgevans]

I initially tried that, but np.isnan can't handle strings. x here is either NaN or a list of strings. I also only want that conditional to be true if x is a single null value. I agree that is np.nan isn't ideal: what would be the better way of doing this?

[jreback]

pls add a release note. pls show the results of the benchmark for this (there is a join_split one, but you may have to add one)

[jreback]

you can do (you might want to see which is better perf):

I think this can ONLY? return lists or scalars?

if isinstance(list, x): # might need com.is_list_like if this can return list/tuple/ndarray/Series
    # ok
else:
    [ x ]
...

In [8]: x = [ 'foo' ]

In [9]: pd.lib.isscalar(x) and pd.isnull(x)
Out[9]: False

In [10]: x = 'foo'

In [11]: pd.lib.isscalar(x) and pd.isnull(x)
Out[11]: False

In [12]: x = np.nan

In [13]: pd.lib.isscalar(x) and pd.isnull(x)
Out[13]: True

[jreback]

pls also add a note to 0.16.2 performance imprv section.

[cgevans]

Here is the benchmark result:

Invoked with :
--ncalls: 3
--repeats: 3


-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
strings_join_split_expand                    |  48.8970 | 1554.2669 |   0.0315 |
strings_join_split                           |  36.1670 |  36.3259 |   0.9956 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [5e65f02] : PERF: Increase performance of string split when expand=True
Base   [bc7d48f] : disable some deps on 3.2 build

[jreback]

make this str.split when expand=True

[jreback]

minor whatsnew note change. pls ping when revised and will merge.

[cgevans]

Done.

[jreback]

excellent ty!