(WIP) PERF: improve .str perf for all-string values (about 2x-)

[sinhrks]

Related to #10081. Make a short path using numpy's string funcs when all the target values are strings. Otherwise, use current path.

Followings are current comparison results:

import pandas as pd
import numpy as np
import string
import random

np.random.seed(1)
s = [''.join([random.choice(string.ascii_letters + string.digits) for i in range(3)]) for i in range(1000000)]

# s_str uses short path
s_str = pd.Series(s)

# set object 
s[-1] = 1

# s_obj uses current path
s_obj = pd.Series(s)

%timeit s_str.str.lower()
#1 loops, best of 3: 696 ms per loop
%timeit s_obj.str.lower()
#1 loops, best of 3: 1.46 s per loop

%timeit s_str.str.split('a')
#1 loops, best of 3: 1.55 s per loop
%timeit s_obj.str.split('a')
#1 loops, best of 3: 3.52 s per loop

The logic has an overhead to check whether target values are all-string using lib.is_string_array. But this should be speed-up in most cases because it takes relatively shorter time than string ops, and (I believe) values should be all-string in most cases.

%timeit pd.lib.is_string_array(s_str.values)
#10 loops, best of 3: 21.9 ms per loop

If it looks OK, I'll work on all the funcs which is supported by numpy.

• Add all numpy funcs
  • http://docs.scipy.org/doc/numpy/reference/routines.char.html
• Add tests to check numpy results for all-string values are identical as current results.
• Add vbench (both for all-string and object-included)

[sinhrks]

Ah noticed above comparison is not fair, preparing valid ones...

[sinhrks]

I misunderstood the differnce of str and object is caused by numpy logic. As numpy looks to use similar logic as pandas thus cannot expect such a performance gain... Allow me to close this.

[jorisvandenbossche]

Where did the initial speed-up come from then?

[sinhrks]

The above difference exists on current master depending on dtypes handled. I've misunderstand it is caused be the logic i've changed, but not.