BUG: HDFStore.append with encoded string itemsize

[TomAugspurger]

Closes #11234

Failure came when the maximum length of the unencoded string
was smaller than the maximum encoded length.

Need to run a perf check still. We end up having to call _convert_string_array twice, once before we know the min_itemsize, and a second time just before appending once we do know the min itemsize.

[TomAugspurger]

So initially I had to call _convert_string_array twice, the second time just to change the length. But it seems like we can just astype the converted string to the new length, which is quite a bit faster.

In [1]: from pandas.io.pytables import _convert_string_array

In [2]: a = np.array(10000 * ['ë́'])

In [3]: c =  _convert_string_array(a, 'utf-8')

In [4]: %timeit _convert_string_array(a, 'utf-8')
100 loops, best of 3: 5.37 ms per loop

In [5]: %timeit c.astype('|S6')
10000 loops, best of 3: 91.4 µs per loop

All the tests passed, but I've never worked with numpy's fixed length strings. Hopefully I'm not breaking any rules here.

[jreback]

use double backticks around HDFStore.append

[jreback]

right that is ok (though maybe pass copy=False) (e.g. if we didn't change it don't want to copy again)

[TomAugspurger]

Didn't know that was a kwarg to astype, cool.

I'll ping you when it's green.

[jreback]

yep.....these 0.17.1 will have to wait till after I release 0.17.0 (on friday).....but getting them lined up

[jreback]

merged via 26db172

thanks @TomAugspurger