BUG: when reading an HDF5 file with an encoding, default to using this encoding

[jreback]

http://stackoverflow.com/questions/32553207/values-missing-when-loaded-from-pandas-hdf5-file/32587108?noredirect=1#comment53034458_32587108

e.g.

if I

df.to_hdf('foo.h5','df',encoding='utf-8')

then

pd.read_hdf('foo.h5','df')

should pick up the encoding for each table (unless the user overrides by passing in encoding=...

[andreabedini]

I am investigating this since it has been reported on the PyTables mailing list too. It seems that specifying the encoding doesn't help much.

In [29]: store = pd.HDFStore('feeds.h5', mode='w')

In [30]: store.append('feeds', feeds_series[84:86], min_itemsize=200, encoding='utf-8')

In [31]: store.flush()

In [32]: store.select('feeds',encoding='utf-8')
Out[32]: 
84                                                     
85    April 2014. Blir MDG det nye arbeider @partiet...
Name: feeds, dtype: object

In [33]: feeds_series[84:86]
Out[33]: 
84    De statsbærende partiene Ap og Høyre må ta sky...
85    April 2014. Blir MDG det nye arbeider @partiet...
Name: feeds, dtype: object

[andreabedini]

It's getting more interesting. If you save only the row 84 it works, same if you save 84, 85 and 86. Only when you save 84 and 85, 84 disappears.

In [41]: store = pd.HDFStore('feeds.h5', mode='w')

In [42]: store.append('a', feeds_series[84:85], min_itemsize=200, encoding='utf-8')

In [43]: store['a']
Out[43]: 
84    De statsbærende partiene Ap og Høyre må ta sky...
Name: feeds, dtype: object

In [44]: store.append('b', feeds_series[84:86], min_itemsize=200, encoding='utf-8')

In [45]: store['b']
Out[45]: 
84                                                     
85    April 2014. Blir MDG det nye arbeider @partiet...
Name: feeds, dtype: object

In [46]: store.append('c', feeds_series[84:87], min_itemsize=200, encoding='utf-8')

In [47]: store['c']
Out[47]: 
84    De statsbærende partiene Ap og Høyre må ta sky...
85    April 2014. Blir MDG det nye arbeider @partiet...
86                       MDG: Hasj for kjøtt. #valg2015
Name: feeds, dtype: object

[kawochen]

@andreabedini This has been fixed on master I think.

[andreabedini]

@kawochen ah cool, I can stop debugging then :P pointer to the solution?

[kawochen]

@andreabedini I'm not sure where it was fixed, but the bug (about line 84) is reproducible with 0.16.2 but not with master.

[jreback]

We were not properly decoding, see here

So the OP that started this (on SO and PyTables mailing list) is fixed in master by the above, if you specify the encoding. THIS issue is about using the encoding that is already recorded in the meta-data (and using that).

[rspeer]

On the duplicate bug I reported, @jreback said "you need to pass the encoding when reading".

How do I pass the encoding? I don't see that as an option in http://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_hdf.html .

[jreback]

yeah it's missing from the doc string that's should be addressed in the xref issue

[jreback]

encoding='utf8' or whatever u passed when u created it

[rspeer]

Sorry, that didn't work. I even tried re-saving it with encoding='utf-8'. The bytes of text saved in the file are indeed UTF-8, but they're still getting read as the empty string.