ERR: validate encoding on to_stata

[ozak]

It seems pandas in python3.5 causes issues due to encoding. For example the following generates a corrupt output file

import pandas as pd
df1 = pd.DataFrame(np.array([1,2,3,4]), columns=['var1'])
df1.to_stata('corrupt.dta', write_index=False, encoding='utf8')

while

df1.to_stata('not-corrupt.dta', write_index=False)

generates a correct file. I imagine this may be due to use of encoding and the difference in the treatment between python 2 and python 3, which breaks compatibility of scripts across python versions. I guess it would be nice if it does not take this option into account on python 3, unless the error is caused by something else.

[jreback]

In [8]: df1 = pd.DataFrame(np.array([1,2,3,4]), columns=['var1'])
   ...: df1.to_stata('corrupt.dta', write_index=False, encoding='latin1')
   ...: 
   ...: 
   ...: 

In [9]: pd.read_stata('corrupt.dta')
Out[9]: 
   var1
0     1
1     2
2     3
3     4

doc-string

Signature: df1.to_stata(fname, convert_dates=None, write_index=True, encoding='latin-1', byteorder=None, time_stamp=None, data_label=None, variable_labels=None)
Docstring:
A class for writing Stata binary dta files from array-like objects

Parameters
----------
fname : str or buffer
    String path of file-like object
convert_dates : dict
    Dictionary mapping columns containing datetime types to stata
    internal format to use when wirting the dates. Options are 'tc',
    'td', 'tm', 'tw', 'th', 'tq', 'ty'. Column can be either an integer
    or a name. Datetime columns that do not have a conversion type
    specified will be converted to 'tc'. Raises NotImplementedError if
    a datetime column has timezone information
write_index : bool
    Write the index to Stata dataset.
encoding : str
    Default is latin-1. Unicode is not supported
byteorder : str
    Can be ">", "<", "little", or "big". default is `sys.byteorder`
time_stamp : datetime
    A datetime to use as file creation date.  Default is the current
    time.
dataset_label : str
    A label for the data set.  Must be 80 characters or smaller.
variable_labels : dict
    Dictionary containing columns as keys and variable labels as
    values. Each label must be 80 characters or smaller.

I would say this is technically correct, passing a unicode encoding is invalid. But I think we should simply reject these, rather than actually write an invalid format. want to do a PR to do this? (now I am not sure which encoding stata can actually support, any idea?)

[ozak]

I am a it confused now. From the docs it should fail in both python 2 and 3, but in python 2

df1.to_stata('corrupt.dta', write_index=False, encoding='utf8')

generates the correct file. So, I would think it may be better to just ignore the option in python 3, but keep it in python 2.

I think stata 14 now supports UTF8 as a default, but it actually may be more general, not 100% sure (see here). I'll try to find some time to write a PR. I'll see what the code actually does. I'll let you know.

[jreback]

cc @bashtage any ideas here?

[bashtage]

There is no UTF8 in Stata. Only ASCII And the simple 8 bit encoding Latin-1.

[bashtage]

In Stata I mean in to_stata. Adding UTF8 is a major effort since the current format is all fixed width and I don't see much of a case for making the effort.

[jreback]

@bashtage so we should then validate encoding='ascii'|'latin1'|None as the only allowed encodings at all.

[ozak]

So why doesn't it generate a corrupt file in python 2, but does in python 3? I have the same pandas version in both, so it may not be pandas specific?

[jreback]

because its actually encoding it in PY3 with the passed in encoding (utf8), rather than the default of latin1.

[ozak]

So in PY2 it is not encoded as UTF8 even when giving the option? How is it saved then? What in pandas is affecting the IO to Stata that corrupts the file? Given that Stata14 uses UTF8 as the default it should not have an issue opening UTF8 encoded files.

[ozak]

Just noticed one more thing

import pandas as pd
df1 = pd.DataFrame(np.array([u'á',u'Ö']), columns=['var1'])
df1.to_stata('not-corrupt.dta', write_index=False, encoding='utf8')

df = pd.read_stata('corrupt3.dta')
df == df1

generates a usable file in both PY2 and PY3. Still, the data is wrong as seen in the example.

[jreback]

So in PY2 it is not encoded as UTF8 even when giving the option?

not really sure it is actually encoding as utf8 internally. lots of things in py2 are wonky. it probably happens to work.

[jreback]

in any event. seems you have a bunch of tests cases! it seems easy enough to simply validate the encoding that is passed and raise if its not valid.

[ozak]

Indeed.

I guess my example shows that PY2 is not encoding at all.

[TiesdeKok]

I would like to report that with the current encoding options it is not possible for Chinese users to interact with Pandas due to their Stata files containing Chinese characters. This has been reported by the user @lianyonghui123 on my ipystata package: TiesdeKok/ipystata#28

[bashtage]

There is no Unicode support in the pandas stata functions. They only support the fixed encoding latin1. It would be a relatively large rewrite to add Unicode support.

…

On Tue, Sep 5, 2017, 13:14 Ties de Kok ***@***.***> wrote: I would like to report that with the current encoding options it is not possible for Chinese users to interact with Pandas due to their Stata files containing Chinese characters. This has been reported by the user @lianyonghui123 <https://github.com/lianyonghui123> on my ipystata package: TiesdeKok/ipystata#28 <TiesdeKok/ipystata#28> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#15723 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFU5RSCFIyLpKXRO_sa8CifMaN_QpvHCks5sfTsggaJpZM4Mg5rV> .