Unicode column misalignment

[wesm]

In [17]: open('/home/wesm/tmp/foo.csv', 'rb').read()
Out[17]: '\xe6\xb8\xac\xe8\xa9\xa6\xe4\xb8\x80,\xe6\xb8\xac\xe8\xa9\xa6\xe4\xb8\x89\r\[email protected],\xe6\xb8\xac\xe8\xa9\xa6\xe4\xb8\x80\r\[email protected],\xe6\xb8\xac\xe8\xa9\xa6\xe4\xba\x8c\r\[email protected],\xe6\xb8\xac\xe8\xa9\xa6\xe4\xb8\x89\r\n'

In [18]: read_csv('/home/wesm/tmp/foo.csv', encoding='utf-8')
Out[18]: 
               測試一  測試三
0  [email protected]  測試一
1  [email protected]  測試二
2  [email protected]  測試三

In [24]: df
Out[24]: 
               測試一  測試三
0  [email protected]  測試一
1  [email protected]  測試二
2  [email protected]  測試三

In [25]: df.columns[0]
Out[25]: u'\u6e2c\u8a66\u4e00'

In [26]: df.columns[1]
Out[26]: u'\u6e2c\u8a66\u4e09'

[wesm]

Actually, this may just be that monospace is not possible with chinese characters

[changhiskhan]

it would significantly impact performance but we could use unicodedata.east_asian_width to check whether the chars are double width. Maybe do this after we include a .pandas file so if you do work with east asian fonts you can have it on by default?

[ariddell]

Did some work on this. Turns out the monospace Chinese characters in question are exactly 2 monospace ASCII characters wide.

commit 1002a365fb81291403ec43d253a5e97fdf3234f4 closes #2612

>>> df
               測試一  測試三
0  abc@example.com  測試一
1  def@example.com  測試二
2  ghi@example.com  測試三

Now:

>>> df
               測試一           測試三
0  abc@example.com        測試一   
1  def@example.com        測試二   
2  ghi@example.com        測試三   

The fix correctly calculates the width of the three (Chinese) character data as six display characters. Not sure why it's not also fixing the header display.

[jreback]

let's push to 0.14, once .pandasrc in place this is easy to allow in as an option

[sinhrks]

Looked little, and colwidth should handle other 4 Eastern Asia widthes('Na', 'N', 'H', 'A') . Also, I think common.adjoin and format._make_fixed_width should be fixed to change the number of padding spaces.

My current result

https://github.com/sinhrks/pandas/tree/unicode_justify

df = pd.DataFrame(np.random.randn(3, 2), columns=[u'パンダ子パンダ孫パンダ', u'もう笹飽きた'])
print(df)

[ayapi]

Hello, this is very important issue for Japanese, Chinese, Korean
Please take actions.

[jreback]

@sinhrks I think you have a branch with a possible fix?

can you reinvigorate when you have a chance. thxs.

[sinhrks]

Sure. The blocker was how to write a test both work on py2 and 3 (can't use escaped unicode because it changes eastern asia width). Now we can use u.

[jreback]

right. further if we need to have an option (ok by me), i would use display.unicode.* (e.g. create a new namespace)