BUG in clipboard (linux, python2) with unicode and separator

[pijucha]

This is probably a known bug but I couldn't find a github issue.

There is a disabled test test_clipboard.py which fails with the following error

======================================================================
FAIL: test_round_trip_frame_sep (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/users/piotr/workspace/pandas-pijucha/pandas/io/tests/test_clipboard.py", line 73, in test_round_trip_frame_sep
    self.check_round_trip_frame(dt, sep=',')
  File "/home/users/piotr/workspace/pandas-pijucha/pandas/io/tests/test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "/home/users/piotr/workspace/pandas-pijucha/pandas/util/testing.py", line 1276, in assert_frame_equal
    right.columns))
  File "/home/users/piotr/workspace/pandas-pijucha/pandas/util/testing.py", line 1022, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame are different

DataFrame shape (number of columns) are different
[left]:  2, Index([u'en', u'es'], dtype='object')
[right]: 0, Index([], dtype='object')

Code Sample, a copy-pastable example if possible

More explicitly (the example from the above test):

nonascii = pd.DataFrame({'en': 'in English'.split(), 'es': 'en español'.split()})

nonascii.to_clipboard(sep=',')

read_clipboard(sep=',', index_col=0)
Out[154]: 
Empty DataFrame
Columns: []
Index: [0       in       en, 1  English  español]

read_clipboard()
Out[155]: 
        en       es
0       in       en
1  English  español

Expected Output

read_clipboard(sep=',', index_col=0)
Out[134]: 
        en       es
0       in       en
1  English  español

read_clipboard()
Out[135]: 
              ,en,es
0            0,in,en
1  1,English,español

output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 4.1.20-1
machine: x86_64
processor: Intel(R)_Core(TM)_i5-2520M_CPU_@_2.50GHz
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.18.1+240.gbb6b5e5
nose: 1.3.7
pip: 8.1.2
setuptools: 21.2.0
Cython: 0.24.1
numpy: 1.11.0

---

There are probably 2 issues in the code.

1. `.encode('utf-8') is called on a py2 string, which raises if there is a non-ascii character in the string, and then
2. to_clipboard falls back to to_string method.
   (In this case, fixing 1 solves the problem. But in general, if something else raises and we fall back here, a separator is ignored.)

I don't know what to do about 2, but 1 seems to be easy.
Part of the code in util.clipboard.py calls subprocess.Popen.communicate(), which operates on byte types (bytes in PY3 and strings in PY2). So, encode/decode are needed only in PY3.

I believe this 6d4fdb0 fixes the problem. But for now I tested only one pair of functions (in KDE) and couldn't possibly test it on OS X.

[jreback]

this is the same as #12580 / #12529 ; title is not very clear though

[pijucha]

OK. Actually, I saw that but thought it was purely windows related. The bug here is an incorrect use of subprocess in py2.

[jreback]

I get the same on macosx / py2. so your report is prob better here. actually we cannot repro this on the builds anyhow which would be ideal.

ok will re-open and make this an xref issue of that one.

[pijucha]

I see now that pandas.util.clipboard is probably entirely taken from pyperclip and it actually expects unicode input. So maybe it's better not tinker inside and just pass unicode to it. I tested this pijucha@e53dcb0 and it seems to work for both windows and linux.

Apparently, it also solves #12529. So, indeed, these issues are more closely related than I thought.

[jlou2u]

+1 to fix this when possible. It's the only test that fails for me (OSX) in the codebase. More of an annoyance but still...

I'd also suggest not changing/setting default values for kwargs when to_clipboard is called, seems confusing at best and I think functionality is unchanged (proposed fix in jlou2u@01277af that includes pijucha's change)

I haven't done anything with travis before but looking at .travis.yml it seems that xsel is only added onto python 3 builds, but not python 2 builds. I think pandas.util.clipboard will raise import error if it can't find a clipboard utility and test_clipboard.py will raise nose.SkipTest("no clipboard found") if it can't find a clipboard. That's my best guess at why this can't be reproduced in the builds.

[aileronajay]

This test is failing for me on OSX, (with the latest code)

[aileronajay]

i was able to change _copyOSX function in pandas.util.clipboard.py

def _copyOSX(text):
p = Popen(['pbcopy', 'w'], stdin=PIPE, close_fds=True)
try:
p.communicate(input=text.encode('utf-8'))
except UnicodeDecodeError:
p.communicate(input=text)

to make the test pass. The test fails because to_clipboard fails for the data frame and it falls back to the string to clipboard. The to_clipboard fails because, we are trying to encode from ascii to utf8 when we call encode, but the str is already in UTF-8 when we have non ascii characters in the dataframe, hence when it tries to read the non ascii character using ascii, we get a UnicodeDecodeError. By capturing the UnicodeDecodeError exception and passing the string as it (as it is unicode encoded) we can make it work

[pijucha]

@aileronajay The thing is the file pandas.util.clipboard.py contains external code (unmodified Pyperclip v1.5.15 - if nothing has changed since July) and it expects unicode on input. I believe it's better to provide it with such input instead of tinkering inside (I tried both). The fix linked in my comment above should do the trick but I can't test it on OSX.

If I have some time this week I can submit a PR. Unless someone else can do it faster and better.

[aileronajay]

created a pull request (containing the same change as your commit pijucha/pandas@e53dcb0 ), fb922d6