Fix #12529 / Improve to_clipboard for objects containing unicode #12580

dalito · 2016-03-09T23:57:00Z

Superceded by #14599

closes to_clipboard is no longer Excel compatible #12529
tests added / passed
passes git diff upstream/master | flake8 --diff
whatsnew entry

I added the whatsnew entry to 0.18.0.

…code.

jreback · 2016-03-09T23:58:56Z

move to 0.18.1

jreback · 2016-03-10T00:00:08Z

use the option display.encoding for the actual encoding

jreback · 2016-03-10T00:00:16Z

pls add a test

dalito · 2016-03-10T00:16:30Z

use the option display.encoding for the actual encoding

pandas.options.display.encoding is utf-8 on my German Win10 or Win7. However, the encoding I have to use to get correct behaviour is "cp1250" (which is what is returned by locale.getdefaultlocale()[1]).

I'll look into adding a test tomorrow.

jreback · 2016-03-10T00:22:53Z

hmm ok

look in the code and see how display.encoding is determined as well

it may be that that is wrong (iow not taking into account the locale) or this is just different

dalito · 2016-03-10T22:54:26Z

Before writing a new test I ran the tests in "test_clipboard.py" on the unmodified 0.18.0rc1 installation. Surprisingly they fail:

py-2|C:\dev\Python27_64\Lib\site-packages\pandas\io\tests>nosetests test_clipboard.py
.EE.
======================================================================
ERROR: test_round_trip_frame (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 81, in test_round_trip_frame
    self.check_round_trip_frame(dt)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 866, in raise_assert_detail
    [right]: {3}""".format(obj, message, left, right)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 9: ordinal not in range(128)

======================================================================
ERROR: test_round_trip_frame_sep (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 73, in test_round_trip_frame_sep
    self.check_round_trip_frame(dt, sep=',')
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 866, in raise_assert_detail
    [right]: {3}""".format(obj, message, left, right)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf1' in position 9: ordinal not in range(128)

----------------------------------------------------------------------
Ran 4 tests in 0.328s

FAILED (errors=2)

dalito · 2016-03-10T22:57:41Z

After fixing pandas\util\testing.py, I still get a failure:

py-2|C:\dev\Python27_64\Lib\site-packages\pandas\io\tests>nosetests test_clipboard.py
.FF.
======================================================================
FAIL: test_round_trip_frame (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 81, in test_round_trip_frame
    self.check_round_trip_frame(dt)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 867, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 1] are different

DataFrame.iloc[:, 1] values are different (50.0 %)
[left]:  [en, espa\xf1ol]
[right]: [en, espa\xc3\xb1ol]

======================================================================
FAIL: test_round_trip_frame_sep (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 73, in test_round_trip_frame_sep
    self.check_round_trip_frame(dt, sep=',')
  File "C:\dev\Python27_64\Lib\site-packages\pandas\io\tests\test_clipboard.py", line 69, in check_round_trip_frame
    tm.assert_frame_equal(data, result, check_dtype=False)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 1115, in assert_frame_equal
    obj='DataFrame.iloc[:, {0}]'.format(i))
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 999, in assert_series_equal
    check_less_precise, obj='{0}'.format(obj))
  File "pandas\src\testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas\src\testing.c:3809)
  File "pandas\src\testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas\src\testing.c:2685)
  File "C:\dev\Python27_64\Lib\site-packages\pandas\util\testing.py", line 867, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.iloc[:, 1] are different

DataFrame.iloc[:, 1] values are different (50.0 %)
[left]:  [en, espa\xf1ol]
[right]: [en, espa\xc3\xb1ol]

----------------------------------------------------------------------
Ran 4 tests in 0.391s

FAILED (failures=2)

jreback · 2016-03-11T00:34:11Z

yeah these currently fail on windows py2 (and I have a single/different one failing on macosx), but nothing fails on linux (which is where Travis runs). They all work on py3.

I think we have an issue about that. But if not, can you open one. Something is still not being encoded correctly. Of course if you can figure out would be great!

jreback · 2016-05-07T17:50:27Z

can you rebase / update

pijucha · 2016-07-24T22:32:28Z

@dalito Are you still working on it?

If I'm not mistaken the errors you're getting are caused by pandas.util.clipboard (pyperclip), which expects unicode input.

This solution: pijucha@e53dcb0 works for me in python2 (both windows and linux). You can test it and include in your PR if it also works for you.

pijucha · 2016-07-24T22:56:42Z

pandas/io/clipboard.py

+        except UnicodeEncodeError:
+            # try again with encoding from locale
+            from locale import getdefaultlocale
+            obj.to_csv(buf, sep=sep, encoding=getdefaultlocale()[1], **kwargs)


I believe using default locale is not necessary. utf-8 should do the job (plus decoding buf.getvalue to utf-8) . The windows code of pyperclip (pandas.util.clipboard) would ultimately convert it back to unicode.

It is a while ago since I looked at this. I just remember that I found some edge cases which still failed. I'll try to find my test cases to see if what you propose works.

Thanks. I didn't want to submit a new PR while yours is open. But probably more important is whether this works for your edge cases and on your system.

jreback · 2016-09-09T22:37:30Z

can you rebase / update?

jorisvandenbossche · 2016-11-18T09:34:48Z

Closing this PR as this will be handled in #14599. @dalito Thanks for your work on this, and you are certainly welcome to test the other PR.

Fixes #12529 - Improvement of to_clipboard for objects containing uni…

deee809

…code.

dalito changed the title ~~Fixes #12529 / Improves to_clipboard for objects containing unicode~~ Fix #12529 / Improve to_clipboard for objects containing unicode Mar 9, 2016

Moved note from whatsnew 0.18.0 to 0.18.1

da25218

jreback added Unicode Unicode strings Error Reporting Incorrect or improved errors from pandas labels Mar 10, 2016

jreback mentioned this pull request Jul 22, 2016

BUG in clipboard (linux, python2) with unicode and separator #13747

Closed

pijucha reviewed Jul 24, 2016
View reviewed changes

pijucha mentioned this pull request Nov 8, 2016

BUG in clipboard (linux, python2) with unicode and separator (GH13747) #14599

Closed

3 tasks

jorisvandenbossche closed this Nov 18, 2016

jorisvandenbossche added this to the No action milestone Nov 18, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #12529 / Improve to_clipboard for objects containing unicode #12580

Fix #12529 / Improve to_clipboard for objects containing unicode #12580

dalito commented Mar 9, 2016 •

edited by jorisvandenbossche

Loading

jreback commented Mar 9, 2016

jreback commented Mar 10, 2016

jreback commented Mar 10, 2016

dalito commented Mar 10, 2016

jreback commented Mar 10, 2016

dalito commented Mar 10, 2016

dalito commented Mar 10, 2016

jreback commented Mar 11, 2016

jreback commented May 7, 2016

pijucha commented Jul 24, 2016

pijucha Jul 24, 2016

dalito Jul 27, 2016

pijucha Jul 28, 2016

jreback commented Sep 9, 2016

jorisvandenbossche commented Nov 18, 2016

Fix #12529 / Improve to_clipboard for objects containing unicode #12580

Fix #12529 / Improve to_clipboard for objects containing unicode #12580

Conversation

dalito commented Mar 9, 2016 • edited by jorisvandenbossche Loading

jreback commented Mar 9, 2016

jreback commented Mar 10, 2016

jreback commented Mar 10, 2016

dalito commented Mar 10, 2016

jreback commented Mar 10, 2016

dalito commented Mar 10, 2016

dalito commented Mar 10, 2016

jreback commented Mar 11, 2016

jreback commented May 7, 2016

pijucha commented Jul 24, 2016

pijucha Jul 24, 2016

Choose a reason for hiding this comment

dalito Jul 27, 2016

Choose a reason for hiding this comment

pijucha Jul 28, 2016

Choose a reason for hiding this comment

jreback commented Sep 9, 2016

jorisvandenbossche commented Nov 18, 2016

dalito commented Mar 9, 2016 •

edited by jorisvandenbossche

Loading