Test for Python 3.5 with C locale #14114

nbonnotte · 2016-08-28T17:35:45Z

As @jreback suggested, I'm adding an alternate py3 build to change LOCALE, based on the 3.4 slow build, to reveal some encoding bugs (see #12337)

I'm new to configuring Travis, I just hope this will work as I expect and that the tests will fail.

closes BUG: UnicodeEncodeError in test_to_latex_filename (pandas.tests.test_format.TestDataFrameFormatting) #12337
tests added but not passed?
passes git diff upstream/master | flake8 --diff
whatsnew entry

codecov-io · 2016-08-28T20:27:00Z

Current coverage is 85.24% (diff: 100%)

Merging #14114 into master will not change coverage

@@             master     #14114   diff @@
==========================================
  Files           140        140          
  Lines         50556      50556          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          43095      43095          
  Misses         7461       7461          
  Partials          0          0

Powered by Codecov. Last update 289cd6d...dadf73c

nbonnotte · 2016-08-29T18:07:46Z

This is disappointing. I added a travis job with LC_ALL=C, and the tests still succeed: https://travis-ci.org/pydata/pandas/jobs/155766390

The new locale seems to be taken into account, since travis says

$ ci/script.sh
inside ci/script.sh
Setting LC_ALL to C
/home/travis/build/pydata/pandas/pandas/computation/__init__.py:19: UserWarning: The installed version of numexpr 2.4.4 is not supported in pandas and will be not be used
  UserWarning)
pandas detected console encoding: ANSI_X3.4-1968

But on my computer,

$ LC_ALL=C nosetests pandas.tests.formats.test_format                                                     
..................S..............................................SSS.........E........................................................................................................
======================================================================
ERROR: test_to_latex_filename (pandas.tests.formats.test_format.TestDataFrameFormatting)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/nicolas/Documents/Projects/pandas/pandas/tests/formats/test_format.py", line 2808, in test_to_latex_filename
    df.to_latex(path)
  File "/home/nicolas/Documents/Projects/pandas/pandas/core/frame.py", line 1655, in to_latex
    encoding=encoding)
  File "/home/nicolas/Documents/Projects/pandas/pandas/formats/format.py", line 661, in to_latex
    latex_renderer.write_result(f)
  File "/home/nicolas/Documents/Projects/pandas/pandas/formats/format.py", line 897, in write_result
    buf.write(' & '.join(crow))
UnicodeEncodeError: 'ascii' codec can't encode character '\xdf' in position 7: ordinal not in range(128)

----------------------------------------------------------------------
Ran 182 tests in 3.860s

FAILED (SKIP=4, errors=1)

I'm missing something, obviously. I'm going to try with Python 3.5, which is what I'm using on my computer, but I don't see how that could change things.

jreback · 2016-08-31T16:33:28Z

.travis.yml

+        apt:
+          packages:
+          - xsel
+#    In allow_failures


remove this line (as its duped)

What line should I remove, exactly?

jreback · 2016-08-31T16:37:01Z

so the JOB_NAME determines which files to pick up in the ci\requirements- dir. You need to add a copy of the files for the job which you copied. For something like this we would only need minimal dependencies (so its faster). you can zonk most of them (e.g. I assume you copied requirements-3.5.run (and .pip and .build), so edit the .run and remove evetything but pytz,dateutil,numpy

nbonnotte · 2016-09-03T07:09:34Z

There is no requirements-3.5.pip . And I guess you meant JOB_TAG determines the files to pickup in the ci/ folder?

I don't understand, the new test does not show up in Travis. It was the case with the previous commit, even though the new test was not in the "allowed failures". But it disappeared with my last commit...

jorisvandenbossche · 2016-09-03T09:28:10Z

@nbonnotte You have to add it in both places (both in the base matrix, as repeat it in the allowed failures section. As you had it in the first commit I think)

nbonnotte · 2016-09-05T11:07:04Z

It works! I mean, it fails: https://travis-ci.org/pydata/pandas/jobs/157552565

jreback · 2016-09-05T12:57:20Z

great!

I think something up with the locale string itself - it doesn't seem to fully parse (error above the latex one); the final error is tagged in another issue

jreback · 2016-09-05T14:58:21Z

======================================================================
ERROR: test_set_locale (pandas.tools.tests.test_util.TestLocaleUtils)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tools/tests/test_util.py", line 72, in test_set_locale
    lang, enc = LOCALE_OVERRIDE.split('.')
ValueError: not enough values to unpack (expected 2, got 1)

this looks wrong

nbonnotte · 2016-09-05T17:02:16Z

@jreback I've set the locale to "C", which unlike "en_US.UTF-8" or "fr_FR.UTF-8" doesn't have any dot in it. Is it wrong? How should I set the system encoding to ascii?

Edit: I'll try with ANSI_X3.4-1968

jreback · 2016-09-05T17:21:39Z

@nbonnotte I have no idea what to put here

- LOCALE_OVERRIDE="zh_CN.UTF-8

looks like you need, country.encoding maybe? look thru python docs

jreback · 2016-09-05T17:23:24Z

http://stackoverflow.com/questions/19709026/how-can-i-list-all-available-windows-locales-in-python-console maybe

nbonnotte · 2016-09-05T18:00:44Z

ANSI_X3.4-1968 doesn't work:

  File "/home/travis/build/pydata/pandas/pandas/__init__.py", line 4, in <module>
    locale.setlocale(locale.LC_ALL, 'ANSI_X3.4-1968')
  File "/home/travis/miniconda/envs/pandas/lib/python3.5/locale.py", line 594, in setlocale
    return _setlocale(category, locale)
locale.Error: unsupported locale setting

What locale should we use?

nbonnotte · 2016-09-05T18:08:25Z

@jreback I have searched! The thing is, all xx_XX.encoding have encodings that can tolerate some accents: 8859-1, UTF-8... Still, I'm going to try with 8859-1, just in case.

I doubt zh_CN.UTF-8 would work, as I think it should set the encoding to utf-8, which is exactly what we need to avoid.

BTW, is it not wrong to assume that all locale settings match this pattern, xx_XX.encoding ? A user could set their locale to C. Maybe we can edit

  File "/home/travis/build/pydata/pandas/pandas/tools/tests/test_util.py", line 72, in test_set_locale
    lang, enc = LOCALE_OVERRIDE.split('.')

jreback · 2016-09-05T18:10:11Z

I was giving I am example of what other tests use

the locale utility is only for testing
u can change it to work with C I think

nbonnotte · 2016-09-05T19:42:07Z

$ ci/script.sh
inside ci/script.sh
ci/script.sh: line 13: warning: setlocale: LC_ALL: cannot change locale (en_US.iso88591): No such file or directory

Which locale are available, then?

jreback · 2016-09-05T19:51:20Z

no idea
maybe this: http://stackoverflow.com/questions/14547631/python-locale-error-unsupported-locale-setting

nbonnotte · 2016-09-06T04:57:47Z

@jreback is this an acceptable solution?

jreback · 2016-09-06T10:20:34Z

.travis.yml

+        - CLIPBOARD=xsel
+        - CACHE_NAME="35_ascii"
+        - USE_CACHE=true
+        addons:


you can take the CLIPBOARD and FULL_DEPS vars off both section

jreback · 2016-09-08T10:25:18Z

can you update & rebase

nbonnotte · 2016-09-10T06:07:25Z

@jreback all done!

As soon as the tests are confirmed, I'll correct the bug #12337

jreback · 2016-09-10T14:39:34Z

thanks!

ok if you'd submit on the fix for the issue would be great!

jreback added Bug Unicode Unicode strings IO LaTeX to_latex labels Aug 31, 2016

jreback reviewed Aug 31, 2016
View reviewed changes

nbonnotte changed the title ~~Test for Python 3.4 with C locale~~ Test for Python 3.5 with C locale Sep 3, 2016

jreback added this to the 0.19.0 milestone Sep 5, 2016

jreback reviewed Sep 6, 2016
View reviewed changes

nbonnotte added 3 commits September 9, 2016 07:38

Test for Python 3.4 with C locale

3b859ce

Travis conf: new test with python 3.5 and LC_ALL=C

3b4c6a5

New files requirements-3.5_ASCII.*

c825f86

nbonnotte added 2 commits September 9, 2016 07:38

Base matrix configuration

b876296

New tentative with C locale

dadf73c

jreback closed this in 1e61aed Sep 10, 2016

jreback mentioned this pull request Sep 21, 2016

TST: 3.5 c-locale #14275

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test for Python 3.5 with C locale #14114

Test for Python 3.5 with C locale #14114

nbonnotte commented Aug 28, 2016

codecov-io commented Aug 28, 2016 •

edited

Loading

nbonnotte commented Aug 29, 2016

jreback Aug 31, 2016

nbonnotte Sep 5, 2016

jreback commented Aug 31, 2016

nbonnotte commented Sep 3, 2016

jorisvandenbossche commented Sep 3, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016 •

edited

Loading

jreback commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 6, 2016

jreback Sep 6, 2016

jreback commented Sep 8, 2016 •

edited

Loading

nbonnotte commented Sep 10, 2016

jreback commented Sep 10, 2016

Test for Python 3.5 with C locale #14114

Test for Python 3.5 with C locale #14114

Conversation

nbonnotte commented Aug 28, 2016

codecov-io commented Aug 28, 2016 • edited Loading

Current coverage is 85.24% (diff: 100%)

nbonnotte commented Aug 29, 2016

jreback Aug 31, 2016

Choose a reason for hiding this comment

nbonnotte Sep 5, 2016

Choose a reason for hiding this comment

jreback commented Aug 31, 2016

nbonnotte commented Sep 3, 2016

jorisvandenbossche commented Sep 3, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016 • edited Loading

jreback commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 5, 2016

jreback commented Sep 5, 2016

nbonnotte commented Sep 6, 2016

jreback Sep 6, 2016

Choose a reason for hiding this comment

jreback commented Sep 8, 2016 • edited Loading

nbonnotte commented Sep 10, 2016

jreback commented Sep 10, 2016

codecov-io commented Aug 28, 2016 •

edited

Loading

nbonnotte commented Sep 5, 2016 •

edited

Loading

jreback commented Sep 8, 2016 •

edited

Loading