test failures and errors with v0.11rc1

[bmu]

I get one failure and 7 errors when I run the test suite on v0.11rc1.

This is the traceback of the error:

FAIL: testWLS (pandas.stats.tests.test_ols.TestOLS)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../pandas/pandas/stats/tests/test_ols.py", line 107, in testWLS
    self._check_wls(X, Y, weights)
  File "/.../pandas/pandas/stats/tests/test_ols.py", line 130, in _check_wls
    self.checkMovingOLS('rolling', x, y, weights=weights)
  File "/.../pandas/pandas/stats/tests/test_ols.py", line 204, in checkMovingOLS
    result_index=n)
  File "/.../pandas/pandas/stats/tests/test_ols.py", line 243, in compare
    assert_almost_equal(ref, res)
  File "/.../pandas/pandas/util/testing.py", line 139, in assert_almost_equal
    assert_almost_equal(a[i], b[i], check_less_precise)
  File "/.../pandas/pandas/util/testing.py", line 170, in assert_almost_equal
    1, a / b, decimal=decimal, err_msg=err_msg(a, b), verbose=False)
  File "/.../python2.7/site-packages/numpy/testing/utils.py", line 468, in assert_almost_equal
    raise AssertionError(msg)
AssertionError: 
Arrays are not almost equal to 5 decimals expected 1.62656 but got 1.62097

For test_axis_limits, test_line_plot_datetime_frame,
test_line_plot_datetime_series, test_line_plot_inferred_freq, test_line_plot_period_frame,
test_line_plot_period_series and test_tsplot I get a ValueError:

ValueError: matplotlib display text must have all code points < 128 or use Unicode strings

The reason for the errors seems to be my language setting, which is de_DE.UTF-8. I never got these errors in the pandas test suite before so maybe they are also related to the recent update to matplotlib 1.2.1.

print_versions.py 

INSTALLED VERSIONS
------------------
Python: 2.7.4.final.0
Cython: 0.18
Numpy: 1.7.1
Scipy: 0.12.0
statsmodels: 0.5.0.dev-1bbd4ca
    patsy: 0.1.0
scikits.timeseries: Not installed
dateutil: 2.1
pytz: 2013b
PyTables: 2.4.0
    numexpr: 2.0.1
matplotlib: 1.2.1
openpyxl: Not installed
xlrd: Not installed
xlwt: 0.7.4
sqlalchemy: 0.8.0

[ghost]

Can you update to master and paste the output of ci/print_versions.py?

[bmu]

@y-p You are to fast! I just finished the ticket and added print_version.py output.

[bmu]

I run the test suite on current master.

[ghost]

updating to recent statsmodels should take care of the WLS fail (it was fixed quite a while ago,
but no major release since then). I'll check out the others later.

[ghost]

I can't repro the mpl failures after setting $LANG and $LC_ALL, using mpl 1.2.1.
Can you see whether you get the same errors with 0.10.1 and mpl 1.2.1 vs 1.2.0?

[bmu]

Seems to be a strange problem: if I call test.sh I get the unicode errors, if I run nosetests pandas.tseries.tests.test_plotting or python pandas/tseries/tests/test_plotting.py all tests are passed without any errors.

statsmodels 0.5.0.dev-1bbd4ca is the version which is installed when you run pip install -U statsmodels, so maybe it is better to skip the test, even if this is fixed now?

[ghost]

I've pushed an update this morning, if the test fails now it spits out an error message
specifying the minimum commit needed. Hopefully that's enough to point users in the right
direction. The test failure does let users know they need to upgrade, that's a good thing.

can't repro with test.sh either.

can you post the value of pd.options.display.encoding
and put a print pd.options.display.encoding in one of the failing tests
to see what it reports?

[bmu]

It is UTF-8.

I don't get any errors with 0.10.1 and mpl 1.2.0 or mpl 1.2.1.

[ghost]

Please post a full backtrace for one of the failing mpl tests.
If possible, try to use nose's --pdb(-fail) option to get the string
value that's making mpl yell.

[bmu]

I figured it out a bit more: the errors disappear if I set backend: Agg in my matplotlibrc instead of Qt4Agg which I use normally. So maybe there is something wrong with the test setup (normally the backend is set to Agg in the setup of TestTSPlot, not sure why this doesn't seems to work for a few tests).

The full traceback of test_axis_limits is

======================================================================
ERROR: test_axis_limits (pandas.tseries.tests.test_plotting.TestTSPlot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../pandas/pandas/tseries/tests/test_plotting.py", line 353, in test_axis_limits
    _test(ax)
  File "/.../pandas/pandas/tseries/tests/test_plotting.py", line 334, in _test
    ax.get_figure().canvas.draw()
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 148, in draw
    FigureCanvasAgg.draw(self)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/backends/backend_agg.py", line 440, in draw
    self.figure.draw(self.renderer)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/figure.py", line 1006, in draw
    func(*args)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
    a.draw(renderer)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1052, in draw
    renderer)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/axis.py", line 1001, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/text.py", line 752, in get_window_extent
    bbox, info = self._get_layout(self._renderer)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/text.py", line 309, in _get_layout
    clean_line, ismath = self.is_math_text(line)
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/text.py", line 994, in is_math_text
    if cbook.is_math_text(s):
  File "/.../.virtualenvs/myenv/local/lib/python2.7/site-packages/matplotlib/cbook.py", line 1797, in is_math_text
    "matplotlib display text must have all code points < 128 or use "
ValueError: matplotlib display text must have all code points < 128 or use Unicode strings

and for the other tests it is

======================================================================
ERROR: test_line_plot_datetime_frame (pandas.tseries.tests.test_plotting.TestTSPlot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/.../pandas/pandas/tseries/tests/test_plotting.py", line 175, in test_line_plot_datetime_frame
    _check_plot_works(df.plot, freq)
  File "/.../pandas/pandas/tseries/tests/test_plotting.py", line 988, in _check_plot_works
    plt.savefig(path)
  File "/.../python2.7/site-packages/matplotlib/pyplot.py", line 472, in savefig
    return fig.savefig(*args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/figure.py", line 1370, in savefig
    self.canvas.print_figure(*args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/backends/backend_qt4agg.py", line 161, in print_figure
    FigureCanvasAgg.print_figure(self, *args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/backend_bases.py", line 2096, in print_figure
    **kwargs)
  File "/.../python2.7/site-packages/matplotlib/backends/backend_agg.py", line 492, in print_png
    FigureCanvasAgg.draw(self)
  File "/.../python2.7/site-packages/matplotlib/backends/backend_agg.py", line 440, in draw
    self.figure.draw(self.renderer)
  File "/.../python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/figure.py", line 1006, in draw
    func(*args)
  File "/.../python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/axes.py", line 2086, in draw
    a.draw(renderer)
  File "/.../python2.7/site-packages/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/.../python2.7/site-packages/matplotlib/axis.py", line 1052, in draw
    renderer)
  File "/.../python2.7/site-packages/matplotlib/axis.py", line 1001, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "/.../python2.7/site-packages/matplotlib/text.py", line 752, in get_window_extent
    bbox, info = self._get_layout(self._renderer)
  File "/.../python2.7/site-packages/matplotlib/text.py", line 309, in _get_layout
    clean_line, ismath = self.is_math_text(line)
  File "/.../python2.7/site-packages/matplotlib/text.py", line 994, in is_math_text
    if cbook.is_math_text(s):
  File "/.../python2.7/site-packages/matplotlib/cbook.py", line 1797, in is_math_text
    "matplotlib display text must have all code points < 128 or use "
ValueError: matplotlib display text must have all code points < 128 or use Unicode strings

(the difference is only in the first lines, the error occurs in _check_plot_works)

[ghost]

mpl and unicode now sounds related to #3360 , can you run through that and see if it fits?

specifically, are you getting the same behaviour as in this comment?

[bmu]

I cannot reproduce the behaviour in this comment:

>>> import sys
>>> sys.getdefaultencoding()
'ascii'
>>> import matplotlib.pyplot as plt
>>> sys.getdefaultencoding()
'ascii'
>>> import matplotlib as mpl
>>> mpl.__version__
'1.2.1'

[ghost]

That's with Qt4Agg as the backend in .mplrc?

[bmu]

Yes

[ghost]

I can't replicate with backend = Qt4Agg in my .mplrc

can you break in with pdb (nosetests --pdb) and verify that at the point of failure
the active backend is indeed != 'Agg'? matplotlib.get_backend().

Also, just crudely, does it go away if you place the backend init code
at the package fixture level?

Put the following in /pandas/tools/tests/init.py

   def setUpPackage():
        try:
            import matplotlib as mpl
            mpl.use('Agg', warn=False)
        except ImportError:
            raise nose.SkipTest

[bmu]

Seems that the problem is, that the xlables are of type string, not unicode. They are unicode if I use plt.plot_date:

In [64]: ser = tm.makeTimeSeries()

In [65]: fig, ax = plt.subplots()

In [66]: ax.plot_date(ser.index, ser)
Out[66]: [<matplotlib.lines.Line2D at 0x57c1090>]

In [67]: xlabels = ax.get_xmajorticklabels()

In [68]: for label in xlabels:
    print type(label.get_text())
   ....:     
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>
<type 'unicode'>

In [69]: plt.close()

In [70]: ax = ser.plot()

In [71]: xlabels = ax.get_xmajorticklabels()

In [72]: for label in xlabels:
    print type(label.get_text())
   ....:     
<type 'str'>
<type 'str'>
<type 'str'>

[bmu]

This is a simple example: if I run it with export LANG=en_GB.UTF8 before ipython --pylab it runs fine, if I run it with export LANG=de_DE.UTF8 it fails:

In [1]: idx = pd.date_range('2013-01-01', '2013-04-30')

In [2]: s = pd.Series(np.random.rand(idx.size), index=idx)

In [3]: s.plot()
Out[3]: <matplotlib.axes.AxesSubplot at 0x338c510>
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_qt4.py", line 366, in idle_draw
    self.draw()
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_qt4agg.py", line 148, in draw
    FigureCanvasAgg.draw(self)
  File "/usr/lib/pymodules/python2.7/matplotlib/backends/backend_agg.py", line 440, in draw
    self.figure.draw(self.renderer)
  File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/figure.py", line 1006, in draw
    func(*args)
  File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/axes.py", line 2086, in draw
    a.draw(renderer)
  File "/usr/lib/pymodules/python2.7/matplotlib/artist.py", line 54, in draw_wrapper
    draw(artist, renderer, *args, **kwargs)
  File "/usr/lib/pymodules/python2.7/matplotlib/axis.py", line 1052, in draw
    renderer)
  File "/usr/lib/pymodules/python2.7/matplotlib/axis.py", line 1001, in _get_tick_bboxes
    extent = tick.label1.get_window_extent(renderer)
  File "/usr/lib/pymodules/python2.7/matplotlib/text.py", line 752, in get_window_extent
    bbox, info = self._get_layout(self._renderer)
  File "/usr/lib/pymodules/python2.7/matplotlib/text.py", line 309, in _get_layout
    clean_line, ismath = self.is_math_text(line)
  File "/usr/lib/pymodules/python2.7/matplotlib/text.py", line 994, in is_math_text
    if cbook.is_math_text(s):
  File "/usr/lib/pymodules/python2.7/matplotlib/cbook.py", line 1797, in is_math_text
    "matplotlib display text must have all code points < 128 or use "
ValueError: matplotlib display text must have all code points < 128 or use Unicode strings

So I don't think it is the test setup. Can you reproduce this?

[ghost]

yep. That repros here. good.

[ghost]

well, it does fail (silently) in qtconsole, but not in the terminal. so maybe it repros.

[ghost]

Do the labels contain non-ascii when you usually run? is february a 16 letter word with 3 umlauts
in german for example?

[bmu]

March is a month with an umlaut (März).

[bmu]

Seems that if pandas.tseries.plotting.TimeSeries_DateFormatter is called a Period.strftime(...) is returned. So Period.strftime doesn't returnes unicode.

[bmu]

if I change line 973 in tseries.plotting to return (Period(ordinal=int(x), freq=self.freq).strftime(fmt)).decode('utf-8')
it works (however I think this isn't the right place to change it)

[ghost]

Good, close enough. I think you mean tseries.converter though.

[bmu]

yes tseries.converter, sorry.

[ghost]

Ok, looks like ultimately the date formatting gets done in a c routine,
so probably just need to box the result back up as unicode.

I'll see if I can put this together later on today.

Good job tracking it down. 👍

[ghost]

can you give #3410 a whirl?

I was able to repro everything you noted, and that fixes it for me.

[bmu]

I works! really good job removing the errors 👍

[ghost]

Merged. Thanks for reporting and all the subsequent digging, you
made it very easy to fix, so thank you.