pandas.io.tests.test_parsers.Test*.test_yy_format fails in 0.18.0

[neirbowj]

Code Sample, a copy-pastable example if possible

$ nosetests-2.7 pandas.io.tests.test_parsers:TestCParserHighMemory.test_yy_format
F
======================================================================
FAIL: test_yy_format (pandas.io.tests.test_parsers.TestCParserHighMemory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/pandas/io/tests/test_parsers.py", line 1064, in test_yy_format
    tm.assert_frame_equal(rs, xp)
  File "/usr/local/lib/python2.7/site-packages/pandas/util/testing.py", line 1097, in assert_frame_equal
    obj='{0}.index'.format(obj))
  File "/usr/local/lib/python2.7/site-packages/pandas/util/testing.py", line 729, in assert_index_equal
    obj=obj, lobj=left, robj=right)
  File "pandas/src/testing.pyx", line 58, in pandas._testing.assert_almost_equal (pandas/src/testing.c:3809)
  File "pandas/src/testing.pyx", line 147, in pandas._testing.assert_almost_equal (pandas/src/testing.c:2685)
  File "/usr/local/lib/python2.7/site-packages/pandas/util/testing.py", line 880, in raise_assert_detail
    raise AssertionError(msg)
AssertionError: DataFrame.index are different

DataFrame.index values are different (100.0 %)
[left]:  DatetimeIndex(['2031-09-01 00:10:00', '2028-09-02 10:20:00',
               '2031-09-03 08:30:00'],
              dtype='datetime64[ns]', name=u'date_time', freq=None)
[right]: DatetimeIndex(['2009-01-31 00:10:00', '2009-02-28 10:20:00',
               '2009-03-31 08:30:00'],
              dtype='datetime64[ns]', name=u'date_time', freq=None)

----------------------------------------------------------------------
Ran 1 test in 0.085s

FAILED (failures=1)

Also pandas.io.tests.test_parsers:TestCParserLowMemory.test_yy_format and pandas.io.tests.test_parsers:TestPythonParser.test_yy_format.

This appears to be in the class of bugs described in #3341.

$ python
Python 2.7.11 (default, Jan  9 2016, 17:23:07)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import dateutil.parser
>>> p = dateutil.parser.parser()
>>> p.parse('090131', dayfirst=False, yearfirst=False)
datetime.datetime(2031, 9, 1, 0, 0)
>>> p.parse('090131', dayfirst=False, yearfirst=True)
datetime.datetime(2009, 1, 31, 0, 0)
>>> p.parse('090131', dayfirst=True, yearfirst=False)
datetime.datetime(2031, 1, 9, 0, 0)
>>> p.parse('090131', dayfirst=True, yearfirst=True)
datetime.datetime(2009, 1, 31, 0, 0)
>>>

Expected Output

Expect tests to pass.

output of pd.show_versions()

This is reproducible with both of the following configurations.

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.11.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.2-STABLE
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.0
nose: 1.3.7
pip: 8.0.2
setuptools: 20.0
Cython: None
numpy: 1.10.4
scipy: 0.16.1
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.0
pytz: 2015.7
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.5
matplotlib: 1.5.0
openpyxl: 2.3.3
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.8.4
lxml: 3.5.0
bs4: 4.4.1
html5lib: 0.9999999
httplib2: None
apiclient: None
sqlalchemy: 0.7.10
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.38.0

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.4.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.2-STABLE
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.0
nose: 1.3.7
pip: None
setuptools: 20.0
Cython: 0.23.4
numpy: 1.10.4
scipy: 0.16.1
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.0
pytz: 2015.7
blosc: None
bottleneck: 1.0.0
tables: 3.2.2
numexpr: 2.5
matplotlib: None
openpyxl: 2.3.3
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.8.4
lxml: 3.5.0
bs4: 4.4.1
html5lib: 0.9999999
httplib2: None
apiclient: None
sqlalchemy: 0.7.10
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.38.0

[jreback]

looks like a but in dateutil, not shocked. pls open an issue there. We are testing with several older versions of dateutil, but not the newest, as we just conda install (where its at the stable 2.4.1). Ok will use this issue to add some testing. Not really sure what we can do though about fixing this.

In [1]: import dateutil.parser

In [2]: p = dateutil.parser.parser()

In [3]: p.parse(StringIO('090131'), dayfirst=False, yearfirst=False)
Out[3]: datetime.datetime(2009, 1, 31, 0, 0)

In [4]: dateutil.__version__
Out[4]: '2.4.1'

In [1]: import dateutil.parser
In [3]: >>> p = dateutil.parser.parser()

In [4]: p.parse(StringIO('090131'), dayfirst=False, yearfirst=False)
Out[4]: datetime.datetime(2031, 9, 1, 0, 0)

In [5]: import dateutil

In [6]: dateutil.__version__
Out[6]: '2.5.0'

xref #12585 (though my suggested fix actually would work in this case I think, if you want to take a crack at it). Ideally we would reduce dependency on dateutil

[jreback]

@gfyoung can you see if this is still an issue?

[gfyoung]

Not really sure what you would like me to confirm here. I haven't seen any test failures relating to this as of late.

[jreback]

actually this is fine. old versions of dateutil are borked, but newer are fine.