Skip to content

test_constructor_compound_dtypes and test_invalid_index_types fail in parse_datetime_string #11169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
yarikoptic opened this issue Sep 22, 2015 · 4 comments
Closed

test_constructor_compound_dtypes and test_invalid_index_types fail in parse_datetime_string #11169

yarikoptic opened this issue Sep 22, 2015 · 4 comments
Labels
Testing pandas testing functions or related to the test suite
Milestone
0.19.0

Comments

@yarikoptic
Copy link
Contributor

Debian stretch/sid, cython 0.22.1-2, pandas v0.17.0rc1-92-gc6bcc99

$> locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

nosetest pandas leads to

$> nosetests -s -v --failed pandas --pdb
#1982 test_invalid_index_types (pandas.tseries.tests.test_frequencies.TestFrequencyInference) ... > /usr/lib/python2.7/dist-packages/dateutil/parser.py(310)parse()
-> res, skipped_tokens = self._parse(timestr, **kwargs)
(Pdb) p timestr
u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df'
(Pdb) self._parse(timestr, **kwargs)
(Pdb) up
> /usr/lib/python2.7/dist-packages/dateutil/parser.py(748)parse()
-> return DEFAULTPARSER.parse(timestr, **kwargs)
(Pdb) p timestr
u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df'
(Pdb) p DEFAULTPARSER
<dateutil.parser.parser object at 0x7f79dcacf090>
*(Pdb) p timestr.encode()
*** UnicodeEncodeError: UnicodeEncodeError('ascii', u'\u05e4\u05d0\u05e3\u05df\u05e1\u05da\u05e8\u05e1\u05e7\u05df', 0, 10, 'ordinal not in range(128)')
*(Pdb) p timestr.encode('utf-8')
'\xd7\xa4\xd7\x90\xd7\xa3\xd7\x9f\xd7\xa1\xd7\x9a\xd7\xa8\xd7\xa1\xd7\xa7\xd7\x9f'
(Pdb) 
ERROR
#3617 test_constructor_compound_dtypes (pandas.tests.test_frame.TestDataFrame) ... > /usr/lib/python2.7/dist-packages/dateutil/parser.py(310)parse()
-> res, skipped_tokens = self._parse(timestr, **kwargs)
(Pdb) 
ERROR

======================================================================
ERROR: test_invalid_index_types (pandas.tseries.tests.test_frequencies.TestFrequencyInference)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tests/test_frequencies.py", line 565, in test_invalid_index_types
    self.assertRaises(ValueError, lambda : frequencies.infer_freq(i))
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/testing.py", line 1880, in assertRaises
    _callable(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tests/test_frequencies.py", line 565, in <lambda>
    self.assertRaises(ValueError, lambda : frequencies.infer_freq(i))
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/frequencies.py", line 855, in infer_freq
    index = pd.DatetimeIndex(index)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/index.py", line 315, in __new__
    subarr = tools.to_datetime(data, box=False)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 276, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 395, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 372, in _convert_listlike
    require_iso8601=require_iso8601)
  File "pandas/tslib.pyx", line 1828, in pandas.tslib.array_to_datetime (pandas/tslib.c:34125)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2017, in pandas.tslib.array_to_datetime (pandas/tslib.c:33760)
    raise
  File "pandas/tslib.pyx", line 2011, in pandas.tslib.array_to_datetime (pandas/tslib.c:33658)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1602, in pandas.tslib.parse_datetime_string (pandas/tslib.c:27245)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

======================================================================
ERROR: test_constructor_compound_dtypes (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_frame.py", line 14395, in test_constructor_compound_dtypes
    f('M8[ns]')
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tests/test_frame.py", line 14384, in f
    columns=["A", "B", "C"], dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 264, in __init__
    arrays, columns = _to_arrays(data, columns, dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5216, in _to_arrays
    dtype=dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5299, in _list_to_arrays
    coerce_float=coerce_float)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5366, in _convert_object_array
    arrays = [ convert(arr) for arr in content ]
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/frame.py", line 5363, in convert
    arr = com._possibly_cast_to_datetime(arr, dtype)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/core/common.py", line 1998, in _possibly_cast_to_datetime
    value = to_datetime(value, errors=errors)._values
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/util/decorators.py", line 89, in wrapper
    return func(*args, **kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 276, in to_datetime
    unit=unit, infer_datetime_format=infer_datetime_format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 395, in _to_datetime
    return _convert_listlike(arg, box, format)
  File "/home/yoh/deb/gits/pkg-exppsy/pandas/pandas/tseries/tools.py", line 372, in _convert_listlike
    require_iso8601=require_iso8601)
  File "pandas/tslib.pyx", line 1828, in pandas.tslib.array_to_datetime (pandas/tslib.c:34125)
    cpdef array_to_datetime(ndarray[object] values, errors='raise',
  File "pandas/tslib.pyx", line 2017, in pandas.tslib.array_to_datetime (pandas/tslib.c:33760)
    raise
  File "pandas/tslib.pyx", line 2011, in pandas.tslib.array_to_datetime (pandas/tslib.c:33658)
    oresult[i] = parse_datetime_string(val, dayfirst=dayfirst,
  File "pandas/tslib.pyx", line 1602, in pandas.tslib.parse_datetime_string (pandas/tslib.c:27245)
    dt = parse_date(date_string, default=_DEFAULT_DATETIME,
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 748, in parse
    return DEFAULTPARSER.parse(timestr, **kwargs)
  File "/usr/lib/python2.7/dist-packages/dateutil/parser.py", line 310, in parse
    res, skipped_tokens = self._parse(timestr, **kwargs)
TypeError: 'NoneType' object is not iterable

----------------------------------------------------------------------
Ran 2 tests in 59.805s
@jreback
Copy link
Contributor

jreback commented Sep 22, 2015

yeh, the we these on windows because its actually not giving us the correct error (its in a weird encoding).

I never could figure out why this works on linux/osx but not elsewhere.

@yarikoptic maybe the same issue.

@sinhrks

@yarikoptic
Copy link
Contributor Author

The same as ... ? in my case it is as non-windows as it gets (Debian GNU/Linux ;))

@jreback
Copy link
Contributor

jreback commented Sep 22, 2015

what I mean is that we skip these on windows (never could really figure it out), yet it works on our travis builds (linux) and macosx. So something is different. I think its some kind of encoding issue.

@jreback jreback added the Testing pandas testing functions or related to the test suite label Sep 22, 2015
yarikoptic added a commit to neurodebian/pandas that referenced this issue Oct 11, 2015
@yarikoptic
Started to trigger failures of test_constructor_compound_dtypes and t…
fa35488 
…est_invalid_index_types -- disabled those for now, see pandas-dev#11169
@FRidh
Copy link

FRidh commented Oct 26, 2015

Both also fail with Nix. See #11287.

@sinhrks sinhrks mentioned this issue Jul 18, 2016
Closed
4 tasks
@sinhrks sinhrks added this to the 0.19.0 milestone Jul 18, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Testing pandas testing functions or related to the test suite
Projects
None yet
Milestone
0.19.0
Development

Successfully merging a pull request may close this issue.

PERF: improve DTI string parse sinhrks/pandas
4 participants
@yarikoptic @jreback @sinhrks @FRidh