Skip to content

ENH: ISO8601-compliant datetime string conversion in iterrows() and Series construction. #19762

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 44 commits into from
Feb 25, 2018
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
518ab47
add require_iso8601 parameter and documentation in dataframe method i…
minggli Feb 19, 2018
156adbb
remove blank line
minggli Feb 19, 2018
6d06cf1
expose require_iso8601 parameter
minggli Feb 19, 2018
f2617dd
expose require_iso8601 parameter
minggli Feb 19, 2018
09ae4e5
expose require_8601 parameter
minggli Feb 19, 2018
7ea24ec
remove redundant TODO
minggli Feb 19, 2018
fac665b
revert pandas.core.frame
minggli Feb 20, 2018
068fde2
revert pandas.core.series
minggli Feb 20, 2018
8ceeb62
update documentation for typo and versionadded tag
minggli Feb 20, 2018
d105732
change default behaviour to require iso8601 and revert unnecessary ch…
minggli Feb 20, 2018
26fd14f
add whatsnew documentation for `require_iso8601` parameter in to_date…
minggli Feb 20, 2018
ab5214a
new test case in test_maybe_infer_to_datetimelike for non-iso8601 str…
minggli Feb 20, 2018
37aa8dd
comment with issue number
minggli Feb 20, 2018
7d9b27d
Merge branch 'master' into bugfixs/19671
minggli Feb 20, 2018
389a9d9
example for to_datetime api
minggli Feb 21, 2018
959ae62
reference to iso8601 standard
minggli Feb 22, 2018
700fa38
blank line before issue comment
minggli Feb 22, 2018
f8159c2
test datetime require iso8601 parameter
minggli Feb 22, 2018
3708f4b
add wikipedia reference to ISO 8601 standard
minggli Feb 22, 2018
cb798d2
add wikipedia reference to ISO 8601 standard
minggli Feb 22, 2018
2e27f22
fix url
minggli Feb 22, 2018
75268a8
Merge branch 'master' into bugfixs/19671
minggli Feb 22, 2018
8384d5e
private argument _require_iso8601 and remove example and param doc
minggli Feb 23, 2018
1665922
remove whatsnew entry
minggli Feb 23, 2018
21f7c15
modified kwarg
minggli Feb 23, 2018
5dc7a37
modified kwarg
minggli Feb 23, 2018
f9240b5
use kwargs to hide require_iso8601
minggli Feb 23, 2018
6e67070
revert core.tools.datetimes
minggli Feb 24, 2018
9e6e2a7
remove test case
minggli Feb 24, 2018
27fdfac
replace to_datetime call with internal conversion func
minggli Feb 24, 2018
6aea33d
revert test_tools
minggli Feb 24, 2018
998920c
Merge branch 'master' into PR_TOOL_MERGE_PR_19762
jreback Feb 24, 2018
14946f8
use DTI constructor
jreback Feb 24, 2018
9e11b43
test case for issue 19671, iterrows
minggli Feb 24, 2018
2fe7057
using klass for construction
minggli Feb 25, 2018
910f759
test DataFrame only
minggli Feb 25, 2018
0b72b72
fix a typo
minggli Feb 25, 2018
acdec06
Merge branch 'master' into PR_TOOL_MERGE_PR_19762
jreback Feb 25, 2018
e69f4ab
correction
jreback Feb 25, 2018
5b12cfc
fix test_iterrows
minggli Feb 25, 2018
a9d85ae
Merge remote-tracking branch 'upstream/master' into bugfixs/19671
minggli Feb 25, 2018
a5a1f57
whatsnew entry
minggli Feb 25, 2018
793ea23
imperative xfail in test
minggli Feb 25, 2018
08d2718
doc
jreback Feb 25, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.23.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -766,7 +766,6 @@ Datetimelike
- Bug in :func:`to_datetime` where passing an out-of-bounds datetime with ``errors='coerce'`` and ``utc=True`` would raise ``OutOfBoundsDatetime`` instead of parsing to ``NaT`` (:issue:`19612`)
- Bug in :class:`DatetimeIndex` and :class:`TimedeltaIndex` addition and subtraction where name of the returned object was not always set consistently. (:issue:`19744`)
- Bug in :class:`DatetimeIndex` and :class:`TimedeltaIndex` addition and subtraction where operations with numpy arrays raised ``TypeError`` (:issue:`19847`)
-

Timedelta
^^^^^^^^^
Expand Down Expand Up @@ -918,6 +917,7 @@ Reshaping
- :func:`Series.rename` now accepts ``axis`` as a kwarg (:issue:`18589`)
- Comparisons between :class:`Series` and :class:`Index` would return a ``Series`` with an incorrect name, ignoring the ``Index``'s name attribute (:issue:`19582`)
- Bug in :func:`qcut` where datetime and timedelta data with ``NaT`` present raised a ``ValueError`` (:issue:`19768`)
- Bug in :func:`DataFrame.iterrows`, which would infers strings not compliant to `ISO8601 <https://en.wikipedia.org/wiki/ISO_8601>`_ to datetimes (:issue:`19671`)

Other
^^^^^
Expand Down
17 changes: 12 additions & 5 deletions pandas/core/dtypes/cast.py
Original file line number Diff line number Diff line change
Expand Up @@ -904,16 +904,23 @@ def maybe_infer_to_datetimelike(value, convert_dates=False):
def try_datetime(v):
# safe coerce to datetime64
try:
v = tslib.array_to_datetime(v, errors='raise')
# GH19671
v = tslib.array_to_datetime(v,
require_iso8601=True,
errors='raise')
except ValueError:

# we might have a sequence of the same-datetimes with tz's
# if so coerce to a DatetimeIndex; if they are not the same,
# then these stay as object dtype
# then these stay as object dtype, xref GH19671
try:
from pandas import to_datetime
return to_datetime(v)
except Exception:
from pandas._libs.tslibs import conversion
from pandas import DatetimeIndex

values, tz = conversion.datetime_to_datetime64(v)
return DatetimeIndex(values).tz_localize(
'UTC').tz_convert(tz=tz)
except (ValueError, TypeError):
pass

except Exception:
Expand Down
4 changes: 2 additions & 2 deletions pandas/core/internals.py
Original file line number Diff line number Diff line change
Expand Up @@ -2602,8 +2602,8 @@ def _maybe_coerce_values(self, values):
"""Input validation for values passed to __init__. Ensure that
we have datetime64ns, coercing if necessary.

Parametetrs
-----------
Parameters
----------
values : array-like
Must be convertible to datetime64

Expand Down
4 changes: 4 additions & 0 deletions pandas/tests/dtypes/test_cast.py
Original file line number Diff line number Diff line change
Expand Up @@ -301,6 +301,10 @@ def test_maybe_infer_to_datetimelike(self):
[NaT, 'b', 1]]))
assert result.size == 6

# GH19671
result = Series(['M1701', Timestamp('20130101')])
assert result.dtype.kind == 'O'


class TestConvert(object):

Expand Down
15 changes: 14 additions & 1 deletion pandas/tests/frame/test_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@
from numpy.random import randn
import numpy as np

from pandas import DataFrame, Series, date_range, timedelta_range, Categorical
from pandas import (DataFrame, Series, date_range, timedelta_range,
Categorical, SparseDataFrame)
import pandas as pd

from pandas.util.testing import (assert_almost_equal,
Expand Down Expand Up @@ -214,6 +215,18 @@ def test_iterrows(self):
exp = self.mixed_frame.loc[k]
self._assert_series_equal(v, exp)

def test_iterrows_iso8601(self):
# GH19671
if self.klass == SparseDataFrame:
pytest.xfail(reason='SparseBlock datetime type not implemented.')

s = self.klass(
{'non_iso8601': ['M1701', 'M1802', 'M1903', 'M2004'],
'iso8601': date_range('2000-01-01', periods=4, freq='M')})
for k, v in s.iterrows():
exp = s.loc[k]
self._assert_series_equal(v, exp)

def test_itertuples(self):
for i, tup in enumerate(self.frame.itertuples()):
s = self.klass._constructor_sliced(tup[1:])
Expand Down