Skip to content

BUG: datetime.date objects don't get parsed for non-ISO formats #50109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 11, 2022
Merged

BUG: datetime.date objects don't get parsed for non-ISO formats #50109

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b707682
parse datetime.date in non-ISO path
Dec 7, 2022
24d9e0b
reword
Dec 7, 2022
d04a8b3
Merge remote-tracking branch 'upstream/main' into parse-date-in-strptime
Dec 7, 2022
0a5633b
Merge branch 'main' into parse-date-in-strptime
MarcoGorelli Dec 7, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v2.0.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -698,7 +698,7 @@ Datetimelike
- Bug in subtracting a ``datetime`` scalar from :class:`DatetimeIndex` failing to retain the original ``freq`` attribute (:issue:`48818`)
- Bug in ``pandas.tseries.holiday.Holiday`` where a half-open date interval causes inconsistent return types from :meth:`USFederalHolidayCalendar.holidays` (:issue:`49075`)
- Bug in rendering :class:`DatetimeIndex` and :class:`Series` and :class:`DataFrame` with timezone-aware dtypes with ``dateutil`` or ``zoneinfo`` timezones near daylight-savings transitions (:issue:`49684`)
- Bug in :func:`to_datetime` was raising ``ValueError`` when parsing :class:`Timestamp`, ``datetime``, or ``np.datetime64`` objects with non-ISO8601 ``format`` (:issue:`49298`, :issue:`50036`)
- Bug in :func:`to_datetime` was raising ``ValueError`` when parsing :class:`Timestamp`, ``datetime.datetime``, ``datetime.date``, or ``np.datetime64`` objects with non-ISO8601 ``format`` (:issue:`49298`, :issue:`50036`)
-

Timedelta
Expand Down
6 changes: 6 additions & 0 deletions pandas/_libs/tslibs/strptime.pyx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
"""Strptime-related classes and functions.
"""
from cpython.datetime cimport (
PyDate_Check,
PyDateTime_Check,
date,
import_datetime,
Expand Down Expand Up @@ -34,6 +35,7 @@ from pandas._libs.tslibs.np_datetime cimport (
check_dts_bounds,
npy_datetimestruct,
npy_datetimestruct_to_datetime,
pydate_to_dt64,
pydatetime_to_dt64,
)
from pandas._libs.tslibs.timestamps cimport _Timestamp
Expand Down Expand Up @@ -173,6 +175,10 @@ def array_strptime(
check_dts_bounds(&dts)
result_timezone[i] = val.tzinfo
continue
elif PyDate_Check(val):
iresult[i] = pydate_to_dt64(val, &dts)
check_dts_bounds(&dts)
continue
Comment on lines +178 to +181
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as

pandas/pandas/_libs/tslib.pyx

Lines 544 to 547 in 7c0278e

elif PyDate_Check(val):
seen_datetime = True
iresult[i] = pydate_to_dt64(val, &dts)
check_dts_bounds(&dts)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to worry about found_naive? (i guess we dont currently in the tslib code?)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure we need to, datetime.dates can't have a timezone, so perhaps it's fine to just parse?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK to punt on this for now

elif is_datetime64_object(val):
iresult[i] = get_datetime64_nanos(val, NPY_FR_ns)
continue
Expand Down
11 changes: 11 additions & 0 deletions pandas/tests/tools/test_to_datetime.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import calendar
from collections import deque
from datetime import (
date,
datetime,
timedelta,
timezone,
Expand Down Expand Up @@ -469,6 +470,16 @@ def test_to_datetime_mixed_datetime_and_string(self):
expected = to_datetime([d1, d2]).tz_convert(pytz.FixedOffset(-60))
tm.assert_index_equal(res, expected)

@pytest.mark.parametrize(
"format", ["%Y-%m-%d", "%Y-%d-%m"], ids=["ISO8601", "non-ISO8601"]
)
def test_to_datetime_mixed_date_and_string(self, format):
# https://github.com/pandas-dev/pandas/issues/50108
d1 = date(2020, 1, 2)
res = to_datetime(["2020-01-01", d1], format=format)
expected = DatetimeIndex(["2020-01-01", "2020-01-02"])
tm.assert_index_equal(res, expected)

@pytest.mark.parametrize(
"fmt",
["%Y-%d-%m %H:%M:%S%z", "%Y-%m-%d %H:%M:%S%z"],
Expand Down