Skip to content

DEPR: passing mixed offsets with utc=False into to_datetime #54014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
bcbe2ac
add raising FutureWarning in _return_parsed_timezone_results and in a…
natmokval Jul 5, 2023
ba99e93
correct the definition of _return_parsed_timezone_results, added a te…
natmokval Jul 5, 2023
441e17f
fix tests in pandas/tests/extension/test_arrow.py
natmokval Jul 6, 2023
5e568bd
correct the definition of _return_parsed_timezone_results
natmokval Jul 7, 2023
2aa5c10
fix an exanple in docs: Parsing a CSV with mixed timezones
natmokval Jul 7, 2023
8197005
correct def _array_to_datetime_object, add a test for mixed format, f…
natmokval Jul 7, 2023
ca4b214
correct example in whatsnew/v0.24.0.rst and fix pylint failures
natmokval Jul 7, 2023
a0e970a
correct str for message in FutureWarning in the test with format mixed
natmokval Jul 7, 2023
156ea8a
fix an error in an example in whatsnew/v0.24.0.rst
natmokval Jul 7, 2023
643d3d6
correct examples and the description of the param utc in docstring o…
natmokval Jul 8, 2023
61d4deb
update whatsnew/v2.1.0.rst
natmokval Jul 9, 2023
f2bbedf
correct docstring for to_datetime, example in whatsnew/v0.24.0.rst, r…
natmokval Jul 10, 2023
5c4904a
refactor tests for to_datetime
natmokval Jul 13, 2023
f5e5e1e
refactor test for to_datetime
natmokval Jul 14, 2023
88f3845
add example to whatsnew/v2.1.0.rst
natmokval Jul 17, 2023
3d74972
correct the example
natmokval Jul 17, 2023
88ed6c1
correct the example in whatsnew/v2.1.0.rst
natmokval Jul 18, 2023
180ccce
Merge branch 'main' into DEPR-to_datetime-mixed-offsets-with-utc=False
natmokval Jul 19, 2023
6ecd997
correct def _array_to_datetime_object and fix test for read_json
natmokval Jul 20, 2023
dc7c54d
add catch_warnings to filter the warning in test_read_datetime
natmokval Jul 24, 2023
b5bbd2b
correct msg in catch_warnings to filter the warning in test_read_date…
natmokval Jul 24, 2023
1220130
Merge branch 'main' into DEPR-to_datetime-mixed-offsets-with-utc=False
natmokval Jul 25, 2023
0549e6d
Merge branch 'main' into DEPR-to_datetime-mixed-offsets-with-utc=False
natmokval Jul 25, 2023
b01c3ef
catch the warning in test_from_csv_with_mixed_offsets
natmokval Jul 25, 2023
04ef036
reword whatsnew
MarcoGorelli Jul 26, 2023
c42f143
add catch_warnings to converter
natmokval Jul 26, 2023
5e5adc6
Merge branch 'main' into DEPR-to_datetime-mixed-offsets-with-utc=False
natmokval Jul 26, 2023
3aa091f
Merge branch 'DEPR-to_datetime-mixed-offsets-with-utc=False' of https…
natmokval Jul 26, 2023
acee8de
describe how to maintain the old behavior
natmokval Jul 27, 2023
b7a2207
add an example how to get the old behavior and : correct the warning…
natmokval Jul 27, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/whatsnew/v0.24.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -643,7 +643,7 @@ Parsing datetime strings with different UTC offsets will now create an Index of
Passing ``utc=True`` will mimic the previous behavior but will correctly indicate
that the dates have been converted to UTC

.. ipython:: python
.. code-block:: ipython

pd.to_datetime(["2015-11-18 15:30:00+05:30",
"2015-11-18 16:30:00+06:30"], utc=True)
Expand Down
19 changes: 11 additions & 8 deletions pandas/_libs/tslib.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -561,14 +561,6 @@ cpdef array_to_datetime(
# (with individual dateutil.tzoffsets) are returned
is_same_offsets = len(out_tzoffset_vals) == 1
if not is_same_offsets:
warnings.warn(
"In a future version of pandas, parsing datetimes with mixed time "
"zones will raise a warning unless `utc=True`. "
"Please specify `utc=True to opt in to the new behaviour "
"and silence this warning.",
FutureWarning,
stacklevel=find_stack_level(),
)
return _array_to_datetime_object(values, errors, dayfirst, yearfirst)
else:
tz_offset = out_tzoffset_vals.pop()
Expand Down Expand Up @@ -628,6 +620,7 @@ cdef _array_to_datetime_object(
# 1) NaT or NaT-like values
# 2) datetime strings, which we return as datetime.datetime
# 3) special strings - "now" & "today"
unique_timezones = set()
for i in range(n):
# Analogous to: val = values[i]
val = <object>(<PyObject**>cnp.PyArray_MultiIter_DATA(mi, 1))[0]
Expand Down Expand Up @@ -657,6 +650,7 @@ cdef _array_to_datetime_object(
tzinfo=tsobj.tzinfo,
fold=tsobj.fold,
)
unique_timezones.add(tsobj.tzinfo)

except (ValueError, OverflowError) as ex:
ex.args = (f"{ex}, at position {i}", )
Expand All @@ -674,6 +668,15 @@ cdef _array_to_datetime_object(

cnp.PyArray_MultiIter_NEXT(mi)

if len(unique_timezones) > 1:
warnings.warn(
"In a future version of pandas, parsing datetimes with mixed time "
"zones will raise a warning unless `utc=True`. "
Comment on lines +673 to +674
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The message here says that this will raise a warning in the future. But is that indeed the intent, or should that be "error" instead?
(my understanding of the discussion was that it would error in the future)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're right, thanks - @natmokval fancy addressing this in a separate PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure, I replaced "warning" with "error" in warning message and made a new PR

"Please specify `utc=True to opt in to the new behaviour "
"and silence this warning.",
FutureWarning,
stacklevel=find_stack_level(),
)
return oresult_nd, None


Expand Down
24 changes: 24 additions & 0 deletions pandas/tests/tools/test_to_datetime.py
Original file line number Diff line number Diff line change
Expand Up @@ -3680,3 +3680,27 @@ def test_from_numeric_arrow_dtype(any_numeric_ea_dtype):
result = to_datetime(ser)
expected = Series([1, 2], dtype="datetime64[ns]")
tm.assert_series_equal(result, expected)


@pytest.mark.parametrize(
"date, date_expected, warning",
[
(["2020-01-01 00:00+00:00", "2020-01-01 00:00+02:00", ""], None, FutureWarning),
(
["2020-01-01 00:00+00:00", ""],
[Timestamp("2020-01-01 00:00+00:00"), "NaT"],
None,
),
],
)
def test_to_datetime_with_empty_str_utcFalse_format_mixed(date, date_expected, warning):
msg = "In a future version of pandas, parsing datetimes with mixed time zones "
"will raise a warning unless `utc=True`."

if warning is not None:
with tm.assert_produces_warning(warning, match=msg):
to_datetime(date, format="mixed")
else:
result = to_datetime(date, format="mixed")
expected = Index(date_expected, dtype=object)
tm.assert_index_equal(result, expected)