Skip to content

BUG: Fix issue with datetime[ns, tz] input in Block.setitem GH32395 #32479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 25 commits into from
May 25, 2020
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
198474d
BUG: Fix issue with datetime[ns, tz] input in Block.setitem GH32395
h-vishal Mar 5, 2020
6eba1c9
Added tests for series, moved tests to test_loc.py
h-vishal Mar 6, 2020
11b4f29
Add dtype to series object in tests to suppress warning
h-vishal Mar 6, 2020
1b3cba1
Fix whatsnew
h-vishal Mar 6, 2020
d83cff9
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Mar 6, 2020
4e66228
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Mar 7, 2020
ac85aa3
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Mar 9, 2020
2d25c7a
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Mar 27, 2020
3e78985
Move whatsnew to 1.1.0
h-vishal Mar 27, 2020
7e600c7
Genralise fix for other extension array types
h-vishal Apr 2, 2020
05a788c
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Apr 2, 2020
b136ce3
Add new whatsnew
h-vishal Apr 3, 2020
d433d7c
Fix doc
h-vishal Apr 3, 2020
00199f6
Fix version number in whatsnew
h-vishal Apr 3, 2020
01f01fd
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Apr 3, 2020
8cfa045
TST: Fix precision test in tests.computation.test_eval.check_alignment()
h-vishal Apr 3, 2020
4f8fccd
Revert "TST: Fix precision test in
h-vishal Apr 4, 2020
0f8a913
CLN: Test style tests.indexing.test_loc
h-vishal Apr 4, 2020
9d4d1df
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Apr 4, 2020
a01a676
Modify tests after review
h-vishal Apr 6, 2020
26f8ed3
Merge remote-tracking branch 'remotes/upstream/master' into issue-32395
h-vishal Apr 6, 2020
3caf161
Merge remote-tracking branch 'upstream/master' into issue-32395
simonjayhawkins May 7, 2020
8d9e6d7
Merge remote-tracking branch 'upstream/master' into issue-32395
simonjayhawkins May 20, 2020
fbc31c8
nits
simonjayhawkins May 20, 2020
c7e7cf5
update tests
simonjayhawkins May 20, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/whatsnew/v1.0.2.rst
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ Fixed regressions
- Fixed regression where :func:`read_pickle` raised a ``UnicodeDecodeError`` when reading a py27 pickle with :class:`MultiIndex` column (:issue:`31988`).
- Fixed regression in :class:`DataFrame` arithmetic operations with mis-matched columns (:issue:`31623`)
- Fixed regression in :meth:`GroupBy.agg` calling a user-provided function an extra time on an empty input (:issue:`31760`)
- Fixed regression in :meth:`DataFrame.loc`, :meth:`Series.loc` throwing an error when a ``datetime64[ns, tz]`` value is provided (:issue:`32395`)
- Joining on :class:`DatetimeIndex` or :class:`TimedeltaIndex` will preserve ``freq`` in simple cases (:issue:`32166`)
-

Expand Down
3 changes: 3 additions & 0 deletions pandas/core/internals/blocks.py
Original file line number Diff line number Diff line change
Expand Up @@ -839,6 +839,9 @@ def setitem(self, indexer, value):

# coerce if block dtype can store value
values = self.values
if is_object_dtype(values.dtype) and isinstance(value, DatetimeArray):
value = value.astype(object)

if self._can_hold_element(value):
# We only get here for non-Extension Blocks, so _try_coerce_args
# is only relevant for DatetimeBlock and TimedeltaBlock
Expand Down
40 changes: 40 additions & 0 deletions pandas/tests/indexing/test_loc.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
""" test label based indexing with loc """
from datetime import timezone
from io import StringIO
import re

Expand Down Expand Up @@ -1015,3 +1016,42 @@ def test_loc_slice_disallows_positional():
with tm.assert_produces_warning(FutureWarning):
# GH#31840 deprecated incorrect behavior
df.loc[1:3, 1] = 2


def test_loc_setitem_df_datetime64tz_column_with_index():
df = pd.DataFrame(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add a comment with the issue number

pd.date_range("2020-01-01", "2020-01-06", 6, tz=timezone.utc), columns=["data"]
)
df2 = pd.DataFrame(index=df.index)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use

result =
expected =

as its not really clear what you are comparing here

df2.loc[df.index, "data"] = df["data"]
tm.assert_numpy_array_equal(np.array(df.data), np.array(df2.data))
assert df2.data.dtype == np.object


def test_loc_setitem_df_datetime64tz_column_without_index():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to repeat things like this, instead we aleady have generic tests in : pandas/tests/extension//base/setitem.py

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@h-vishal this is the key comment. Please do not repeat things like this. We already have fixtures for all of these.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jreback I have pushed a commit with the requested changes.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it doesn't look like anything has changed. there should be NO tests here, rather in pandas/tests/extension/base/setitem

df = pd.DataFrame(
pd.date_range("2020-01-01", "2020-01-06", 6, tz=timezone.utc), columns=["data"]
)
df2 = pd.DataFrame(index=df.index)
df2.loc[:, "data"] = df["data"]
tm.assert_series_equal(df.data, df2.data)


def test_loc_setitem_series_datetime64tz_with_index():
s1 = pd.Series(
pd.date_range("2020-01-01", "2020-01-06", 6, tz=timezone.utc), name="data"
)
s2 = pd.Series(index=s1.index, dtype=np.object, name="data")
s2.loc[s1.index] = s1
tm.assert_numpy_array_equal(np.array(s1), np.array(s2))
assert s2.dtype == np.object


def test_loc_setitem_series_datetime64tz_without_index():
s1 = pd.Series(
pd.date_range("2020-01-01", "2020-01-06", 6, tz=timezone.utc), name="data"
)
s2 = pd.Series(index=s1.index, dtype=np.object, name="data")
s2.loc[:] = s1
tm.assert_numpy_array_equal(np.array(s1), np.array(s2))
assert s2.dtype == np.object