Skip to content

BUG: during interchanging from non-pandas tz-aware data #54287

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jul 31, 2023
Merged

BUG: during interchanging from non-pandas tz-aware data #54287

Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c89377c
TST: add a test for interchanging from non-pandas tz-aware
natmokval Jul 28, 2023
ddcf978
TST: correct the test for interchanging from non-pandas tz-aware
natmokval Jul 28, 2023
42aeee0
check an error in the test
natmokval Jul 28, 2023
9448242
correct test
natmokval Jul 31, 2023
da6fe09
add extra except in set_nulls
natmokval Jul 31, 2023
15a1fa6
fix bug during interchanging from non-pandas tz-aware
natmokval Jul 31, 2023
dd2c3eb
correct test
natmokval Jul 31, 2023
c575157
correct wording
natmokval Jul 31, 2023
0d5774e
resolve conflict
natmokval Jul 31, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions pandas/tests/interchange/test_impl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -295,3 +295,19 @@ def test_datetimetzdtype(tz, unit):
)
df = pd.DataFrame({"ts_tz": tz_data})
tm.assert_frame_equal(df, from_dataframe(df.__dataframe__()))


def test_interchange_from_non_pandas_tz_aware():
# GH 54239
import pyarrow as pa
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
import pyarrow as pa
pa = pytest.importorskip("pyarrow")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I tried it, but still have the same error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will help CI builds that don't have pyarrow installed

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh, that is the reason why many checks failed

import pyarrow.compute as pc

arr = pa.array([datetime(2020, 1, 1), None, datetime(2020, 1, 2)])
arr = pc.assume_timezone(arr, "Asia/Kathmandu")
result = from_dataframe(pa.table({"arr": arr}).to_pandas())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the interchange protocol means we can avoid to_pandas calls - so long as __dataframe__ is available (which it is for pyarrow tables) we can just do:

Suggested change
result = from_dataframe(pa.table({"arr": arr}).to_pandas())
result = from_dataframe(pa.table({"arr": arr}))

Copy link
Contributor Author

@natmokval natmokval Jul 28, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

strange, it doesn't work for me. If I chage the line 307 to result = from_dataframe(pa.table({"arr": arr})), I get an error

pandas.errors.SettingWithCopyError: modifications to a method of a datetimelike object are not supported and are discarded. Change values on the original.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks - that looks like a warning which should be suppressed somewhere

could you please:

  • in this test, just add with tm.assert_produces_warning to silence it and get the test to pass
  • open a new issue about how the snippet above produces a warning?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. I'll do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems I can’t silence it, because it’s an error. Looks like it comes from the method _delegate_method. I corrected the test, now it checks this error.

df = pd.DataFrame(
["2020-01-01 00:00:00+05:45", "NaT", "2020-01-02 00:00:00+05:45"],
columns=["arr"],
)
expected = df.astype("datetime64[ns, Asia/Kathmandu]")
tm.assert_frame_equal(expected, result)