Skip to content

BUG: fix parquet roundtrip with unsigned integer dtypes #31918

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Feb 12, 2020
Merged

BUG: fix parquet roundtrip with unsigned integer dtypes #31918

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a98c3af
BUG: fix parquet roundtrip with unsigned integer dtypes
jorisvandenbossche Feb 12, 2020
f2d9e25
add whatsnew
jorisvandenbossche Feb 12, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions doc/source/whatsnew/v1.0.2.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,8 @@ Bug fixes
**I/O**

- Using ``pd.NA`` with :meth:`DataFrame.to_json` now correctly outputs a null value instead of an empty object (:issue:`31615`)
- Fixed bug in parquet roundtrip with nullable unsigned integer dtypes (:issue:`31896`).


.. ---------------------------------------------------------------------------

Expand Down
4 changes: 4 additions & 0 deletions pandas/core/arrays/integer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,10 @@ def __from_arrow__(
import pyarrow # noqa: F811
from pandas.core.arrays._arrow_utils import pyarrow_array_to_numpy_and_mask

pyarrow_type = pyarrow.from_numpy_dtype(self.type)
if not array.type.equals(pyarrow_type):
array = array.cast(pyarrow_type)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is the cause of a bunch of xfails in test_from_arrow_type_error. does this need to be tightened somehow, or is the test behavior not something we care about?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It could be tightened, for example by checking that the array.type is integral


if isinstance(array, pyarrow.Array):
chunks = [array]
else:
Expand Down
17 changes: 15 additions & 2 deletions pandas/tests/arrays/test_integer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1036,9 +1036,9 @@ def test_arrow_array(data):
assert arr.equals(expected)


@td.skip_if_no("pyarrow", min_version="0.15.1.dev")
@td.skip_if_no("pyarrow", min_version="0.16.0")
def test_arrow_roundtrip(data):
# roundtrip possible from arrow 1.0.0
# roundtrip possible from arrow 0.16.0
import pyarrow as pa

df = pd.DataFrame({"a": data})
Expand All @@ -1048,6 +1048,19 @@ def test_arrow_roundtrip(data):
tm.assert_frame_equal(result, df)


@td.skip_if_no("pyarrow", min_version="0.16.0")
def test_arrow_from_arrow_uint():
# https://github.com/pandas-dev/pandas/issues/31896
# possible mismatch in types
import pyarrow as pa

dtype = pd.UInt32Dtype()
result = dtype.__from_arrow__(pa.array([1, 2, 3, 4, None], type="int64"))
expected = pd.array([1, 2, 3, 4, None], dtype="UInt32")

tm.assert_extension_array_equal(result, expected)


@pytest.mark.parametrize(
"pandasmethname, kwargs",
[
Expand Down
13 changes: 8 additions & 5 deletions pandas/tests/io/test_parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -533,25 +533,28 @@ def test_additional_extension_arrays(self, pa):
df = pd.DataFrame(
{
"a": pd.Series([1, 2, 3], dtype="Int64"),
"b": pd.Series(["a", None, "c"], dtype="string"),
"b": pd.Series([1, 2, 3], dtype="UInt32"),
"c": pd.Series(["a", None, "c"], dtype="string"),
}
)
if LooseVersion(pyarrow.__version__) >= LooseVersion("0.15.1.dev"):
if LooseVersion(pyarrow.__version__) >= LooseVersion("0.16.0"):
expected = df
else:
# de-serialized as plain int / object
expected = df.assign(a=df.a.astype("int64"), b=df.b.astype("object"))
expected = df.assign(
a=df.a.astype("int64"), b=df.b.astype("int64"), c=df.c.astype("object")
)
check_round_trip(df, pa, expected=expected)

df = pd.DataFrame({"a": pd.Series([1, 2, 3, None], dtype="Int64")})
if LooseVersion(pyarrow.__version__) >= LooseVersion("0.15.1.dev"):
if LooseVersion(pyarrow.__version__) >= LooseVersion("0.16.0"):
expected = df
else:
# if missing values in integer, currently de-serialized as float
expected = df.assign(a=df.a.astype("float64"))
check_round_trip(df, pa, expected=expected)

@td.skip_if_no("pyarrow", min_version="0.15.1.dev")
@td.skip_if_no("pyarrow", min_version="0.16.0")
def test_additional_extension_types(self, pa):
# test additional ExtensionArrays that are supported through the
# __arrow_array__ protocol + by defining a custom ExtensionType
Expand Down