Skip to content

BUG: Write compliant Parquet with pyarrow #43690

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 6 commits into from
Closed

BUG: Write compliant Parquet with pyarrow #43690

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e3f1687
Write compliant Parquet with `pyarrow` if supported
judahrand Sep 21, 2021
39010eb
Import `pa_version_under5p0` in `compat` submodule
judahrand Sep 22, 2021
cc7e6a8
Add whatsnew entry
judahrand Sep 22, 2021
17b4d17
Add test for compliant Parquet
judahrand Sep 22, 2021
5f06acd
Add `check_round_trip` to test
judahrand Sep 22, 2021
4025014
Replace long What's New entry with one liner
judahrand Sep 30, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 38 additions & 2 deletions doc/source/whatsnew/v1.4.0.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,9 +186,45 @@ Now the float-dtype is respected. Since the common dtype for these DataFrames is

res

.. _whatsnew_140.notable_bug_fixes.notable_bug_fix3:
.. _whatsnew_140.notable_bug_fixes.write_compliant_parquet_nested_type:

notable_bug_fix3
Write compliant Parquet nested types if possible
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to add a whole sub-section on this. a single line note is fine.

Copy link
Author

@judahrand judahrand Sep 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just figured that since it could break people's stuff it was worth calling out but shall remove if you'd prefer?

Let me know.

Copy link
Contributor

@jreback jreback Sep 22, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the entire point is this wont' break anyone, yes just a single entry is fine

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you want to expand on this? This change will change the data that Pandas outputs. If a user is expecting and handling the current output (not necessarily back into Pandas as Parquet is a cross platform format) then this could break their code elsewhere?

Am I misunderstanding your point?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a single line is fine here

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

When using :meth:`DataFrame.to_parquet` to write a DataFrame to Parquet, if any of the columns contained arrays
of values the :mod:`pyarrow` engine would write a non-compliant format. This behavior is now fixed when the installed
version of PyArrow is at least ``4.0.0``.

https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#nested-types

.. ipython:: python

import pandas as pd
import pyarrow.parquet as pq

df = pd.DataFrame({"int_array_col": [[1, 2, 3], [4, 5, 6]]})
df.to_parquet("/tmp/sample_df")
parquet_table = pq.read_table("/tmp/sample_df")

*Previous behavior*:

.. code-block:: ipython

In [4]: parquet_table.schema.types
Out[4]:
[ListType(list<item: int64>)]

*New behavior*:

.. code-block:: ipython

In [4]: parquet_table.schema.types
Out[4]:
[ListType(list<element: int64>)]

.. _whatsnew_140.notable_bug_fixes.notable_bug_fix4:

notable_bug_fix4
^^^^^^^^^^^^^^^^

.. ---------------------------------------------------------------------------
Expand Down
2 changes: 2 additions & 0 deletions pandas/compat/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
pa_version_under2p0,
pa_version_under3p0,
pa_version_under4p0,
pa_version_under5p0,
)

PY39 = sys.version_info >= (3, 9)
Expand Down Expand Up @@ -155,4 +156,5 @@ def get_lzma_file(lzma):
"pa_version_under2p0",
"pa_version_under3p0",
"pa_version_under4p0",
"pa_version_under5p0",
]
7 changes: 7 additions & 0 deletions pandas/io/parquet.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@
FilePathOrBuffer,
StorageOptions,
)
from pandas.compat import pa_version_under4p0
from pandas.compat._optional import import_optional_dependency
from pandas.errors import AbstractMethodError
from pandas.util._decorators import doc
Expand Down Expand Up @@ -179,6 +180,12 @@ def write(
mode="wb",
is_dir=partition_cols is not None,
)

# Output compliant Parquet if PyArrow supports it and the user hasn't
# explicitly set the desired behavior
if not pa_version_under4p0 and "use_compliant_nested_type" not in kwargs:
kwargs["use_compliant_nested_type"] = True

try:
if partition_cols is not None:
# writes to multiple files under the given path
Expand Down