ENH: Implement io.nullable_backend config for read_parquet #49039
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mroeschke
added
IO Data
phofl
reviewed
Oct 11, 2022
doc/source/whatsnew/v1.6.0.rst
Outdated
| @@ -33,6 +33,7 @@ Other enhancements | |||
| - :meth:`Series.add_suffix`, :meth:`DataFrame.add_suffix`, :meth:`Series.add_prefix` and :meth:`DataFrame.add_prefix` support an ``axis`` argument. If ``axis`` is set, the default behaviour of which axis to consider can be overwritten (:issue:`47819`) | |||
| - :func:`assert_frame_equal` now shows the first element where the DataFrames differ, analogously to ``pytest``'s output (:issue:`47910`) | |||
| - Added new argument ``use_nullable_dtypes`` to :func:`read_csv` to enable automatic conversion to nullable dtypes (:issue:`36712`) | |||
| - Added new global configuration, ``io.nullable_backend`` to allow ``use_nullable_dtypes=True`` to return pyarrow-backed dtypes when set to ``"pyarrow"`` in :func:`read_parquet` (:issue:`48957`) | |||
There was a problem hiding this comment.
Should engine default to arrow if the option Is set to arrow?
There was a problem hiding this comment.
I'd be a little hesitant to override a user specifying pd.read_*(..., engine="not-arrow") and this option forcing arrow to be used instead
There was a problem hiding this comment.
Agreed, but we could make engine=NoDefault and only override if not given.
But I agree that we should consider this carefully
There was a problem hiding this comment.
Fair. Another consideration is that not all IO methods will have associated arrow parsers (immediately). e.g. read_html
phofl
reviewed
Oct 13, 2022
| @@ -1021,6 +1021,43 @@ def test_read_parquet_manager(self, pa, using_array_manager): | |||
| else: | |||
| assert isinstance(result._mgr, pd.core.internals.BlockManager) | |||
|
|
|||
| def test_read_use_nullable_types_pyarrow_config(self, pa, df_full): | |||
There was a problem hiding this comment.
This is failing in the min versions build
|
@phofl any follow-ups here? |
phofl
reviewed
Oct 21, 2022
pandas/tests/io/test_parquet.py
Outdated
| @@ -1021,6 +1021,46 @@ def test_read_parquet_manager(self, pa, using_array_manager): | |||
| else: | |||
| assert isinstance(result._mgr, pd.core.internals.BlockManager) | |||
|
|
|||
| @pytest.mark.xfail( | |||
| pa_version_under2p0, reason="Timezone conversion incorrect for pyarrow < 2.0" | |||
phofl
approved these changes
Oct 22, 2022
|
thx @mroeschke |
noatamir
pushed a commit
to noatamir/pandas
that referenced
this pull request
Nov 9, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
io.nullable_type="pandas"|"pyarrow"to control IO readeruse_nullable_dtype#48957 (Replace xxxx with the Github issue number)doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.