ENH: Add global option io.nullable_type="pandas"|"pyarrow"
to control IO reader use_nullable_dtype
#48957
Labels
Arrow
pyarrow functionality
Enhancement
IO Data
IO issues that don't fit into a more specific label
NA - MaskedArrays
Related to pd.NA and nullable extension arrays
xref: #29752, #36712
From the above issues and
read_parquet
, it appears that generally ause_nullable_dtypes: bool
option will be added toread_*
functions to allow users to opt into pandas nullable types.Additionally in 1.5 with
ArrowDtype
, the nullable type returned could be backed by pyarrow instead of pandas' implementation. This could be advantageous forread_*(engine="pyarrow")
capable readers where an option likeio.nullable_type="pyarrow"
would just preserve the pyarrow object from the pyarrow parsing function instead of converting it to a numpy object.The proposal would be to add a new global option
io.nullable_type="pandas"|"pyarrow"
(default="pandas") such that whenpd.read_*(..., use_nullable_dtype=True)
, the nullable backing type would be dictated by the globalio.nullable_type
setting.The text was updated successfully, but these errors were encountered: