Skip to content

BUG: pandas.read_parquet () dtype_backend argument does not get the default value as documented #59332

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
2 of 3 tasks
gadi-armony-bruker opened this issue Jul 26, 2024 · 4 comments · Fixed by #59071
Closed
2 of 3 tasks
Labels
Docs IO Data IO issues that don't fit into a more specific label
Milestone

Comments

@gadi-armony-bruker
Copy link

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq

fields = [
    pa.field('int_column',pa.int32()),
    pa.field('float_column', pa.float32()),
    pa.field('bool_column', pa.bool_()),
]
schema = pa.schema(fields)
data = {'int_column': [1, 2, None, 4],
        'float_column': [1.5, 2.5, 3.5, None],
        'bool_column': [True, True, None, False]}
pa_table = pa.table(raw_data, schema=schema)
pq.write_table(pa_table, './example_table.parquet')

df1 = pd.read_parquet('./example_table.parquet')
df2 = pd.read_parquet('./example_table.parquet', dtype_backend='numpy_nullable')

print(df1.dtypes)
print(df2.dtypes)

Issue Description

According to the documentation of pd.read_parquet(), the default value for dtype_backend is 'numpy_nullable'.
Data types differ when not passing any argument from when passing 'numpy_nullable'.
Specifically, if not passing any argument the data types are the not nullable ones.

Expected Behavior

not passing an argument should give the same result as passing the default value.

Installed Versions

NSTALLED VERSIONS

commit : d9cdd2e
python : 3.12.4.final.0
python-bits : 64
OS : Linux
OS-release : 6.8.0-38-generic
Version : #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun 7 15:25:01 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0.post0
setuptools : 69.5.1
pip : 24.0
Cython : None
pytest : 7.4.4
hypothesis : None
sphinx : 7.3.7
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 5.2.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.4
IPython : 8.25.0
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
bottleneck : 1.3.7
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : 3.8.4
numba : 0.59.1
numexpr : 2.8.7
odfpy : None
openpyxl : 3.1.2
pandas_gbq : None
pyarrow : 17.0.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2024.3.1
scipy : 1.13.1
sqlalchemy : 2.0.30
tables : 3.9.2
tabulate : 0.9.0
xarray : 2023.6.0
xlrd : None
zstandard : 0.22.0
tzdata : 2023.3
qtpy : 2.4.1
pyqt5 : None

@gadi-armony-bruker gadi-armony-bruker added Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Jul 26, 2024
@Angel-212
Copy link

Thank you for submitting this issue.
The function signature lists lib.no_default as the default for dtype_backend and seems to be out of sync with the documentation here.
Seems like the function docstring needs an update.

@ApoorvApoorv
Copy link
Contributor

take

@rhshadrach
Copy link
Member

Thanks for the report - there is already a PR in the works for fixing this: #59071.

@rhshadrach rhshadrach added Docs IO Data IO issues that don't fit into a more specific label and removed Bug Needs Triage Issue that has not been reviewed by a pandas team member labels Aug 4, 2024
@luke396
Copy link
Contributor

luke396 commented Aug 6, 2024

Apologies for the delayed update, #59071 will be completed soon.

@ApoorvApoorv ApoorvApoorv removed their assignment Aug 9, 2024
@rhshadrach rhshadrach added this to the 3.0 milestone Aug 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Docs IO Data IO issues that don't fit into a more specific label
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants