BUG: pandas 1.3.2 incompatible with fastparquet 0.7.0

[mmac-m3a]

• I have checked that this issue has not already been reported.
  I've searched listing of open issues for "parquet" and found no matching entry describing this problem

• I have confirmed this bug exists on the latest version of pandas.
  Exists in 1.3.2

• (optional) I have confirmed this bug exists on the master branch of pandas.
  Exists also in the master branch.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

import pandas as pd

df = pd.read_parquet("df.parquet")

This fails with the following stack trace:

In [8]: df = pd.read_parquet("df.parquet")
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-8-3c208bf199a1> in <module>
----> 1 df = pd.read_parquet("df.parquet")

~/.virtual_envs/kapi2/lib/python3.9/site-packages/pandas/io/parquet.py in read_parquet(path, engine, columns, storage_options, use_nullable_dtypes, **kwargs)
    493     impl = get_engine(engine)
    494
--> 495     return impl.read(
    496         path,
    497         columns=columns,

~/.virtual_envs/kapi2/lib/python3.9/site-packages/pandas/io/parquet.py in read(self, path, columns, storage_options, **kwargs)
    342             path = handles.handle
    343
--> 344         parquet_file = self.api.ParquetFile(path, **parquet_kwargs)
    345
    346         result = parquet_file.to_pandas(columns=columns, **kwargs)

TypeError: __init__() got an unexpected keyword argument 'pandas_nulls'

Problem description

AFAICT, the root cause of the problem is the "api" version test in io/parquet.py It needs to test self.api.__version__ > Version("0.7.0"). ParquetFile constructor in fastparquet 0.7.0 does not support pandas_nulls argument. This argument is supported from version 0.7.1.

fastparquet 0.7.1 unfortunately does not build/install on darwin/arm64 (well, at least for me) :-/.

Expected Output

Obviously, the expected behavior is that the DataFrame is read from the file.

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit : 5f648bf python : 3.9.5.final.0 python-bits : 64 OS : Darwin OS-release : 20.5.0 Version : Darwin Kernel Version 20.5.0: Sat May 8 05:10:31 PDT 2021; root:xnu-7195.121.3~9/RELEASE_ARM64_T8101 machine : arm64 processor : arm byteorder : little LC_ALL : None LANG : en_US.UTF-8 LOCALE : en_US.UTF-8

pandas : 1.3.2
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.2.4
setuptools : 56.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.26.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : 2021.07.0
fastparquet : 0.7.0
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[simonjayhawkins]

Thanks @mmac-m3a for the report. xref #42919, #42987 cc @lithomas1

[lithomas1]

Shoot messed it up. That kwarg is only in 0.7.1 not 0.7.0. Will patch soon. In the meantime, do you mind upgrading to fastparquet 0.7.1? 0.7.0 has incorrect behavior

[mmac-m3a]

I'd love to use fastparquet 0.7.1, but it fails to install on my M1 Macbook :-(. 0.7.0 installs fine. I'm back at 0.6.3 at the moment.

[mmac-m3a]

FYI, I've figured out why fastparquet 0.7.1 fails to install on M1 Mac. See: dask/fastparquet#661 and scipy/oldest-supported-numpy#23 and scipy/oldest-supported-numpy#24

It would still be helpful if pandas worked with an installable version of fastparquet on "every" platform.