Skip to content

Commit 84a39f9

Browse files
authored
COMPAT: handle pyarrow deprecation of timestamps_to_ms in .from_pandas with pyarrow < 0.6.0 (#17447)
closes #17438
1 parent 25d5299 commit 84a39f9

File tree

3 files changed

+16
-6
lines changed

3 files changed

+16
-6
lines changed

ci/requirements-3.5.sh

+1-1
Original file line numberDiff line numberDiff line change
@@ -8,4 +8,4 @@ echo "install 35"
88
conda remove -n pandas python-dateutil --force
99
pip install python-dateutil
1010

11-
conda install -n pandas -c conda-forge feather-format pyarrow=0.4.1
11+
conda install -n pandas -c conda-forge feather-format pyarrow=0.5.0

doc/source/whatsnew/v0.21.0.txt

+1-1
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ Other Enhancements
125125
- :func:`DataFrame.select_dtypes` now accepts scalar values for include/exclude as well as list-like. (:issue:`16855`)
126126
- :func:`date_range` now accepts 'YS' in addition to 'AS' as an alias for start of year (:issue:`9313`)
127127
- :func:`date_range` now accepts 'Y' in addition to 'A' as an alias for end of year (:issue:`9313`)
128-
- Integration with `Apache Parquet <https://parquet.apache.org/>`__, including a new top-level :func:`read_parquet` and :func:`DataFrame.to_parquet` method, see :ref:`here <io.parquet>`.
128+
- Integration with `Apache Parquet <https://parquet.apache.org/>`__, including a new top-level :func:`read_parquet` and :func:`DataFrame.to_parquet` method, see :ref:`here <io.parquet>`. (:issue:`15838`, :issue:`17438`)
129129
- :func:`DataFrame.add_prefix` and :func:`DataFrame.add_suffix` now accept strings containing the '%' character. (:issue:`17151`)
130130
- `read_*` methods can now infer compression from non-string paths, such as ``pathlib.Path`` objects (:issue:`17206`).
131131
- :func:`pd.read_sas()` now recognizes much more of the most frequently used date (datetime) formats in SAS7BDAT files (:issue:`15871`).

pandas/io/parquet.py

+14-4
Original file line numberDiff line numberDiff line change
@@ -58,13 +58,23 @@ def __init__(self):
5858
"\nor via pip\n"
5959
"pip install -U pyarrow\n")
6060

61+
self._pyarrow_lt_050 = LooseVersion(pyarrow.__version__) < '0.5.0'
62+
self._pyarrow_lt_060 = LooseVersion(pyarrow.__version__) < '0.6.0'
6163
self.api = pyarrow
6264

63-
def write(self, df, path, compression='snappy', **kwargs):
65+
def write(self, df, path, compression='snappy',
66+
coerce_timestamps='ms', **kwargs):
6467
path, _, _ = get_filepath_or_buffer(path)
65-
table = self.api.Table.from_pandas(df, timestamps_to_ms=True)
66-
self.api.parquet.write_table(
67-
table, path, compression=compression, **kwargs)
68+
if self._pyarrow_lt_060:
69+
table = self.api.Table.from_pandas(df, timestamps_to_ms=True)
70+
self.api.parquet.write_table(
71+
table, path, compression=compression, **kwargs)
72+
73+
else:
74+
table = self.api.Table.from_pandas(df)
75+
self.api.parquet.write_table(
76+
table, path, compression=compression,
77+
coerce_timestamps=coerce_timestamps, **kwargs)
6878

6979
def read(self, path):
7080
path, _, _ = get_filepath_or_buffer(path)

0 commit comments

Comments
 (0)