REGR: to_parquet raising with bytes filename #48995
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phofl
added
Regression
jbrockmendel
reviewed
Oct 7, 2022
doc/source/whatsnew/v1.5.1.rst
Outdated
| @@ -84,6 +84,7 @@ Fixed regressions | |||
| - Fixed Regression in :meth:`DataFrameGroupBy.apply` when user defined function is called on an empty dataframe (:issue:`47985`) | |||
| - Fixed regression in :meth:`DataFrame.apply` when passing non-zero ``axis`` via keyword argument (:issue:`48656`) | |||
| - Fixed regression in :meth:`Series.groupby` and :meth:`DataFrame.groupby` when the grouper is a nullable data type (e.g. :class:`Int64`) or a PyArrow-backed string array, contains null values, and ``dropna=False`` (:issue:`48794`) | |||
| - Fixed regrssion in :meth:`DataFrame.to_parquet` raising when file name was specified as ``bytes`` (:issue:`48944`) | |||
mroeschke
reviewed
Oct 10, 2022
| df = pd.DataFrame(data={"A": [0, 1], "B": [1, 0]}) | ||
| with tm.ensure_clean("test.parquet") as path: | ||
| with open(path.encode(), "wb") as f: | ||
| df.to_parquet(f) |
There was a problem hiding this comment.
Could you parameterize over the engine (pyarrow and fastparquet)?
There was a problem hiding this comment.
I was originally referring to engine in to_parquet, but thanks for also add it in read_parquet
There was a problem hiding this comment.
Sorry, mixed that up. to_parquet does not work, because it does not support BufferedWriter as path for fast parquet
Traceback (most recent call last):
File "/Users/patrick/Library/Application Support/JetBrains/PyCharm2022.2/scratches/scratch.py", line 272, in <module>
df.to_parquet(f, engine="fastparquet")
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/pandas/util/_decorators.py", line 207, in wrapper
return func(*args, **kwargs)
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/pandas/core/frame.py", line 2835, in to_parquet
return to_parquet(
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/pandas/io/parquet.py", line 420, in to_parquet
impl.write(
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/pandas/io/parquet.py", line 301, in write
self.api.write(
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/fastparquet/writer.py", line 1231, in write
write_simple(filename, data, fmd,
File "/Users/patrick/mambaforge/envs/random/lib/python3.10/site-packages/fastparquet/writer.py", line 880, in write_simple
with open_with(fn, mode) as f:
TypeError: expected str, bytes or os.PathLike object, not BufferedWriter
There was a problem hiding this comment.
Ah okay. Guess it's a limitation in fastparquet then.
mroeschke
approved these changes
Oct 12, 2022
|
Thanks @phofl |
meeseeksmachine
pushed a commit
to meeseeksmachine/pandas
that referenced
this pull request
Oct 12, 2022
phofl
added a commit
that referenced
this pull request
Oct 13, 2022
…es filename) (#49061) Backport PR #48995: REGR: to_parquet raising with bytes filename Co-authored-by: Patrick Hoefler <[email protected]>
noatamir
pushed a commit
to noatamir/pandas
that referenced
this pull request
Nov 9, 2022
* REGR: to_parquet raising with bytes filename * Add check * Fix typo * Parametrize
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.