DOC: Add documentation on parquet categorical data handling #58245
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
abeltavares
commented
Apr 13, 2024
•
abeltavares
commented
- closes BUG: Parquet size grows exponential for categorical data #55776
- All [code checks passed]
abeltavares
force-pushed
the
BUG/Parquet-size-grows-exponential-for-categorical-data
branch
from
8a5fe98 to
c74b597
Compare
|
This PR is ready for review. Thanks. |
|
/preview |
|
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/58245/ |
mroeschke
reviewed
Apr 15, 2024
pandas/core/frame.py
Outdated
| categories, not just those present in the data. This behavior | ||
| is expected and consistent with pandas' handling of categorical data. | ||
| * To manage file size and ensure a more predictable roundtrip process, | ||
| consider using `remove_unused_categories` on the DataFrame before saving. |
There was a problem hiding this comment.
Suggested change
| consider using `remove_unused_categories` on the DataFrame before saving. | |
| consider using :meth:`Categorical.remove_unused_categories` on the DataFrame before saving. |
mroeschke
reviewed
Apr 15, 2024
pandas/core/frame.py
Outdated
Comment on lines
2886
to
2887
| is expected and consistent with pandas' handling of categorical data. | ||
| * To manage file size and ensure a more predictable roundtrip process, |
There was a problem hiding this comment.
Suggested change
| is expected and consistent with pandas' handling of categorical data. | |
| * To manage file size and ensure a more predictable roundtrip process, | |
| is expected and consistent with pandas' handling of categorical data. | |
| To manage file size and ensure a more predictable roundtrip process, |
mroeschke
added
IO CSV
abeltavares
force-pushed
the
BUG/Parquet-size-grows-exponential-for-categorical-data
branch
from
c74b597 to
d5020e1
Compare
mroeschke
approved these changes
Apr 19, 2024
|
Thanks @abeltavares |
pmhatre1
pushed a commit
to pmhatre1/pandas-pmhatre1
that referenced
this pull request
May 7, 2024
…ev#58245) Co-authored-by: Abel Tavares <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.