DOC: Document that read_hdf can use pickle #35554
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jreback
requested changes
Aug 4, 2020
| There is a indexing bug in ``PyTables`` < 3.2 which may appear when querying stores using an index. | ||
| If you see a subset of results being returned, upgrade to ``PyTables`` >= 3.2. | ||
| Stores created previously will need to be rewritten using the updated version. | ||
| Pandas uses PyTables for reading and writing HDF5 files, which allows |
There was a problem hiding this comment.
note that this only applies to fixed stores; tables do not use pickle
|
lgtm ex Jeff's comments |
|
FYI, I tagged this for 1.1.1. Wouldn't normally backport doc-only changes but this is tangentially security related so I think it's better to backport. |
jreback
approved these changes
Aug 6, 2020
|
thanks @TomAugspurger |
meeseeksmachine
pushed a commit
to meeseeksmachine/pandas
that referenced
this pull request
Aug 6, 2020
jreback
pushed a commit
that referenced
this pull request
Aug 6, 2020
Co-authored-by: Tom Augspurger <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This documents that
read_hdf, which uses PyTables, might invoke pickle to deserialize arrays that were serialize with pickle.PyTables clearly documents this at http://www.pytables.org/usersguide/libref/declarative_classes.html#tables.ObjectAtom, but pandas users may not realize that if they aren't aware that pandas uses pytables under the hood.
I've tried to include the warning in all the methods that eventually read data, but it's a bit hard to say if I've gotten them all.
Ideally we would also like to provide an
allow_pickle=Falsekeyword, but that would be best implemented in PyTables, and used by us. I opened PyTables/PyTables#813 for discussion.