Skip to content

format ignored in HDFStore.open #13291

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
simonkamronn opened this issue May 26, 2016 · 1 comment · Fixed by #24155
Closed

format ignored in HDFStore.open #13291

simonkamronn opened this issue May 26, 2016 · 1 comment · Fixed by #24155
Labels
Error Reporting Incorrect or improved errors from pandas good first issue IO HDF5 read_hdf, HDFStore
Milestone

Comments

@simonkamronn
Copy link

simonkamronn commented May 26, 2016

new_store = pd.HDFStore('data/test.h5', format='table')
new_store['df'] = pd.DataFrame(np.random.rand(5,1))
new_store.put('df2', pd.DataFrame(np.random.rand(5,1)), format='table')
print('df stored as table: %s' % new_store.get_storer('df').is_table)
print('df2 stored as table: %s' % new_store.get_storer('df2').is_table)

df stored as table: False
df2 stored as table: True

INSTALLED VERSIONS

commit: None
python: 2.7.11.final.0
python-bits: 64
OS: Linux
OS-release: 3.16.0-30-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.0

@jreback jreback added Difficulty Novice Error Reporting Incorrect or improved errors from pandas IO HDF5 read_hdf, HDFStore labels May 26, 2016
@jreback jreback added this to the Next Major Release milestone May 26, 2016
@jreback
Copy link
Contributor

jreback commented May 26, 2016

format is not a defined parameter to HDFStore; only to put/append. So I'll mark this as an issue to raise an exception if format is passed. We accept **kwargs becuase we need to pass args to the file handle open function.

In [1]: pd.HDFStore?
Init signature: pd.HDFStore(self, path, mode=None, complevel=None, complib=None, fletcher32=False, **kwargs)
Docstring:
dict-like IO interface for storing pandas objects in PyTables
either Fixed or Table format.

Parameters
----------
path : string
    File path to HDF5 file
mode : {'a', 'w', 'r', 'r+'}, default 'a'

    ``'r'``
        Read-only; no data can be modified.
    ``'w'``
        Write; a new file is created (an existing file with the same
        name would be deleted).
    ``'a'``
        Append; an existing file is opened for reading and writing,
        and if the file does not exist it is created.
    ``'r+'``
        It is similar to ``'a'``, but the file must already exist.
complevel : int, 1-9, default 0
        If a complib is specified compression will be applied
        where possible
complib : {'zlib', 'bzip2', 'lzo', 'blosc', None}, default None
        If complevel is > 0 apply compression to objects written
        in the store wherever possible
fletcher32 : bool, default False
        If applying compression use the fletcher32 checksum

Examples
--------
>>> from pandas import DataFrame
>>> from numpy.random import randn
>>> bar = DataFrame(randn(10, 4))
>>> store = HDFStore('test.h5')
>>> store['foo'] = bar   # write to HDF5
>>> bar = store['foo']   # retrieve
>>> store.close()
File:           ~/pandas/pandas/io/pytables.py
Type:           type

quintusdias added a commit to quintusdias/pandas_dev that referenced this issue May 28, 2016
@jreback jreback modified the milestones: Contributions Welcome, 0.24.0 Dec 9, 2018
nickponvert added a commit to AllenInstitute/visual_behavior_analysis that referenced this issue Apr 11, 2019
This addresses issue #513 and makes PR #514 unnecessary. For future reference, here's the explanation for the #513 fix: 

This PR addresses issue #513 by changing all of the calls to pandas.read_hdf() to no longer pass 'format' as a kwarg. We used to do this, and the passed kwarg was ignored by HDFStore.open because it wasn't a defined keyword arg for that method, but no alarms were raised.

See: pandas-dev/pandas#13291

In newer versions of pandas there is an exception raised if you try to pass 'format' to the read_hdf() method. Removing this kwarg:

- Won't change the output of read_hdf, as the kwarg was being ignored before, and
- Allows me to run Marina's old notebooks with VBA analysis code and pandas==0.24.2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Error Reporting Incorrect or improved errors from pandas good first issue IO HDF5 read_hdf, HDFStore
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants