Skip to content

PERF: HDFStore __unicode__ method #16514

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 55 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
6faa5a6
PERF: HDFStore has faster __unicode__, new info() method with old beh…
Kiv May 26, 2017
c8af4cf
ENH: added margins_name parameter for crosstab (#16489)
cmohl2013 May 26, 2017
840de2f
TST: ujson tests are not being run (#16499) (#16500)
abarber4gh May 26, 2017
4ed801b
DOC: Remove preference for pytest paradigm in assert_raises_regex (#1…
gfyoung May 27, 2017
c570eaf
TST: Specify HTML file encoding on PY3 (#16526)
neirbowj May 29, 2017
44d2a12
BUG: Fixed tput output on windows (#16496)
TomAugspurger May 30, 2017
1a9cb5b
BUG: Incorrect handling of rolling.cov with offset window (#16244)
keitakurita May 30, 2017
0a9f548
TST: Avoid global state in matplotlib tests (#16539)
TomAugspurger May 31, 2017
f7149a2
DOC: Update to docstring of DataFrame(dtype) (#14764) (#16487)
VincentLa May 31, 2017
36d6171
DOC: correct docstring examples (#3439) (#16432)
ProsperousHeart May 31, 2017
ab9bc9a
Fix unbound local with bad engine (#16511)
jtratner May 31, 2017
9a9c315
return empty MultiIndex for symmetrical difference on equal MultiInde…
Tafkas May 31, 2017
79cc4a9
BUG: select_as_multiple doesn't respect start/stop kwargs GH16209 (#1…
JosephWagner May 31, 2017
0db4de5
BUG: Bug in .resample() and .groupby() when aggregating on integers (…
jreback May 31, 2017
c193235
COMPAT: cython str-to-int can raise a ValueError on non-CPython (#16563)
mattip May 31, 2017
b9febe0
CLN: raise correct error for Panel sort_values (#16532)
pepicello May 31, 2017
98ed54d
BUG: Fixed pd.unique on array of tuples (#16543)
TomAugspurger Jun 1, 2017
6d761b4
BUG: Allow non-callable attributes in aggregate function. Fixes GH164…
pvomelveny Jun 1, 2017
ed542ee
Strictly monotonic (#16555)
TomAugspurger Jun 1, 2017
f92ec38
COMPAT: Consider Python 2.x tarfiles file-like (#16533)
gfyoung Jun 1, 2017
3f70fda
BUG: Fixed to_html ignoring index_names parameter
CRP Jun 1, 2017
785887a
BUG: fixed wrong order of ordered labels in pd.cut()
economy Jun 1, 2017
746c3cb
fix linting
jreback Jun 1, 2017
885522a
TST: writing invalid table names to sqlite (#16464)
Jun 1, 2017
79beeb6
TST: Skip test_database_uri_string if pg8000 importable (#16528)
neirbowj Jun 1, 2017
a7c95f2
DOC: Remove incorrect elements of PeriodIndex docstring (#16553)
tui-rob Jun 1, 2017
50479ae
TST: Make HDF5 fspath write test robust (#16575)
TomAugspurger Jun 1, 2017
b8ca9fc
ENH: add .ngroup() method to groupby objects (#14026) (#14026)
dsm054 Jun 1, 2017
e331c78
make null lowercase a missing value (#16534)
OlegShteynbuk Jun 1, 2017
e24e57c
MAINT: Drop has_index_names input from read_excel (#16522)
gfyoung Jun 1, 2017
ec535e9
BUG: reimplement MultiIndex.remove_unused_levels (#16565)
rhendric Jun 2, 2017
9e71f08
Adding 'n/a' to list of strings denoting missing values (#16079)
chrisgorgo Jun 2, 2017
32512b9
API: Make is_strictly_monotonic_* private (#16576)
TomAugspurger Jun 2, 2017
36670fc
DOC: change doc build to python 3.6 (#16545)
jorisvandenbossche Jun 2, 2017
5d7a020
DOC: whatsnew 0.20.2 edits (#16587)
jreback Jun 2, 2017
882ea0f
DOC: Fix typo in timeseries.rst (#16590)
funnycrab Jun 4, 2017
9d0be9d
PERF: vectorize _interp_limit (#16592)
TomAugspurger Jun 4, 2017
b3769f1
DOC: Fix typo in merge doc for validate kwarg (#16595)
benjello Jun 4, 2017
a0174eb
BUG: convert numpy strings in index names in HDF #13492 (#16444)
makmanalp Jun 4, 2017
9771514
ERRR: Raise error in usecols when column doesn't exist but length mat…
bpraggastis Jun 4, 2017
cf5f2d8
DOC: Whatsnew fixups (#16596)
TomAugspurger Jun 4, 2017
1415b95
DOC: Update release.rst
TomAugspurger Jun 4, 2017
93aabe7
BUG: pickle compat with UTC tz's (#16611)
jreback Jun 6, 2017
3ebd719
Fix some lgtm alerts (#16613)
jhelie Jun 7, 2017
fd171eb
BLD: fix numpy on 3.6 build as 1.13 was released but no deps are buil…
jreback Jun 8, 2017
4b0ef03
BUG: Fix Series.get failure on missing NaN (#8569) (#16619)
dsm054 Jun 8, 2017
1b159af
TST: NaN in MultiIndex should not become a string (#7031) (#16625)
dsm054 Jun 8, 2017
8eb0c7f
TST: verify we can add and subtract from indices (#8142) (#16629)
dsm054 Jun 8, 2017
6fa83d3
BUG: conversion of Series to Categorical (#16557)
preddy5 Jun 9, 2017
aba51b6
BLD: fix numpy on 2.7 build as 1.13 was released but no deps are buil…
jreback Jun 9, 2017
fdb54df
CLN: make license file machine readable (#16649)
tswast Jun 9, 2017
41b3968
fix pytest-xidst version as 1.17 appears buggy (#16652)
jreback Jun 10, 2017
9d4c88d
COMPAT: numpy 1.13 test compat (#16654)
jreback Jun 10, 2017
8f6e50a
Revert "fix pytest-xidst version as 1.17 appears buggy (#16652)" (#16…
jreback Jun 10, 2017
1de16b6
Add ASV benchmark.
Kiv Jun 11, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/api.rst
Original file line number Diff line number Diff line change
Expand Up @@ -99,6 +99,7 @@ HDFStore: PyTables (HDF5)
HDFStore.append
HDFStore.get
HDFStore.select
HDFStore.info

Feather
~~~~~~~
Expand Down
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v0.21.0.txt
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ Backwards incompatible API changes
- Accessing a non-existent attribute on a closed :class:`HDFStore` will now
raise an ``AttributeError`` rather than a ``ClosedFileError`` (:issue:`16301`)

- :class:`pandas.HDFStore`'s string representation is now faster and less detailed. For the previous behavior, use ``pandas.HDFStore.info()``. (:issue:`16503`).

.. _whatsnew_0210.api:

Other API Changes
Expand Down Expand Up @@ -77,7 +79,6 @@ Performance Improvements
~~~~~~~~~~~~~~~~~~~~~~~~



.. _whatsnew_0210.bug_fixes:

Bug Fixes
Expand Down
59 changes: 33 additions & 26 deletions pandas/io/pytables.py
Original file line number Diff line number Diff line change
Expand Up @@ -494,32 +494,7 @@ def __len__(self):
return len(self.groups())

def __unicode__(self):
output = '%s\nFile path: %s\n' % (type(self), pprint_thing(self._path))
if self.is_open:
lkeys = sorted(list(self.keys()))
if len(lkeys):
keys = []
values = []

for k in lkeys:
try:
s = self.get_storer(k)
if s is not None:
keys.append(pprint_thing(s.pathname or k))
values.append(
pprint_thing(s or 'invalid_HDFStore node'))
except Exception as detail:
keys.append(k)
values.append("[invalid_HDFStore node: %s]"
% pprint_thing(detail))

output += adjoin(12, keys, values)
else:
output += 'Empty'
else:
output += "File is CLOSED"

return output
return '%s\nFile path: %s\n' % (type(self), pprint_thing(self._path))

def __enter__(self):
return self
Expand Down Expand Up @@ -1161,6 +1136,38 @@ def copy(self, file, mode='w', propindexes=True, keys=None, complib=None,

return new_store

def info(self):
"""return detailed information on the store

.. versionadded:: 0.21.0
"""
output = '%s\nFile path: %s\n' % (type(self), pprint_thing(self._path))
if self.is_open:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add this to api.rst

lkeys = sorted(list(self.keys()))
if len(lkeys):
keys = []
values = []

for k in lkeys:
try:
s = self.get_storer(k)
if s is not None:
keys.append(pprint_thing(s.pathname or k))
values.append(
pprint_thing(s or 'invalid_HDFStore node'))
except Exception as detail:
keys.append(k)
values.append("[invalid_HDFStore node: %s]"
% pprint_thing(detail))

output += adjoin(12, keys, values)
else:
output += 'Empty'
else:
output += "File is CLOSED"

return output

# private methods ######
def _check_if_open(self):
if not self.is_open:
Expand Down
37 changes: 20 additions & 17 deletions pandas/tests/io/test_pytables.py
Original file line number Diff line number Diff line change
Expand Up @@ -387,6 +387,7 @@ def test_repr(self):

with ensure_clean_store(self.path) as store:
repr(store)
store.info()
store['a'] = tm.makeTimeSeries()
store['b'] = tm.makeStringSeries()
store['c'] = tm.makeDataFrame()
Expand Down Expand Up @@ -418,8 +419,9 @@ def test_repr(self):
# make a random group in hdf space
store._handle.create_group(store._handle.root, 'bah')

repr(store)
str(store)
assert store.filename in repr(store)
assert store.filename in str(store)
store.info()

# storers
with ensure_clean_store(self.path) as store:
Expand Down Expand Up @@ -4371,11 +4373,11 @@ def test_multiple_open_close(self):

# single
store = HDFStore(path)
assert 'CLOSED' not in str(store)
assert 'CLOSED' not in store.info()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so you should also test str(store) a bit (u changed all of these to .info())

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a test that str(store) prints the filename, is that what you mean? There's not much behavior in there anymore to test...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

point to it it's not clear where it is

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assert store.is_open

store.close()
assert 'CLOSED' in str(store)
assert 'CLOSED' in store.info()
assert not store.is_open

with ensure_clean_path(self.path) as path:
Expand All @@ -4396,20 +4398,20 @@ def f():
store1 = HDFStore(path)
store2 = HDFStore(path)

assert 'CLOSED' not in str(store1)
assert 'CLOSED' not in str(store2)
assert 'CLOSED' not in store1.info()
assert 'CLOSED' not in store2.info()
assert store1.is_open
assert store2.is_open

store1.close()
assert 'CLOSED' in str(store1)
assert 'CLOSED' in store1.info()
assert not store1.is_open
assert 'CLOSED' not in str(store2)
assert 'CLOSED' not in store2.info()
assert store2.is_open

store2.close()
assert 'CLOSED' in str(store1)
assert 'CLOSED' in str(store2)
assert 'CLOSED' in store1.info()
assert 'CLOSED' in store2.info()
assert not store1.is_open
assert not store2.is_open

Expand All @@ -4420,11 +4422,11 @@ def f():
store2 = HDFStore(path)
store2.append('df2', df)
store2.close()
assert 'CLOSED' in str(store2)
assert 'CLOSED' in store2.info()
assert not store2.is_open

store.close()
assert 'CLOSED' in str(store)
assert 'CLOSED' in store.info()
assert not store.is_open

# double closing
Expand All @@ -4433,11 +4435,11 @@ def f():

store2 = HDFStore(path)
store.close()
assert 'CLOSED' in str(store)
assert 'CLOSED' in store.info()
assert not store.is_open

store2.close()
assert 'CLOSED' in str(store2)
assert 'CLOSED' in store2.info()
assert not store2.is_open

# ops on a closed store
Expand Down Expand Up @@ -4784,9 +4786,10 @@ def test_categorical(self):
tm.assert_frame_equal(result, df2)

# Make sure the metadata is OK
assert '/df2 ' in str(store)
assert '/df2/meta/values_block_0/meta' in str(store)
assert '/df2/meta/values_block_1/meta' in str(store)
info = store.info()
assert '/df2 ' in info
assert '/df2/meta/values_block_0/meta' in info
assert '/df2/meta/values_block_1/meta' in info

# unordered
s = Series(Categorical(['a', 'b', 'b', 'a', 'a', 'c'], categories=[
Expand Down