PERF: HDFStore unicode method #16514

Kiv · 2017-05-26T15:58:29Z

HDFStore unicode now only returns file path info. New info() method has the previous behavior of unicode.

import pandas as pd
store = pd.HDFStore('test.h5', 'w')
for i in range(5000):
    store.put('table_{}'.format(i), pd.DataFrame([i]))

# Before
%time str(store)
CPU times: user 26.1 s, sys: 156 ms, total: 26.2 s
Wall time: 26.2 s

# After
%time str(store)
CPU times: user 0 ns, sys: 0 ns, total: 0 ns
Wall time: 40.1 µs

closes HDFStore __unicode__ very slow for many keys #16503
tests added / passed
passes git diff upstream/master --name-only -- '*.py' | flake8 --diff
whatsnew entry

jreback · 2017-05-26T19:08:17Z

pandas/io/pytables.py

@@ -1161,6 +1136,35 @@ def copy(self, file, mode='w', propindexes=True, keys=None, complib=None,

        return new_store

+    def info(self):
+        """return detailed information on the store"""


add a versionadded tag.

jreback · 2017-05-26T19:08:26Z

doc/source/whatsnew/v0.21.0.txt

@@ -75,7 +75,7 @@ Removal of prior version deprecations/changes

 Performance Improvements
 ~~~~~~~~~~~~~~~~~~~~~~~~
-
+- :class:`pandas.HDFStore`'s string representation is now faster and less detailed. For the previous behavior, use ``pandas.HDFStore.info()``. (:issue:`16503`).


move to API breaking changes

jreback · 2017-05-26T19:08:53Z

pandas/io/pytables.py

+    def info(self):
+        """return detailed information on the store"""
+        output = '%s\nFile path: %s\n' % (type(self), pprint_thing(self._path))
+        if self.is_open:


add this to api.rst

jreback

lgtm, with some minor comment.

jreback · 2017-05-26T19:10:06Z

ping when fixed / green.

codecov · 2017-05-26T19:59:44Z

Codecov Report

Merging #16514 into master will decrease coverage by <.01%.
The diff coverage is 85.71%.

@@            Coverage Diff             @@
##           master   #16514      +/-   ##
==========================================
- Coverage   90.79%   90.79%   -0.01%     
==========================================
  Files         161      161              
  Lines       51063    51065       +2     
==========================================
  Hits        46365    46365              
- Misses       4698     4700       +2

Flag	Coverage Δ
#multiple	`88.63% <4.76%> (-0.01%)`	⬇️
#single	`40.15% <85.71%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/io/pytables.py	`92.99% <85.71%> (-0.08%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef487d9...1de16b6. Read the comment docs.

gfyoung · 2017-05-28T10:09:56Z

Got a small linting error in pytables.py (hence the Travis failure)
Can you provide some timing info in the PR given that this is a performance change?

Kiv · 2017-05-28T19:17:17Z

@jreback I think this is ready to go?

jreback · 2017-05-28T19:39:11Z

pandas/tests/io/test_pytables.py

@@ -4371,11 +4373,11 @@ def test_multiple_open_close(self):

            # single
            store = HDFStore(path)
-            assert 'CLOSED' not in str(store)
+            assert 'CLOSED' not in store.info()


so you should also test str(store) a bit (u changed all of these to .info())

I added a test that str(store) prints the filename, is that what you mean? There's not much behavior in there anymore to test...

point to it it's not clear where it is

6faa5a6#diff-5066b062c892df33c29b37a8c1c75f75R422

…avior. __unicode__ now only returns file path info, not (expensive) details on all existing keys.

gfyoung · 2017-05-29T22:20:34Z

@Kiv Some timing info in the PR would be useful for future reference. Plus a benchmark test would be good here.

jreback · 2017-06-01T22:33:17Z

@Kiv lgtm. if you would add an asv would be great (you can use your original example, just not as long, maybe 20 nodes or something would be fine).

jreback · 2017-06-10T19:09:38Z

can you add an asv?

* ENH pandas-dev#15972 added margins_name parameter for crosstab * ENH 15972 minor changes as suggested by reviewers * ENH 15972 correction in whatsnew * ENH 15972 style changes in whatsnew

closes pandas-dev#16499

…ndas-dev#16518)

…#16244)

Replaces most uses of implicit global state from matplotlib in test_datetimelike.py. This was potentially causing random failures where a figure expected to be on a new, blank figure would instead plot on an existing axes (that's the guess at least).

…6079)

* DOC: change doc build to python 3.6 * Remove pinning of pyqt to 4.x * Remove pinning of openpyxl * Add xsel to doc build for clipboard

* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation

…ndas-dev#16444) * BUG: Handle numpy strings in index names in HDF5 pandas-dev#13492 * REF: refactor to _ensure_str

…ches (pandas-dev#16460) * pandas-devgh-14671 Check if usecols with type string contains a subset of names, if not throws an error * tests added for pandas-devgh-14671, expected behavior of simultaneous use of usecols and names unclear so these tests are commented out * Review comments

closes pandas-dev#16608

…t for it (pandas-dev#16633)

…t for it (pandas-dev#16633) (pandas-dev#16650)

Splits extra information about the license and copyright holders to AUTHORS.md.

* COMPAT: numpy 1.13 test compat * CI: fix doc build to 1.12

)" (pandas-dev#16657) This reverts commit ec6bf6d. 1.17.1 released that fixes

Kiv · 2017-06-11T11:35:53Z

My git branch got out of date and I don't know how to update it properly. New version with ASV bench is here:

#16666

jreback · 2017-06-11T15:17:13Z

in the future see: http://pandas.pydata.org/pandas-docs/stable/contributing.html#combining-commits. Much much better to push to the same PR. opening new ones is rarely needed (for the same issue)

you just needed to do:

git rebase -i upstream/master
git push yourremove yourbranch -f

Kiv changed the title ~~Issue16503~~ Close #16503 May 26, 2017

Kiv changed the title ~~Close #16503~~ PERF: HDFStore __unicode__ method May 26, 2017

jreback reviewed May 26, 2017

View reviewed changes

jreback approved these changes May 26, 2017

View reviewed changes

jreback added IO HDF5 read_hdf, HDFStore Performance Memory or execution speed performance labels May 26, 2017

jreback added this to the 0.21.0 milestone May 26, 2017

Kiv force-pushed the issue16503 branch from 29343d7 to c65322f Compare May 27, 2017 18:47

Kiv force-pushed the issue16503 branch 2 times, most recently from f5dd983 to 97eaae3 Compare May 28, 2017 12:01

jreback requested changes May 28, 2017

View reviewed changes

PERF: HDFStore has faster __unicode__, new info() method with old beh…

6faa5a6

…avior. __unicode__ now only returns file path info, not (expensive) details on all existing keys.

Kiv force-pushed the issue16503 branch from 97eaae3 to 6faa5a6 Compare May 28, 2017 21:33

jreback approved these changes Jun 1, 2017

View reviewed changes

cmohl2013 and others added 7 commits June 11, 2017 07:57

ENH: added margins_name parameter for crosstab (pandas-dev#16489)

c8af4cf

* ENH pandas-dev#15972 added margins_name parameter for crosstab * ENH 15972 minor changes as suggested by reviewers * ENH 15972 correction in whatsnew * ENH 15972 style changes in whatsnew

TST: ujson tests are not being run (pandas-dev#16499) (pandas-dev#16500)

840de2f

closes pandas-dev#16499

DOC: Remove preference for pytest paradigm in assert_raises_regex (pa…

4ed801b

…ndas-dev#16518)

TST: Specify HTML file encoding on PY3 (pandas-dev#16526)

c570eaf

BUG: Fixed tput output on windows (pandas-dev#16496)

44d2a12

BUG: Incorrect handling of rolling.cov with offset window (pandas-dev…

1a9cb5b

…#16244)

rhendric and others added 25 commits June 11, 2017 07:57

BUG: reimplement MultiIndex.remove_unused_levels (pandas-dev#16565)

ec535e9

Adding 'n/a' to list of strings denoting missing values (pandas-dev#1…

9e71f08

…6079)

API: Make is_strictly_monotonic_* private (pandas-dev#16576)

32512b9

DOC: change doc build to python 3.6 (pandas-dev#16545)

36670fc

* DOC: change doc build to python 3.6 * Remove pinning of pyqt to 4.x * Remove pinning of openpyxl * Add xsel to doc build for clipboard

DOC: whatsnew 0.20.2 edits (pandas-dev#16587)

5d7a020

DOC: Fix typo in timeseries.rst (pandas-dev#16590)

882ea0f

PERF: vectorize _interp_limit (pandas-dev#16592)

9d0be9d

* PERF: vectorize _interp_limit * CLN: remove old implementation * fixup! CLN: remove old implementation

DOC: Fix typo in merge doc for validate kwarg (pandas-dev#16595)

b3769f1

BUG: convert numpy strings in index names in HDF pandas-dev#13492 (pa…

a0174eb

…ndas-dev#16444) * BUG: Handle numpy strings in index names in HDF5 pandas-dev#13492 * REF: refactor to _ensure_str

DOC: Whatsnew fixups (pandas-dev#16596)

cf5f2d8

DOC: Update release.rst

1415b95

BUG: pickle compat with UTC tz's (pandas-dev#16611)

93aabe7

closes pandas-dev#16608

Fix some lgtm alerts (pandas-dev#16613)

3ebd719

BLD: fix numpy on 3.6 build as 1.13 was released but no deps are buil…

fd171eb

…t for it (pandas-dev#16633)

BUG: Fix Series.get failure on missing NaN (pandas-dev#8569) (pandas-…

4b0ef03

…dev#16619)

TST: NaN in MultiIndex should not become a string (pandas-dev#7031) (p…

1b159af

…andas-dev#16625)

TST: verify we can add and subtract from indices (pandas-dev#8142) (p…

8eb0c7f

…andas-dev#16629)

BUG: conversion of Series to Categorical (pandas-dev#16557)

6fa83d3

fix pandas-dev#16524

BLD: fix numpy on 2.7 build as 1.13 was released but no deps are buil…

aba51b6

…t for it (pandas-dev#16633) (pandas-dev#16650)

CLN: make license file machine readable (pandas-dev#16649)

fdb54df

Splits extra information about the license and copyright holders to AUTHORS.md.

fix pytest-xidst version as 1.17 appears buggy (pandas-dev#16652)

41b3968

COMPAT: numpy 1.13 test compat (pandas-dev#16654)

9d4c88d

* COMPAT: numpy 1.13 test compat * CI: fix doc build to 1.12

Revert "fix pytest-xidst version as 1.17 appears buggy (pandas-dev#16652

8f6e50a

)" (pandas-dev#16657) This reverts commit ec6bf6d. 1.17.1 released that fixes

Add ASV benchmark.

1de16b6

Kiv closed this Jun 11, 2017

Kiv deleted the issue16503 branch June 11, 2017 11:14

Kiv mentioned this pull request Jun 11, 2017

PERF: HDFStore __unicode__ method #16666

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: HDFStore unicode method #16514

PERF: HDFStore unicode method #16514

Kiv commented May 26, 2017 •

edited

Loading

jreback May 26, 2017

jreback May 26, 2017

jreback May 26, 2017

jreback left a comment

jreback commented May 26, 2017

codecov bot commented May 26, 2017 •

edited

Loading

gfyoung commented May 28, 2017

Kiv commented May 28, 2017

jreback May 28, 2017

Kiv May 28, 2017

jreback May 28, 2017

Kiv May 28, 2017

gfyoung commented May 29, 2017

jreback commented Jun 1, 2017

jreback commented Jun 10, 2017

Kiv commented Jun 11, 2017

jreback commented Jun 11, 2017 •

edited

Loading

PERF: HDFStore __unicode__ method #16514

PERF: HDFStore __unicode__ method #16514

Conversation

Kiv commented May 26, 2017 • edited Loading

jreback May 26, 2017

Choose a reason for hiding this comment

jreback May 26, 2017

Choose a reason for hiding this comment

jreback May 26, 2017

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented May 26, 2017

codecov bot commented May 26, 2017 • edited Loading

Codecov Report

gfyoung commented May 28, 2017

Kiv commented May 28, 2017

jreback May 28, 2017

Choose a reason for hiding this comment

Kiv May 28, 2017

Choose a reason for hiding this comment

jreback May 28, 2017

Choose a reason for hiding this comment

Kiv May 28, 2017

Choose a reason for hiding this comment

gfyoung commented May 29, 2017

jreback commented Jun 1, 2017

jreback commented Jun 10, 2017

Kiv commented Jun 11, 2017

jreback commented Jun 11, 2017 • edited Loading

PERF: HDFStore unicode method #16514

PERF: HDFStore unicode method #16514

Kiv commented May 26, 2017 •

edited

Loading

codecov bot commented May 26, 2017 •

edited

Loading

jreback commented Jun 11, 2017 •

edited

Loading