ENH: enable support for iterator with read_hdf in HDFStore (GH3937)

[jreback]

closes #3937

DOC: update v0.11.1.rst

[jtratner]

@jreback I had a slightly different idea about how to do this that I actually think simplifies everything (turn store into a context manager itself). That lets you do something like:

with get_store(path) as store:
     store['foo'] = bar

Check it out here: https://github.com/jtratner/pandas/tree/change-store-to-contextmanager-type

You could actually remove get_store entirely if you didn't default HDFStore to call open in its constructor, but for right now I added a classmethod to do that.

[jreback]

aside from the ability to directly use HDFStore('file') instead of get_store, is this any different?

[jtratner]

Major difference is that it lets you use with store: anywhere, even if
passed a store, so there's no differentiation in functionality based on
where you use it. (e.g., with a file handle, you can always use it in a
with statement after it's created). e.g.

store = get_store('asdfa.h5')
# do some things
with store:
    # do some stuff where it's important to close it up

Whereas right now, you have to use get_store in a with statement.

It also would let you use the iterator in a with statement directly - I was
thinking about my suggestion to put it at the end of the yield statement
and I think I was wrong about it (or at least not covering all the bases):
if an error gets raised during the iteration, the store will never be
closed.

On Wed, Jun 19, 2013 at 7:34 AM, jreback [email protected] wrote:

aside from the ability to directly use HDFStore('file') instead of
get_store, is this any different?

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/3949#issuecomment-19678133
.

[jreback]

I think this will close it always though....e.g

store = get_store(...)
with store:
   ......
store._handle.isopen == False

as an aside, prob should add is_open property to store anyhow

?

[jtratner]

I don't think the code you have there will work as written. Get store returns a context manager not a HDFStore. You only get the store if you call __enter__ on it.

[jtratner]

Another side note would it make more sense to have open raise an error or warning instead of raw input? Causes the test suite to unexpectedly hang when you try to use it.

Maybe instead the check could be (pseudocode)

if is open and mode==w: 
    if not force: raise some exception   

Because I think that the larger issue is that you can call open repeatedly and have it overwrite itself in w mode

[jreback]

my example was using your code, the question is how does the context manager know not to auto_close?

the whole bizness of _quiet and that type of input was some original code.....needs to be changed

[jtratner]

It doesn't, if you use store as a context manager, the expectation should be that it closes on leaving the with context, just like other file objects. Only table iterator needs to know about auto closing.

[jreback]

ok then.....if you want to do a PR, ok....only thing is need a test for it

(also pls add is_open property to top-level store)

@property
def is_open(self):
    try:
         return self._handle.isopen
     except:
         return False

[jtratner]

And actually given the way that you can reopen it, you could repeatedly use it as a context manager and have it open and close each time, so that it's only open when you are specifically choosing to use it (eg if it's important for more than one program to access it) . If we could get rid of 'w' the mode it'd be a lot better, because then it's never problematic to reopen. (no chance of overwriting anything).

[jreback]

fyi, I always open in mode='a', even when I am reading, unfort have to preserve mode='r' just to prevent accidental writing issues

[jtratner]

@jreback can we deprecate support for 'w' mode?

[jreback]

i think u need it
how else to overwrite a file?
( of course can remove it)
but file interfaces allow to overwrite