Pandas pytables interface doesn't create empty table datasets

[damionw]

Pandas used to allow the writing of empty HDF5 datasets through its pytables interface code. However, after upgrading to 0.17 (from 0.11), we've discovered that this behaviour is intentionally
short circuited. The library behaves as though the dataset is being written, but simply ignores the request and the resulting HDF5 file doesn't contain the requested table.

The offending code is in pandas/io/pytables.py:_write_to_group()

    # we don't want to store a table node at all if are object is 0-len
    # as there are not dtypes
    if getattr(value, 'empty', None) and (format == 'table' or append):
        return

We've worked around it by patching our installed copy of pandas, but we'd like to know the provocation behind this code before submitting a pull request. The comment implies that the lack of dtypes in the dataset is the cause, however each pandas column has type information even if empty.

Any clarification would be appreciated

[jreback]

pls show a copy pastable example as well as show versions for old/new (esp pytables versions)

[jreback]

I don't believe this was ever supported in PyTables for table format.

[damionw]

Hard to believe since we were using it :-)

In 2012, Wes McKinnon merged a patch into pandas related to this issue:

#1707

Seen here:

603e5ae

And, as to the current behaviour, why is it not an error ? It doesn't fail, and it just doesn't write any of the supporting structures

[jreback]

you are refering to fixed format. Pls show an example and version, including a clean file with generated meta data.

[damionw]

Example

#! /usr/bin/env python
import pandas as pd

h5 = pd.HDFStore("/tmp/test.h5", mode="w")

try:
df = pd.DataFrame([[_x, _x * 2] for _x in range(12)], columns=['one', 'two'], index=None)

h5.put("full", df[:], format='table', data_columns=list(df.keys())) # Write full dataset
h5.put("empty", df[0:0], format='table', data_columns=list(df.keys())) # Write empty dataset

finally:
h5.close()

h5 = pd.HDFStore("/tmp/test.h5", mode="r")

for key in ["full", "empty"]:
print "*** Examining table [{}] ***".format(key)

try:
    print h5[key].head()
except KeyError as _exception:
    print "{} does not exist in hdf5 file".format(key)

print

h5.close()

[jreback]

In [2]: import pandas as pd

In [3]: pd.__version__
Out[3]: '0.11.0'

In [4]: import numpy as np

In [5]: np.__version__
Out[5]: '1.7.1'

In [6]: import tables

In [7]: tables.__version__
Out[7]: '2.4.0'

In [8]: df = pd.DataFrame([[_x, _x * 2] for _x in range(12)], columns=['one', 'two'], index=None)

In [9]: 

In [9]: df
Out[9]: 
    one  two
0     0    0
1     1    2
2     2    4
3     3    6
4     4    8
5     5   10
6     6   12
7     7   14
8     8   16
9     9   18
10   10   20
11   11   22

In [10]: h5 = pd.HDFStore("test.h5", mode="w")
In [11]: h5.put("full", df[:], format='table', data_columns=list(df.keys())) # Write full dataset

In [12]: h5.put("empty", df[0:0], format='table', data_columns=list(df.keys())) # Write empty dataset

In [13]: h5.close()

In [14]: h5 = pd.HDFStore("test.h5", mode="r")

In [15]: h5
Out[15]: 
<class 'pandas.io.pytables.HDFStore'>
File path: test.h5
/empty            frame        (shape->[1,2]) 
/full             frame        (shape->[12,2])

In [16]: h5.root.empty
Out[16]: 
/empty (Group) ''
  children := ['block0_items' (Array), 'block0_values' (Array), 'axis0' (Array), 'axis1' (Array)]

In [20]: h5['empty']
ValueError: Shape of passed values is (2, 0), indices imply (2, 1)

In [22]: h5['full']
Out[22]: 
    one  two
0     0    0
1     1    2
2     2    4
3     3    6
4     4    8
5     5   10
6     6   12
7     7   14
8     8   16
9     9   18
10   10   20
11   11   22

So this looks broken in that version.

So you will have to be more specific on what you are doing.

[damionw]

The above example writes the underlying data structures in version 0.11, but doesn't understand the dimensions (y==0) during reading

In >0.16 the write is ignored

[jreback]

not sure I understand then. you said it worked in 0.11, what does that have to do with it then?

Hard to believe since we were using it :-)

[damionw]

We're writing a data structure that can be empty. Then we're reading the data structure in another program. The current method silently elides the existence of the table, so the reading program would have to catch an exception and fake the data structure.

[jreback]

and from the comments in the PR

fixed this, though required a bit of a hackjob (pytables doesn't like zero-length objects)

[jreback]

@damionw you are saying that it worked though. I want you to prove it. I don't recall this EVER working for table=True (different option back then), because of the PyTables limitation; it was a work-around for fixed.

[damionw]

It worked because we aren't reading the data structure using pandas (in this use case). However, that should work in pandas too, shouldn't it ? Simply ignoring the write request doesn't seem to me to be the way to handle it.

I'll happily fix it if there isn't a reason not to, which is the question I'm asking

[jreback]

I am still not clear on what you aim to fix or even what's broken. Pls show a complete example. Yes you can't read back the empty in that version, but that has been fixed since.

In [19]: pd.read_hdf('../test.h5','empty').dtypes
Out[19]: 
one    int64
two    int64
dtype: object

In [20]: pd.__version__
Out[20]: '0.18.0+176.gb13ddd5'

[damionw]

Pandas normally writes supporting information for tables into the HDF5, the table name is actually used for the group that contains the various indices and data tables. In version 0.11 this information is written to the HDF5 file whether the data is empty or not. Subsequently, it is not written at all.

The problem being addressed is that, now, the very existence of the desired table is prevented when it is empty.

Your above example will read the entry once its created, but won't allow it to be written (at least on 0.17). Please advise if 0.18 now allows writing empty tables (using table=True)

I'm unsure what more of an example you require for showing the behaviour. Could you help me understand what else I need to provide ?