BUG: DataFrame.to_hdf segfault

[TomAugspurger]

import pandas as pd

# import dask.dataframe as dd
N = 5000
df = pd.DataFrame({'A': [1, 2] * N, 'B': [3, 4] * N, 'C': ['a', 'b'] * N})

df.to_hdf('foo.h5', '/data', format='table', mode='a')

for _ in range(3):
    df.to_hdf('foo.h5', '/data', format='table', mode='a', append=True)

This actually works when N is smaller, perhaps something with a pytables chunksize

[jreback]

does the file exist in the first place? eg use mode='w' in the first case

what version py tables?

[TomAugspurger]

tables 3.2.0

the file is new originally, and then appending in the for loop.

[jreback]

hmm

couple of bugs in 3.2

ok we can out a test for this and see where it's happening

[TomAugspurger]

N=2996 is the magic number, which gives a df.shape of (5992, 3). It works for N=2995 but segfaults for 2996.

[jreback]

looks like a chunk size issue in the PyTables side

[TomAugspurger]

Just saw #10180 (comment)

Not sure if its the same, getting on a plane so I can't check now. I'll try that fix posted there.

[TomAugspurger]

Confirmed to be fixed in pytables 3.2.1

[jreback]

gr8!

we might want to change install docs to recommend >= 3.0.0 and (but not 3.2) or maybe just >= 3.2.1

[jreback]

reoping to discuss whether to amend the install.rst to put a warning (to use >=3.0.0 & < 3.2, or > 3.2).
I normally wouldn't do this type of version usage, but this fixed a very hard to find bug.

thoughts?

[jreback]

closed via 3e83459