Skip to content

ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format #11411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Tux1 opened this issue Oct 22, 2015 · 6 comments · Fixed by #11628
Closed

ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format #11411

Tux1 opened this issue Oct 22, 2015 · 6 comments · Fixed by #11628
Labels
IO HDF5 read_hdf, HDFStore Timezones Timezone data dtype
Milestone

Comments

@Tux1
Copy link
Contributor

Tux1 commented Oct 22, 2015

Hello,

I noticed a bug about exporting DataFrame with columns containing localized datetime. I might be due to some another required libraries updated...
Example :

a = pd.to_datetime(['20120201', '20130201'])
b = pd.to_datetime(['20120201', '20130201'])
dfa = pd.DataFrame([a])
dfb = pd.DataFrame([b])
dfb[0] = dfb[0].dt.tz_localize('UTC')

dfa.to_hdf('tata.hdf', 'df') # success
dfb.to_hdf('toto.hdf', 'df') # fail

It works fine with Series
Thanks guys

@jreback
Copy link
Contributor

jreback commented Oct 22, 2015

you need to use format='table', this is not implemented for format='fixed', the default.

@jreback jreback closed this as completed Oct 22, 2015
@jreback jreback added IO HDF5 read_hdf, HDFStore Timezones Timezone data dtype labels Oct 22, 2015
@Tux1
Copy link
Contributor Author

Tux1 commented Oct 22, 2015

However, it was working until pandas 0.17, no ?

@jreback
Copy link
Contributor

jreback commented Oct 22, 2015

It was never supported in any real way (and not tested), so it 'worked' I guess.

In [7]: dfb.to_hdf('toto.hdf', 'df') # fail
/Users/jreback/miniconda/envs/pandas_0.16.2/lib/python2.7/site-packages/pandas/io/pytables.py:2577: PerformanceWarning: 
your performance may suffer as PyTables will pickle object types that it cannot
map directly to c-types [inferred_type->datetime,key->block1_values] [items->[0]]

  warnings.warn(ws, PerformanceWarning)

@jreback
Copy link
Contributor

jreback commented Oct 22, 2015

this is a dupe of #8165

if you'd like to work on that would be great.

@Tux1
Copy link
Contributor Author

Tux1 commented Oct 22, 2015

Not very easy but I will try yes.
Thanks

@jreback
Copy link
Contributor

jreback commented Oct 22, 2015

actually, will reopen this one and close the other

@jreback jreback reopened this Oct 22, 2015
@jreback jreback added this to the Next Major Release milestone Oct 22, 2015
@jreback jreback changed the title Export DataFrame containing datetime with tz doesn't work in 0.17 ENH: Export DataFrame containing datetime with tz to HDF5 in fixed format Oct 22, 2015
@jreback jreback modified the milestones: 0.17.1, Next Major Release Nov 17, 2015
jreback added a commit that referenced this issue Nov 17, 2015
ENH: Implement export of datetime64[ns, tz] dtypes with a fixed HDF5 #11411
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
IO HDF5 read_hdf, HDFStore Timezones Timezone data dtype
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants