Skip to content

DatetimeIndex loses TZ awareness when pushed from / pulled to HDF5 #9270

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
eoincondron opened this issue Jan 16, 2015 · 3 comments
Closed
Labels
Duplicate Report Duplicate issue or pull request Enhancement IO HDF5 read_hdf, HDFStore

Comments

@eoincondron
Copy link

I found that when retrieving a DataFrame which had a MultiIndex containing a DatetimeIndex from a HDFStore, the TZ awareness was somehow lost.
Here is a demonstration:

import datetime as dt
import pandas as pd
start, end = dt.datetime(2015, 1, 16, 8), dt.datetime(2015, 1, 16, 17)
times = pd.date_range(start, end, freq='60min',  tz='US/Eastern')
mix = pd.MultiIndex.from_arrays([2*range(5), times])
df = pd.DataFrame(1, index= mix, columns =range(3))
df

0   1   2
0   2015-01-16 08:00:00-05:00   1   1   1
1   2015-01-16 09:00:00-05:00   1   1   1
2   2015-01-16 10:00:00-05:00   1   1   1

So far so good. Now push and pull from HDFStore:

store = pd.HDFStore('test')
store['test'] = df.swaplevel(0, 1)
store['test']
        0   1   2
2015-01-16 13:00:00 0   1   1   1
2015-01-16 14:00:00 1   1   1   1
2015-01-16 15:00:00 2   1   1   1
store['test'].index.get_level_values(1).tz is None
True

The TZ awareness is gone. A minor issue but I thought I would bring it up. Note that this happens regardless of the order of the index levels and that there is no problem in the case of a single index.and

@jreback
Copy link
Contributor

jreback commented Jan 16, 2015

This is unimplemented for 'fixed' stores, but works on 'table'.

I'll mark it as an enhancement. PR's are welcome.

In [11]: store.append('test2',df.swaplevel(0,1))

In [12]: store.test2
Out[12]: 
                             0  1  2
2015-01-16 08:00:00-05:00 0  1  1  1
2015-01-16 09:00:00-05:00 1  1  1  1
2015-01-16 10:00:00-05:00 2  1  1  1
2015-01-16 11:00:00-05:00 3  1  1  1
2015-01-16 12:00:00-05:00 4  1  1  1
2015-01-16 13:00:00-05:00 0  1  1  1
2015-01-16 14:00:00-05:00 1  1  1  1
2015-01-16 15:00:00-05:00 2  1  1  1
2015-01-16 16:00:00-05:00 3  1  1  1
2015-01-16 17:00:00-05:00 4  1  1  1

@jreback jreback added Enhancement IO HDF5 read_hdf, HDFStore labels Jan 16, 2015
@jreback jreback added this to the 0.17.0 milestone Jan 16, 2015
@jreback jreback added the Duplicate Report Duplicate issue or pull request label Jan 16, 2015
@jreback
Copy link
Contributor

jreback commented Jan 16, 2015

actually this is a dupe of #8165

@eoincondron
Copy link
Author

Ok, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Duplicate Report Duplicate issue or pull request Enhancement IO HDF5 read_hdf, HDFStore
Projects
None yet
Development

No branches or pull requests

2 participants