Skip to content

Timezone information lost after pandas.concat #7562

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
longislandfarmer opened this issue Jun 24, 2014 · 1 comment
Closed

Timezone information lost after pandas.concat #7562

longislandfarmer opened this issue Jun 24, 2014 · 1 comment
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Milestone

Comments

@longislandfarmer
Copy link

I found that when using pandas.concat on dataframes with timezone information, the timezone might be lost when using 'innder' join:

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: from pandas.util.print_versions import show_versions

In [4]: show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.7.final.0
python-bits: 64
OS: Linux
OS-release: 3.2.0-0.bpo.4-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US

pandas: 0.14.0
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2014.3
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: None
bq: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None

In [5]: dr1 = pd.date_range('20110101 0000','20110101 2359',freq='T',tz='Asia/Tokyo')

In [6]: dr2 = pd.date_range('20110101 0000','20110102 2359',freq='T',tz='Asia/Tokyo')

In [7]: df1 = pd.DataFrame(np.random.randn(5,1440),columns=dr1)

In [8]: df2 = pd.DataFrame(np.random.randn(5,2880),columns=dr2)

In [9]: outer = pd.concat([df1,df2],join='outer')

In [10]: inner = pd.concat([df1,df2],join='inner')

In [11]: print outer.axes[1]
<class 'pandas.tseries.index.DatetimeIndex'>
[2011-01-01 00:00:00+09:00, ..., 2011-01-02 23:59:00+09:00]
Length: 2880, Freq: T, Timezone: Asia/Tokyo

In [12]: print inner.axes[1]
<class 'pandas.tseries.index.DatetimeIndex'>
[2010-12-31 15:00:00, ..., 2011-01-01 14:59:00]
Length: 1440, Freq: T, Timezone: None

Looking at the output [11], the timezone information is kept after concat(), however, if we use 'inner' join, the timezone information is lost!

@jreback
Copy link
Contributor

jreback commented Jun 24, 2014

this is fixed in master/0.14.1 (releasing soon), see #6606

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Reshaping Concat, Merge/Join, Stack/Unstack, Explode Timezones Timezone data dtype
Projects
None yet
Development

No branches or pull requests

2 participants