Skip to content

Timestamp timezones not preserved for certain indexers #12938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
azjps opened this issue Apr 20, 2016 · 6 comments
Closed

Timestamp timezones not preserved for certain indexers #12938

azjps opened this issue Apr 20, 2016 · 6 comments
Labels
Bug Duplicate Report Duplicate issue or pull request Indexing Related to indexing on series/frames, not to indexes themselves Timezones Timezone data dtype

Comments

@azjps
Copy link

azjps commented Apr 20, 2016

Hi, using different indexers on a dataframes with timezone-aware timestamps may return naive timestamps. (A brief search attempt didn't turn up any related issues.)

Code Sample, a copy-pastable example if possible

>>> import pandas as pd
>>> df = pd.Series([pd.tslib.Timestamp('2016-03-30 14:35:25.506088-04:00')]).to_frame()
>>> df = pd.concat([df, df]).reset_index(drop=True)
>>> df[0][0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df.iloc[0,0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df.loc[0,0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df.iat[0,0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df.at[0,0]
Timestamp('2016-03-30 18:35:25.506088')
>>> df.ix[0,0]
Timestamp('2016-03-30 18:35:25.506088')
>>> df.ix[0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df[0].at[0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')
>>> df[0].ix[0]
Timestamp('2016-03-30 14:35:25.506088-0400', tz='pytz.FixedOffset(-240)')

Expected Output

I'd expect df.at[0,0] and df.ix[0,0] to show the same timezone-aware timestamps as the other indexing methods.

output of pd.show_versions()

>>> import pandas as pd
>>> pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.18.12-el6.ia32e.lime.0
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.0
nose: None
pip: 7.1.0
setuptools: 19.4
Cython: None
numpy: 1.11.0
scipy: None
statsmodels: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.5.2
pytz: 2016.3
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
@jreback
Copy link
Contributor

jreback commented Apr 20, 2016

yeah this is probably not exhaustively tested. patches welcome.

@jreback jreback added Bug Indexing Related to indexing on series/frames, not to indexes themselves Timezones Timezone data dtype Difficulty Intermediate labels Apr 20, 2016
@jreback jreback added this to the 0.18.2 milestone Apr 20, 2016
@jorisvandenbossche jorisvandenbossche modified the milestones: Next Major Release, 0.19.0 Aug 21, 2016
@jreback jreback added the Duplicate Report Duplicate issue or pull request label Mar 28, 2017
@jreback
Copy link
Contributor

jreback commented Mar 28, 2017

closing as duplicate of #15822

@jreback jreback closed this as completed Mar 28, 2017
@jorisvandenbossche
Copy link
Member

Running the above example again with latest master:

In [60]: df = pd.Series([pd.Timestamp('2016-03-30 14:35:25', tz='Europe/Brussels')]).to_frame()

In [61]: df = pd.concat([df, df]).reset_index(drop=True)

In [62]: df
Out[62]: 
                          0
0 2016-03-30 14:35:25+02:00
1 2016-03-30 14:35:25+02:00

In [63]: df[0][0]
Out[63]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [64]: df.iloc[0,0]
Out[64]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [65]: df.loc[0,0]
Out[65]: Timestamp('2016-03-30 12:35:25')

In [66]: df.iat[0,0]
Out[66]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [67]: df.at[0,0]
Out[67]: Timestamp('2016-03-30 12:35:25')

In [68]: df[0].loc[0]
Out[68]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [69]: df[0].at[0]
Out[69]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

So the df.loc[0,0] case has regressed to also not return the timezone.

@jreback
Copy link
Contributor

jreback commented Mar 28, 2017

@jorisvandenbossche I think that last is noted in the other issue. If you (or others!) want to try to pin down when this changed would be great (otherwise) I will fix soon.

@jreback
Copy link
Contributor

jreback commented Mar 28, 2017

In [5]: df[0][0]
Out[5]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [6]: df.iloc[0,0]
Out[6]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [7]:  df.loc[0,0]
Out[7]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [8]: df.iat[0,0]
Out[8]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [9]:  df.at[0,0]
Out[9]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [10]: df[0].loc[0]
Out[10]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

In [11]: df[0].at[0]
    ...: 
Out[11]: Timestamp('2016-03-30 14:35:25+0200', tz='Europe/Brussels')

after #15827

@jreback
Copy link
Contributor

jreback commented Mar 28, 2017

tests: 66fb0a3

mattip pushed a commit to mattip/pandas that referenced this issue Apr 3, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Duplicate Report Duplicate issue or pull request Indexing Related to indexing on series/frames, not to indexes themselves Timezones Timezone data dtype
Projects
None yet
Development

No branches or pull requests

3 participants