Skip to content

BUG: Incorrect localization of naive time string with Series and datetime[ns, tz] dtype #17415

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mroeschke opened this issue Sep 1, 2017 · 2 comments · Fixed by #17603
Closed
Labels
Bug Timezones Timezone data dtype
Milestone

Comments

@mroeschke
Copy link
Member

In [2]: ts_string = '2013-01-01 00:00:00' 

In [3]: pd.Series([ts_string], dtype='datetime64[ns, CET]')
Out[3]: 
0   2013-01-01 01:00:00+01:00
dtype: datetime64[ns, CET]

In [4]: pd.Series([pd.Timestamp(ts_string, tz='CET')])
Out[4]: 
0   2013-01-01 00:00:00+01:00
dtype: datetime64[ns, CET]

Problem description

It appears that a Series will localize a naive string to UTC first and then convert it to the specified timezone specified in dtype instead of immediately localizing the Series to the timezone.

Note: It looks to work correctly for numeric timestamps:

In [11]: ts = 1.5e15

In [12]: pd.Series([ts], dtype='datetime64[ns, CET]')
Out[12]: 
0   1970-01-18 09:40:00+01:00
dtype: datetime64[ns, CET]

In [13]: pd.Series([pd.Timestamp(ts, tz='CET')])
Out[13]: 
0   1970-01-18 09:40:00+01:00
dtype: datetime64[ns, CET]

Expected Output

Out[4]: 
0   2013-01-01 00:00:00+01:00

Output of pd.show_versions()

INSTALLED VERSIONS
------------------
commit: 0d676a3ccf1d7aa986416a7488b941496f936d98
python: 2.7.13.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-45-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.21.0.dev+405.g0d676a3
pytest: 3.2.1
pip: 9.0.1
setuptools: 27.2.0
Cython: 0.26
numpy: 1.13.1
scipy: 0.19.1
pyarrow: None
xarray: None
IPython: 5.3.0
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.4.2
numexpr: 2.6.2
feather: None
matplotlib: 2.0.2
openpyxl: 2.4.8
xlrd: 1.0.0
xlwt: 1.2.0
xlsxwriter: 0.9.8
lxml: 3.8.0
bs4: 4.3.2
html5lib: 0.9999999
sqlalchemy: 1.1.13
pymysql: 0.7.9.None
psycopg2: 2.7.1 (dt dec pq3 ext lo64)
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
@mroeschke mroeschke changed the title BUG: Incorrect localization of native time string with Series and datetime[ns, tz] dtype BUG: Incorrect localization of naive time string with Series and datetime[ns, tz] dtype Sep 1, 2017
@gfyoung gfyoung added Bug Timezones Timezone data dtype labels Sep 8, 2017
@gfyoung
Copy link
Member

gfyoung commented Sep 8, 2017

Weird and buggy...nice catch! PR to patch is welcome!

@mroeschke
Copy link
Member Author

Looks like the culprit is here. Not completely sure why the comment mentions things have to be UTC at this point but will attempt to change and see if anything breaks.

mroeschke added a commit to mroeschke/pandas that referenced this issue Nov 6, 2017
mroeschke added a commit to mroeschke/pandas that referenced this issue Nov 25, 2017
mroeschke added a commit to mroeschke/pandas that referenced this issue Nov 27, 2017
mroeschke added a commit to mroeschke/pandas that referenced this issue Dec 11, 2017
…type (pandas-dev#17415)

Add whatsnew

Fix lint error

Rebase and move whatsnew note

Move whatsnew and add tests
mroeschke added a commit to mroeschke/pandas that referenced this issue Dec 12, 2017
…type (pandas-dev#17415)

Add whatsnew

Fix lint error

Rebase and move whatsnew note

Move whatsnew and add tests
mroeschke added a commit to mroeschke/pandas that referenced this issue Dec 13, 2017
…type (pandas-dev#17415)

Add whatsnew

Fix lint error

Rebase and move whatsnew note

Move whatsnew and add tests
mroeschke added a commit to mroeschke/pandas that referenced this issue Dec 14, 2017
…type (pandas-dev#17415)

Add whatsnew

Fix lint error

Rebase and move whatsnew note

Move whatsnew and add tests
mroeschke added a commit to mroeschke/pandas that referenced this issue Jan 11, 2018
…type (pandas-dev#17415)

Add whatsnew

Fix lint error

Rebase and move whatsnew note

Move whatsnew and add tests
@jreback jreback added this to the 0.23.0 milestone Jan 11, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Timezones Timezone data dtype
Projects
None yet
3 participants