Skip to content

BUG: Timestamp.replace chaining not compat with datetime.replace #15683

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
sscherfke opened this issue Mar 14, 2017 · 29 comments · Fixed by #17507
Closed

BUG: Timestamp.replace chaining not compat with datetime.replace #15683

sscherfke opened this issue Mar 14, 2017 · 29 comments · Fixed by #17507
Labels
Bug Timezones Timezone data dtype

Comments

@sscherfke
Copy link

Code Sample, a copy-pastable example if possible

import pytz
import pandas as pd
from datetime import datetime
pytz.timezone('CET').localize(datetime(2016, 3, 27, 1), is_dst=None)
pytz.timezone('CET').localize(pd.Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)

Problem description

The above code runs with Pandas 0.18 but raises the following exception with Pandas 0.19:

>>> pytz.timezone('CET').localize(pd.Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/localhome/stefan/emsconda/envs/popeye/lib/python3.6/site-packages/pytz/tzinfo.py", line 327, in localize
    raise NonExistentTimeError(dt)
pytz.exceptions.NonExistentTimeError: 2016-03-27 01:00:00

Is this an intentional API breakage of 0.19 or a bug?

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.0.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.12-100.fc24.x86_64+debug
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: de_DE.UTF-8
LOCALE: de_DE.UTF-8

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 34.3.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: None
xarray: None
IPython: 5.3.0
sphinx: None
patsy: None
dateutil: 2.6.0
pytz: 2016.10
blosc: 1.5.0
bottleneck: None
tables: None
numexpr: None
matplotlib: 2.0.0
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: 1.1.5
pymysql: None
psycopg2: None
jinja2: None
boto: None
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Mar 14, 2017

This is correct according to pytz doc-string.

In [8]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=True)
Out[8]: Timestamp('2016-03-27 00:00:00+0100', tz='CET')

In [9]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=False)
Out[9]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [10]: pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)
---------------------------------------------------------------------------
NonExistentTimeError                      Traceback (most recent call last)
<ipython-input-10-6cbd34e0bbef> in <module>()
----> 1 pytz.timezone('CET').localize(Timestamp(datetime(2016, 3, 27, 1)), is_dst=None)

/Users/jreback/miniconda3/envs/pandas/lib/python3.5/site-packages/pytz/tzinfo.py in localize(self, dt, is_dst)
    325             # If we refuse to guess, raise an exception.
    326             if is_dst is None:
--> 327                 raise NonExistentTimeError(dt)
    328 
    329             # If we are forcing the pre-DST side of the DST transition, we

NonExistentTimeError: 2016-03-27 01:00:00

@jreback jreback added the Timezones Timezone data dtype label Mar 14, 2017
@jreback
Copy link
Contributor

jreback commented Mar 14, 2017

actually I find the pytz behavior of is_dst=None to be just odd. They are conflating too many things into a single argument I am afraid.

@sscherfke
Copy link
Author

Okay, thx for the feedback.

@sscherfke
Copy link
Author

We did some more research on this issue and found the following:

The problem occurs during DST-changes when we (de)normalize input dates from dates with tzinfo to UTC dates and back from tz-less UTC dates to dates with a tzinfo.

The stdlib docs states:

“Return a date with the same value, except for those parameters given new values by whichever keyword arguments are specified.”

Lets test this:

import pytz
import pandas as pd
from datetime import datetime

# Base datetime and a tzinfo object
dt = datetime(2016, 3, 27, 1)
tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo

# Expected: tzinfo replaced, actual date value unchanged:
print('Datetimes:')
print(dt.replace(tzinfo=tzinfo))
print(dt.replace(tzinfo=tzinfo).replace(tzinfo=None))

# Unexpected behaviour in pandas 0.19.x:
# Other values than tzinfo were changed:
print('Pandas Timestamp:')
print(pd.Timestamp(dt).replace(tzinfo=tzinfo))
print(pd.Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None))

Pandas 0.18.1:

Datetimes:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00
Pandas Timestamp:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00        # ok

Pandas 0.19.2:

Datetimes:
2016-03-27 01:00:00+01:00
2016-03-27 01:00:00
Pandas Timestamp:
2016-03-27 01:00:00+01:00
2016-03-27 00:00:00        # unexpected

The datetime in the last row of the Pandas 0.19.2 output is incorrect.

This readily occurs in the context of pytz as localize() and normalize() do that all the time (here: pytz 2016.10) in pytz.tzinfo.DstTzInfo.localize, line 314, loc_dt = tzinfo.normalize(dt.replace(tzinfo=tzinfo)) is executed and in pytz.tzinfo.DstTzInfo.normalize, line 239, dt = dt.replace(tzinfo=None) is executed.

Interestingly, the issue only occurs if we do the “double replace” but not if we directly initialize a datetime with a tzinfo:

print(pd.Timestamp(datetime(2016, 3, 27, 1, tzinfo=tzinfo))
print(pd.Timestamp(datetime(2016, 3, 27, 1, tzinfo=tzinfo).replace(tzinfo=None))

Pandas 0.18.1:

2016-03-27 01:00:00+01:00
2016-03-27 01:00:00

Pandas 0.19.2:

2016-03-27 01:00:00+01:00
2016-03-27 01:00:00

I looked at the change logs and coudn’t find anything related to this issue.

@sscherfke sscherfke reopened this Mar 15, 2017
@jreback
Copy link
Contributor

jreback commented Mar 15, 2017

I guess this is a bug, Timestamp.replace should act exactly like datetime.replace. It is overriden because it needs to handle parameter validation and nanoseconds. So [21] should match [19]

In [18]: dt.replace(tzinfo=tzinfo)
Out[18]: datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)

In [19]: dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
Out[19]: datetime.datetime(2016, 3, 27, 1, 0)

In [20]: pd.Timestamp(dt).replace(tzinfo=tzinfo)
Out[20]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [21]: pd.Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
Out[21]: Timestamp('2016-03-27 00:00:00')

All that said, I would never use .replace directly, and more naturally simply tz_localize and tz_convert. (including ambiguity over transitions and such).

In [25]: pd.Timestamp(dt).tz_localize(tzinfo)
Out[25]: Timestamp('2016-03-27 01:00:00+0100', tz='CET')

In [26]: pd.Timestamp(dt).tz_localize(tzinfo).tz_localize(None)
Out[26]: Timestamp('2016-03-27 01:00:00')

you are welcome to submit a PR to fix.

@jreback
Copy link
Contributor

jreback commented Mar 15, 2017

f8bd08e is the change (has been modified slightly since then).

@jreback jreback added this to the Next Major Release milestone Mar 15, 2017
@jreback jreback changed the title Unexpected behavior b/w 0.18 and 0.19 when converting timezones with pytz BUG: Timestamp.replace chaining not compat with datetime.replace Mar 15, 2017
@sscherfke
Copy link
Author

Unfortunately, we currently don't have the time to get familiar with the pandas internals and fix this issue ourselves.

@sscherfke
Copy link
Author

sscherfke commented Mar 29, 2017

I added a regression test for this issues as follows:

diff --git a/pandas/tests/tseries/test_timezones.py b/pandas/tests/tseries/test_timezones.py
index 1fc0e1b..75d4872 100644
--- a/pandas/tests/tseries/test_timezones.py
+++ b/pandas/tests/tseries/test_timezones.py
@@ -1233,6 +1233,18 @@ class TestTimeZones(tm.TestCase):
             self.assertEqual(result_pytz.to_pydatetime().tzname(),
                              result_dateutil.to_pydatetime().tzname())

+        # issue 15683
+        dt = datetime(2016, 3, 27, 1)
+        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
+        # This should work:
+        result_dt = dt.replace(tzinfo=tzinfo)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        # self.assertEqual(result_dt, result_pd.to_datetime())  # This fails!!!
+        # This should fail:
+        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        # self.assertEqual(result_dt, result_pd.to_datetime())
+
     def test_index_equals_with_tz(self):
         left = date_range('1/1/2011', periods=100, freq='H', tz='utc')
         right = date_range('1/1/2011', periods=100, freq='H', tz='US/Eastern')

Surprisingly, the assertEqual() using to_datetime() fails. I don't know if this is another issue or not:

>       self.assertEqual(result_dt, result_pd.to_datetime())
E       AssertionError: datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>) != datetime.datetime(2016, 3, 27, 0, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)

I still have no Idea how to fix this.

@sscherfke
Copy link
Author

sscherfke commented Mar 31, 2017

I played around with with it a little bit more and something looks very broken:

>>> import datetime, pandas, pytz
>>>
>>> # Two equal datetimes:
>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> pd = pandas.Timestamp(dt)
>>> dt == pd
True
>>> dt == pd.to_pydatetime()
True
>>> dt.timestamp() == pd.timestamp()
True
>>>
>>> # Let's introduce timezones and stuff breaks:
>>> 
>>> tzinfo = pytz.timezone('CET')
>>> rdt = dt.replace(tzinfo=tzinfo)
>>> rpd = pd.replace(tzinfo=tzinfo)
>>>
>>> rdt == rpd  # What?
False
>>> rdt == rpd.to_pydatetime()  # Really?
False
>>> rdt.timestamp() == rpd.timestamp()  # Why is this True now?
True
>>> # What do we have?
>>> rdt
datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>> rpd  # This *looks* like rdt but is *not equal* to it.
Timestamp('2016-03-27 01:00:00+0100', tz='CET')
>>> rpd.to_pydatetime()  # This is cleary not wanted:
datetime.datetime(2016, 3, 27, 0, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>>
>>> # This seems to be the logical result of the above bug:
>>> ndt = rdt.replace(tzinfo=None)
>>> npd = rpd.replace(tzinfo=None)
>>> ndt
datetime.datetime(2016, 3, 27, 1, 0)
>>> npd
Timestamp('2016-03-27 00:00:00')
>>> npd.to_pydatetime()
datetime.datetime(2016, 3, 27, 0, 0)
>>> ndt == dt
True
>>> npd == pd
False

@sscherfke
Copy link
Author

The Timestamp constructor already seems to be broken:

>>> dttz = datetime.datetime(2016, 3, 27, 1, tzinfo=tzinfo)
>>> pdtz = pandas.Timestamp(2016, 3, 27, 1, tzinfo=tzinfo)
>>> dttz
datetime.datetime(2016, 3, 27, 1, 0, tzinfo=<DstTzInfo 'CET' CET+1:00:00 STD>)
>>> pdtz  # Where is the tzinfo?
Timestamp('2016-03-27 01:00:00')
>>> dttz.timestamp() == pdtz.timestamp()  # Expected
True
>>> dttz == pdtz  # Unexpected
False
>>> dttz == pdtz.to_pydatetime()  # Unexpected
False

@jreback
Copy link
Contributor

jreback commented Mar 31, 2017

@sscherfke datetime.datetime has a different underlying representation

In [1]: dt = pd.Timestamp('2016-03-27 01:00:00', tz='CET')

In [2]: dt
Out[2]: Timestamp('2016-03-27 00:00:00+0100', tz='CET')

In [3]: dt.tz_convert('UTC')
Out[3]: Timestamp('2016-03-26 23:00:00+0000', tz='UTC')

In [4]: dt.tz_convert('UTC').value
Out[4]: 1459033200000000000

In [5]: dt.value
Out[5]: 1459033200000000000

In [6]: dt.tz
Out[6]: <DstTzInfo 'CET' CET+1:00:00 STD>

In [7]: dt.tz_convert('UTC').tz
Out[7]: <UTC>

TImestamp keeps UTC time always and the tz as a parameter. This always efficient manipulation.

You are encourage to use tz_localize/tz_convert as these correctly manipulate all dst / tz's and work across different tz vendors.

the construction as a small issue xref in #15777

@sscherfke
Copy link
Author

Okay, maybe my last example might then not be related to this issue. But the problem with replace(tzinfo) (it does not only replace the tzinfo but also alter the the actual date/time) remains.

I'd really like to help fixing this issue but Pandas has a very huge code base and I'm a very new Pandas user... :-/

@jreback
Copy link
Contributor

jreback commented Apr 4, 2017

@sscherfke well .replace is actually a very straightforward method, though its in cython, and it does call other things.

@sscherfke
Copy link
Author

Yes, the other things is the problem. Finding out what they are supposed to do and what they actually to and which other thing actually it the culprit for this issue. :)

@jreback
Copy link
Contributor

jreback commented Apr 8, 2017

now that #15934 is merged the construction issues should be fixed, FYI.

@sscherfke
Copy link
Author

Yes, the construction issues are fixed now.

I hoped that this might (accidentally) fix this issue, but it doesn't:

>>> import datetime, pandas, pytz
>>> tzinfo = pytz.timezone('CET')
>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> pd = pandas.Timestamp(dt)
>>> dttz = dt.replace(tzinfo=tzinfo)
>>> pdtz1 = pd.replace(tzinfo=tzinfo)
>>> pdtz2 = pandas.Timestamp('2016-03-27 01:00', tz='CET')
>>> dttz == pdtz1
False
>>> dttz == pdtz2
True
>>> for x in [pdtz1, pdtz2]:
...     print(x, x.tzinfo, x.timestamp(), x.value, x.to_pydatetime())
...
2016-03-27 01:00:00+01:00 CET 1459036800.0 1459033200000000000 2016-03-27 00:00:00+01:00
2016-03-27 01:00:00+01:00 CET 1459036800.0 1459036800000000000 2016-03-27 01:00:00+01:00

As you can see, the value of both Timestamp differs, so I guess replace() breaks it somehow.

replace() (when called on a none-timezoned TS) calls the following four methods in this order:

  • pandas_datetime_to_datetimestruct()
  • pandas_datetimestruct_to_datetime()
  • tz_convert_single()
  • create_timestamp_from_ts()

The pandas_a_to_b() methods are neither defined nor imported in the module (??), so I took a closer look at the remaining two.

create_timestamp_from_ts() does not appear to do any calculations on the value.

So I think tz_convert_single() remains as the most probable culprit.

@sscherfke
Copy link
Author

Yes, tz_convert_single() is the culprit. I added a few prints in replace(). Before that method is called at the end, the value is 1459040400000000000 and afterwards it is 1459033200000000000. The difference is 2h (which is wrong – it should be 1h).

@jreback
Copy link
Contributor

jreback commented Apr 19, 2017

yeah this should not be converting, instead it should be localizing.

diff --git a/pandas/_libs/tslib.pyx b/pandas/_libs/tslib.pyx
index c471d46..6356073 100644
--- a/pandas/_libs/tslib.pyx
+++ b/pandas/_libs/tslib.pyx
@@ -732,7 +732,9 @@ class Timestamp(_Timestamp):
 
         # set tz if needed
         if _tzinfo is not None:
-            value = tz_convert_single(value, _tzinfo, 'UTC')
+            value = tz_localize_to_utc(np.array([value], dtype='i8'), _tzinfo,
+                                       ambiguous='raise',
+                                       errors='raise')[0]
 
         result = create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
         return result

@jreback
Copy link
Contributor

jreback commented Apr 19, 2017

this breaks another test, but passes (so you would make this into an actual test)

In [1]: import datetime, pandas, pytz
   ...: tzinfo = pytz.timezone('CET')
   ...: dt = datetime.datetime(2016, 3, 27, 1)
   ...: pd = pandas.Timestamp(dt)
   ...: dttz = dt.replace(tzinfo=tzinfo)
   ...: pdtz1 = pd.replace(tzinfo=tzinfo)
   ...: pdtz2 = pandas.Timestamp('2016-03-27 01:00', tz='CET')
   ...: 

In [2]: dttz == pdtz1
Out[2]: True

In [3]: dttz == pdtz2
Out[3]: True

@sscherfke
Copy link
Author

sscherfke commented Apr 19, 2017

I am very confused about what's happening inside Pandas:

>>> dt = datetime.datetime(2016, 3, 27, 1)
>>> datetime.datetime.fromtimestamp(pandas.Timestamp(dt).timestamp())
datetime.datetime(2016, 3, 27, 1, 0)
>>> datetime.datetime.fromtimestamp(pandas.Timestamp(dt).value / 1000000000)
datetime.datetime(2016, 3, 27, 3, 0)

I thought value would be a high-res UTC timestamp but it is actually two hours ahead of timestamp() (at least in this case).

When value is converted from CET to UTC at the end of replace(), tz_convert_single() detects that value is summer time (CEST) (because 2016-03-27 03:00 is CEST), it calculates an offset of 2h.

*edit: Saw you comments only after I wrote this comment.

@sscherfke
Copy link
Author

sscherfke commented Apr 19, 2017

A test case like this will show the issue and pass when your proposed fix is applied:

    def test_issue_15683(self):
        # issue 15683
        dt = datetime(2016, 3, 27, 1)
        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
        result_dt = dt.replace(tzinfo=tzinfo)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
        self.assertEqual(result_dt, result_pd.to_pydatetime())
        self.assertEqual(result_dt, result_pd)
        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
        self.assertEqual(result_dt, result_pd.to_pydatetime())
        self.assertEqual(result_dt, result_pd)

@jreback
Copy link
Contributor

jreback commented Apr 19, 2017

happy to take a PR to fix as I said.

@sscherfke
Copy link
Author

What about the breaking test?

@jreback
Copy link
Contributor

jreback commented Apr 19, 2017

if you'd like to delve into that would be helpful

@sscherfke
Copy link
Author

More problems (with our fix):

>>> import datetime, pandas, pytz
>>> tzinfo = pytz.timezone('CET')
>>>
>>> # Reference case with datetime.datetime object
>>> pd = pandas.Timestamp('2016-10-30 01:15').to_pydatetime()
>>> pd
datetime.datetime(2016, 10, 30, 1, 15)
>>> tzinfo.localize(pd, is_dst=True)
datetime.datetime(2016, 10, 30, 1, 15, tzinfo=<DstTzInfo 'CET' CEST+2:00:00 DST>)
>>>
>>> # Error in Pandas
>>> pd = pandas.Timestamp('2016-10-30 01:15')
>>> pd
Timestamp('2016-10-30 01:15:00')
>>> tzinfo.localize(pd, is_dst=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 314, in localize
    loc_dt = tzinfo.normalize(dt.replace(tzinfo=tzinfo))
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 242, in normalize
    return self.fromutc(dt)
  File ".../envs/pandas/lib/python3.6/site-packages/pytz/tzinfo.py", line 187, in fromutc
    return (dt + inf[0]).replace(tzinfo=self._tzinfos[inf])
  File "pandas/_libs/tslib.pyx", line 735, in pandas._libs.tslib.Timestamp.replace (pandas/_libs/tslib.c:14931)
    value = tz_localize_to_utc(np.array([value], dtype='i8'), _tzinfo,
  File "pandas/_libs/tslib.pyx", line 4582, in pandas._libs.tslib.tz_localize_to_utc (pandas/_libs/tslib.c:77718)
    raise pytz.AmbiguousTimeError(
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from Timestamp('2016-10-30 02:15:00'), try using the 'ambigu
ous' argument

This is weired as 01:15 is actually not ambiguous (02:15 would be). I guess the problem arises because we convert from our destination tz to UTC in replace().

In the old version (without the fix) there would be no error but a wrong result (1h offset).

@sscherfke
Copy link
Author

If a Timestamp has a tzinfo (e.g., UTC or CET), (Timestamp.value / 1_000_000_000) == pd.timestamp().

If a Timestamp does not have a tzinfo, (Timestamp.value / 1_000_000_000) - pd.timestamp() is the offset of my local timezone to UTC. Why is this by chance? What is this offset and why is it?

@jreback
Copy link
Contributor

jreback commented Apr 24, 2017

@sscherfke not sure what you are asking.

@sscherfke
Copy link
Author

I finally found a solution that works. All tests in test_timezones are passing and our own code seems to work as well. :)

diff --git a/pandas/_libs/tslib.pyx b/pandas/_libs/tslib.pyx
index c471d46..c418059 100644
--- a/pandas/_libs/tslib.pyx
+++ b/pandas/_libs/tslib.pyx
@@ -685,14 +685,16 @@ class Timestamp(_Timestamp):
         cdef:
             pandas_datetimestruct dts
             int64_t value
-            object _tzinfo, result, k, v
+            object _tzinfo, result, k, v, ts_input
             _TSObject ts
 
         # set to naive if needed
         _tzinfo = self.tzinfo
         value = self.value
         if _tzinfo is not None:
-            value = tz_convert_single(value, 'UTC', _tzinfo)
+            value_tz = tz_convert_single(value, _tzinfo, 'UTC')
+            offset = value - value_tz
+            value += offset
 
         # setup components
         pandas_datetime_to_datetimestruct(value, PANDAS_FR_ns, &dts)
@@ -726,16 +728,14 @@ class Timestamp(_Timestamp):
             _tzinfo = tzinfo
 
         # reconstruct & check bounds
-        value = pandas_datetimestruct_to_datetime(PANDAS_FR_ns, &dts)
+        ts_input = datetime(dts.year, dts.month, dts.day, dts.hour, dts.min,
+                            dts.sec, dts.us, tzinfo=_tzinfo)
+        ts = convert_to_tsobject(ts_input, _tzinfo, None, 0, 0)
+        value = ts.value + (dts.ps // 1000)
         if value != NPY_NAT:
             _check_dts_bounds(&dts)
 
-        # set tz if needed
-        if _tzinfo is not None:
-            value = tz_convert_single(value, _tzinfo, 'UTC')
-
-        result = create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
-        return result
+        return create_timestamp_from_ts(value, dts, _tzinfo, self.freq)
 
     def isoformat(self, sep='T'):
         base = super(_Timestamp, self).isoformat(sep=sep)
diff --git a/pandas/tests/tseries/test_timezones.py b/pandas/tests/tseries/test_timezones.py
index 06b6bbb..08b8040 100644
--- a/pandas/tests/tseries/test_timezones.py
+++ b/pandas/tests/tseries/test_timezones.py
@@ -1280,6 +1280,25 @@ class TestTimeZones(tm.TestCase):
             self.assertEqual(result_pytz.to_pydatetime().tzname(),
                              result_dateutil.to_pydatetime().tzname())
 
+    def test_tzreplace_issue_15683(self):
+        """Regression test for issue 15683."""
+        dt = datetime(2016, 3, 27, 1)
+        tzinfo = pytz.timezone('CET').localize(dt, is_dst=False).tzinfo
+
+        result_dt = dt.replace(tzinfo=tzinfo)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo)
+
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        self.assertEqual(result_dt, result_pd)
+        self.assertEqual(result_dt, result_pd.to_pydatetime())
+
+        result_dt = dt.replace(tzinfo=tzinfo).replace(tzinfo=None)
+        result_pd = Timestamp(dt).replace(tzinfo=tzinfo).replace(tzinfo=None)
+
+        self.assertEqual(result_dt.timestamp(), result_pd.timestamp())
+        self.assertEqual(result_dt, result_pd)
+        self.assertEqual(result_dt, result_pd.to_pydatetime())
+
     def test_index_equals_with_tz(self):
         left = date_range('1/1/2011', periods=100, freq='H', tz='utc')
         right = date_range('1/1/2011', periods=100, freq='H', tz='US/Eastern')

@jreback
Copy link
Contributor

jreback commented Apr 24, 2017

ok if u want to put a PR

TomAugspurger pushed a commit to TomAugspurger/pandas that referenced this issue Jul 5, 2017
Adjustments to coding guidelines

Only perform timestamp() check if meth is available.

Added sv
jreback pushed a commit to jreback/pandas that referenced this issue Sep 13, 2017
jreback pushed a commit to jreback/pandas that referenced this issue Sep 18, 2017
jreback pushed a commit to jreback/pandas that referenced this issue Sep 20, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Timezones Timezone data dtype
Projects
None yet
2 participants