BUG: DatetimeIndex.unique shifts tz-aware dates

[danielwlogan]

Code Sample, a copy-pastable example if possible

import pandas as pd
start = pd.Timestamp('2018-7-3 2:00:00', tz='America/Los_Angeles')
end = start + pd.Timedelta('36H')
a = pd.DatetimeIndex(start=start, end=end, freq='12H')
a.normalize().unique()

Problem description

The normalize method correctly sets the DatetimeIndex to midnight but the unique method returns a new DatetimeIndex not set to midnight, rather it returns a DatetimeIndex time as if converted to UTC (07:00 in the example code) but still retains the correct timezone ('America/Los_Angeles' in the example code). The code returns:

DatetimeIndex(['2018-07-03 07:00:00-07:00', '2018-07-04 07:00:00-07:00'], dtype='datetime64[ns, America/Los_Angeles]', freq=None)

Expected Output

DatetimeIndex(['2018-07-03 00:00:00-07:00', '2018-07-04 00:00:00-07:00'], dtype='datetime64[ns, America/Los_Angeles]', freq=None)

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.5.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-24-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_CA.UTF-8
LOCALE: en_CA.UTF-8

pandas: 0.23.1
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: 0.28.3
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: 1.2.7
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: 0.4.1
pandas_datareader: None

[jschendel]

Thanks, I can confirm that this issue is occurring on master.

A simpler demonstration of the bug:

In [2]: pd.__version__
Out[2]: '0.24.0.dev0+219.g1070976'

In [3]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [4]: dti
Out[4]: DatetimeIndex(['2017-01-01 00:00:00-05:00', '2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [5]: dti.unique()
Out[5]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

Surprisingly, we actually have a test for this, but it doesn't look to be working as intended:

pandas/pandas/tests/indexes/datetimes/test_datetime.py

Lines 378 to 385 in 1070976

	@pytest.mark.parametrize('arr, expected', [
	(pd.DatetimeIndex(['2017', '2017']), pd.DatetimeIndex(['2017'])),
	(pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern'),
	pd.DatetimeIndex(['2017'], tz='US/Eastern')),
	])
	def test_unique(self, arr, expected):
	result = arr.unique()
	tm.assert_index_equal(result, expected)

Manually inspecting the test shows that the comparing the indexes themselves returns True, but comparing the index values individuals returns False:

In [6]: result = dti.unique()

In [7]: result
Out[7]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [8]: expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [9]: expected
Out[9]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [10]: result == expected
Out[10]: array([ True])

In [11]: result[0] == expected[0]
Out[11]: False

Notice that result and expected have the same .values:

In [12]: result.values
Out[12]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

In [13]: expected.values
Out[13]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

This has something to do with what DatetimeIndex.unique is doing under the hood, since attempting to reconstruct this from the Timestamp objects in question fails to reproduce the comparisons above:

In [14]: result2 = pd.DatetimeIndex([result[0]])

In [15]: result2
Out[15]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [16]: expected2 = pd.DatetimeIndex([expected[0]])

In [17]: expected2
Out[17]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [18]: result2 == expected2
Out[18]: array([False])

In [19]: result2.values
Out[19]: array(['2017-01-01T10:00:00.000000000'], dtype='datetime64[ns]')

In [20]: expected2.values
Out[20]: array(['2017-01-01T05:00:00.000000000'], dtype='datetime64[ns]')

Finally, note that pd.unique appears to be working correctly:

In [21]: dti
Out[21]: DatetimeIndex(['2017-01-01 00:00:00-05:00', '2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [22]: pd.unique(dti)
Out[22]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

[Setur-sjd]

I faced the same bug yesterday.
It took me very long to find the bug in the big code and funnily the bug does not occur in pandas 0.22.0.
I have pandas 0.23.0
And However something very strange happens that I reproduced and copy here (Sypder Ipython):

df.head(2)
Out[17]: 
timestamp
2018-01-01 00:00:00.509000+01:00    134240
2018-01-01 00:00:02.509000+01:00    134350
Name: Whatever, dtype: int64

df.tail(2)
Out[18]: 
timestamp
2018-04-17 18:37:08.973000+02:00    121490
2018-04-17 18:37:10.973000+02:00    121510
Name: Whatever, dtype: int64

all_days = df.index.normalize().unique()

all_days[0]
Out[20]: Timestamp('2017-12-31 23:00:00+0100', tz='Europe/Berlin')

all_days
Out[21]: 
DatetimeIndex(['2018-01-01 00:00:00+01:00', '2018-01-02 00:00:00+01:00',
               '2018-01-03 00:00:00+01:00', '2018-01-04 00:00:00+01:00',
               '2018-01-05 00:00:00+01:00', '2018-01-06 00:00:00+01:00',
               '2018-01-07 00:00:00+01:00', '2018-01-08 00:00:00+01:00',
               '2018-01-09 00:00:00+01:00', '2018-01-10 00:00:00+01:00',
               ...
               '2018-04-08 00:00:00+02:00', '2018-04-09 00:00:00+02:00',
               '2018-04-10 00:00:00+02:00', '2018-04-11 00:00:00+02:00',
               '2018-04-12 00:00:00+02:00', '2018-04-13 00:00:00+02:00',
               '2018-04-14 00:00:00+02:00', '2018-04-15 00:00:00+02:00',
               '2018-04-16 00:00:00+02:00', '2018-04-17 00:00:00+02:00'],
              dtype='datetime64[ns, Europe/Berlin]', name='timestamp', length=107, freq=None)

However, to reproduce it in Jupyter Notebook (just being curious about it), the bug is shown even different!!

all_days = df.index.normalize().unique()

And that is

all_day[0]

:

Timestamp('2017-12-31 00:00:00')

@danielwlogan @jschendel

[mroeschke]

So there's some funniness when passing a DatetimeIndex into DatetimeIndex._simple_new. This is essentially what happens:

# .unique() takes uniques on UTC dates
In [13]: pd.DatetimeIndex._simple_new(pd.DatetimeIndex(['2017-01-01 05:00:00']), tz='US/Eastern')
Out[13]: DatetimeIndex(['2017-01-01 05:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [14]: pd.DatetimeIndex._simple_new(pd.DatetimeIndex(['2017-01-01 05:00:00']).values, tz='US/Eastern')
Out[14]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

So there might be a bug that lies deeper when setting a DatetimeIndex as opposed to a numpy array on the Block.

But adding the .values makes this correct, as In [11]: in @jschendel's example then returns True

(pandas-dev) matthewroeschke:pandas-mroeschke matthewroeschke$ git diff
diff --git a/pandas/core/indexes/datetimes.py b/pandas/core/indexes/datetimes.py
index 966eff582..42a2e531d 100644
--- a/pandas/core/indexes/datetimes.py
+++ b/pandas/core/indexes/datetimes.py
@@ -1072,8 +1072,7 @@ class DatetimeIndex(DatetimeArrayMixin, DatelikeOps, TimelikeOps,
         else:
             naive = self
         result = super(DatetimeIndex, naive).unique(level=level)
-        return self._simple_new(result, name=self.name, tz=self.tz,
-                                freq=self.freq)
+        return self._simple_new(result.values, name=self.name, tz=self.tz, freq=self.freq)

     def union(self, other):
         """

# With .values patch

In [1]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [2]: result = dti.unique()

In [3]: expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [5]: result
Out[5]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [6]: expected
Out[6]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [7]: result == expected
Out[7]: array([ True])

In [8]: result[0] == expected[0]
Out[8]: True

[mroeschke]

So this appears fixed on master: (maybe due to #20912)

In [1]: dti = pd.DatetimeIndex(['2017', '2017'], tz='US/Eastern')

In [2]: dti.unique()
Out[2]: DatetimeIndex(['2017-01-01 00:00:00-05:00'], dtype='datetime64[ns, US/Eastern]', freq=None)

In [3]: result = dti.unique()

In [4]:  expected = pd.DatetimeIndex(['2017'], tz='US/Eastern')

In [5]: result == expected
Out[5]: array([ True])

In [6]: result[0] == expected[0]
Out[6]: True

In [7]: pd.__version__
Out[7]: '0.24.0.dev0+384.gc272c52a5'

Although we already have a test for unique it would be great to additionally test that the underlying timestamps are equal as noted by @jschendel

[danielwlogan]

Thanks everyone for your work tracking this down and getting a fix in place quickly.