Series.drop() on series with timezone aware index can yield AmbiguousTimeError

[larsrinn]

Code Sample, a copy-pastable example if possible

import datetime
import pandas as pd

series = pd.Series(
    list(range(8760)),
    index=pd.date_range('2018/01/01', periods=8760, freq='H', tz='Europe/Berlin')
)

# trying to drop first day
series = series.drop(
    series.index[series.index < datetime.datetime(year=2018, month=1, day=2)]
)

# yields AmbiguousTimeError 

File "pandas/_libs/tslib.pyx", line 3593, in pandas._libs.tslib.tz_localize_to_utc
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from Timestamp('2018-10-28 02:00:00'), try using the 'ambiguous' argument

Problem description

I can't use drop on series that have a timezone aware datetime index. For me, it is not understandable why the error displayed above is raised.

The workaround I found is:

• find the positions to drop
• convert the series to UTC
• drop
• convert back to original timezone

import datetime
import pandas as pd

series = pd.Series(
    list(range(8760)),
    index=pd.date_range('2018/01/01', periods=8760, freq='H', tz='Europe/Berlin')
)

positions_to_drop = series.index < datetime.datetime(year=2018, month=1, day=2)

tz = series.index.tz
series.index = series.index.tz_convert('UTC')
series = series.drop(
    series.index[positions_to_drop]
)
series.index = series.index.tz_convert(tz)

Expected Output

No error. And the result the previous snippet yields.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.4.final.0
python-bits: 64
OS: Darwin
OS-release: 17.4.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: None
LOCALE: de_DE.UTF-8
pandas: 0.22.0
pytest: 3.4.2
pip: 9.0.1
setuptools: 38.5.2
Cython: None
numpy: 1.14.2
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: 1.7.1
patsy: None
dateutil: 2.6.1
pytz: 2018.3
blosc: None
bottleneck: None
tables: None
numexpr: 2.6.4
feather: None
matplotlib: None
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: 2.7.4 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[TomAugspurger]

Your exact error is in the less than comparison.

In [5]: series.index < datetime.datetime(year=2018, month=1, day=2)
   ...:
   ...:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-1504c4ec0be0> in <module>()
----> 1 series.index < datetime.datetime(year=2018, month=1, day=2)
      2

~/sandbox/pandas/pandas/core/indexes/datetimes.py in wrapper(self, other)
    113             if isinstance(other, datetime):
    114                 # GH#18435 strings get a pass from tzawareness compat
--> 115                 self._assert_tzawareness_compat(other)
    116
    117             other = _to_m8(other, tz=self.tz)

~/sandbox/pandas/pandas/core/indexes/datetimes.py in _assert_tzawareness_compat(self, other)
    673                                 'datetime-like objects.')
    674         elif other_tz is None:
--> 675             raise TypeError('Cannot compare tz-naive and tz-aware '
    676                             'datetime-like objects')
    677

TypeError: Cannot compare tz-naive and tz-aware datetime-like objects

Which I think makes more sense. Compare to a tz-aware datetime

In [20]: series.index < pd.Timestamp('2018-01-02', tz=series.index.tz)
Out[20]: array([ True,  True,  True, ..., False, False, False])

So that works. But coincidentally, we hit that error again, this time in drop. Which looks to be a duplicate of #18031.