AmbigousTimeError on Timestamp.floor during dst change #23521

Safrone · 2018-11-05T20:07:41Z

Code Sample, a copy-pastable example if possible

import pandas as pd
pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').floor('T')

Traceback (most recent call last):
  File "/venv/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 3265, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-42bf9ce66d1d>", line 1, in <module>
    pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').floor('T')
  File "pandas/_libs/tslibs/timestamps.pyx", line 696, in pandas._libs.tslibs.timestamps.Timestamp.floor
  File "pandas/_libs/tslibs/timestamps.pyx", line 667, in pandas._libs.tslibs.timestamps.Timestamp._round
  File "pandas/_libs/tslibs/timestamps.pyx", line 903, in pandas._libs.tslibs.timestamps.Timestamp.tz_localize
  File "pandas/_libs/tslibs/conversion.pyx", line 963, in pandas._libs.tslibs.conversion.tz_localize_to_utc
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from '2018-11-04 01:55:00', try using the 'ambiguous' argument

Problem description

AmbiguousTimeError thrown on floor of timestamp that leads to time within dst change.

Similar issues happen on ceil and round:

pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').ceil('T')
pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').round('T')

Expected Output

Timestamp('2018-11-04 01:55:00-0500', tz='America/New_York')

Output of `pd.show_versions()`

INSTALLED VERSIONS

commit: None
python: 3.5.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-38-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.23.4
pytest: None
pip: 10.0.1
setuptools: 39.1.0
Cython: None
numpy: 1.15.2
scipy: None
pyarrow: None
xarray: None
IPython: 7.0.1
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

The text was updated successfully, but these errors were encountered:

Safrone · 2018-11-05T20:26:41Z

A workaround I found:

import pandas as pd
import pytz
ts = pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York')
ts.astimezone(pytz.UTC).floor('T').astimezone(ts.tzinfo)

Timestamp('2018-11-04 01:55:00-0500', tz='America/New_York')

gfyoung · 2018-11-05T20:28:51Z

cc @mroeschke

I believe we might have fixed this in #22647?

@Safrone : Thanks for filing this! Can you installing the master branch an re-running your code?

Safrone · 2018-11-05T20:57:09Z

I cloned the latest version and am still having issues:

>>> import pandas as pd
>>> pd.__version__
'0.24.0.dev0+917.gf0877eccc'
>>> pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').floor('T')

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "pandas/_libs/tslibs/timestamps.pyx", line 824, in pandas._libs.tslibs.timestamps.Timestamp.floor
    return self._round(freq, RoundTo.MINUS_INFTY, ambiguous, nonexistent)
  File "pandas/_libs/tslibs/timestamps.pyx", line 751, in pandas._libs.tslibs.timestamps.Timestamp._round
    result = result.tz_localize(
  File "pandas/_libs/tslibs/timestamps.pyx", line 1096, in pandas._libs.tslibs.timestamps.Timestamp.tz_localize
    value = tz_localize_to_utc(np.array([self.value], dtype='i8'), tz,
  File "pandas/_libs/tslibs/conversion.pyx", line 1019, in pandas._libs.tslibs.conversion.tz_localize_to_utc
    raise pytz.AmbiguousTimeError(
pytz.exceptions.AmbiguousTimeError: Cannot infer dst time from %r, try using the 'ambiguous' argument

INSTALLED VERSIONS

commit: f0877ec
python: 3.5.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-38-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
pandas: 0.24.0.dev0+917.gf0877eccc
pytest: None
pip: 10.0.1
setuptools: 39.1.0
Cython: 0.29
numpy: 1.15.4
scipy: None
pyarrow: None
xarray: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None
gcsfs: None

mroeschke · 2018-11-05T21:06:44Z

As the error notes, you need to specify the ambiguous argument in floor as these were added in #22647

In [1]: pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').f
   ...: loor('T', ambiguous=True)
Out[1]: Timestamp('2018-11-04 01:55:00-0400', tz='America/New_York')

In [2]: pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').f
   ...: loor('T', ambiguous=False)
Out[2]: Timestamp('2018-11-04 01:55:00-0500', tz='America/New_York')

In [3]: pd.Timestamp('2018-11-04 01:55:17.869342-0500', tz='America/New_York').f
   ...: loor('T', ambiguous='NaT')
Out[3]: NaT

Safrone · 2018-11-05T21:08:25Z

Ah yes you are correct, thank you.

adamkpickering · 2018-11-22T17:11:21Z

I think that the problems here are deeper, and cannot be solved by tacking the ambiguous keyword argument onto floor and others.

In [106]: a = pd.Timestamp('2018-11-04T7:31:33Z')
In [107]: a
Out[107]: Timestamp('2018-11-04 07:31:33+0000', tz='UTC')
In [108]: b = a.tz_convert('America/Edmonton')
In [109]: b
Out[109]: Timestamp('2018-11-04 01:31:33-0600', tz='America/Edmonton')
In [110]: b.floor('H')
---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
<ipython-input-110-7a7b2ce5091f> in <module>()
----> 1 b.floor('H')
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp.floor()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp._round()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp.tz_localize()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/conversion.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.conversion.tz_localize_to_utc()
AmbiguousTimeError: Cannot infer dst time from '2018-11-04 01:00:00', try using the 'ambiguous' argument

In [127]: c = pd.Timestamp('2018-11-04T8:31:33Z')
In [128]: c
Out[128]: Timestamp('2018-11-04 08:31:33+0000', tz='UTC')
In [129]: d = c.tz_convert('America/Edmonton')
In [130]: d
Out[130]: Timestamp('2018-11-04 01:31:33-0700', tz='America/Edmonton')
In [131]: d.floor('H')
---------------------------------------------------------------------------
AmbiguousTimeError                        Traceback (most recent call last)
<ipython-input-131-ebab98f5821c> in <module>()
----> 1 d.floor('H')
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp.floor()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp._round()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/timestamps.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.timestamps.Timestamp.tz_localize()
/usr/local/lib/python3.5/dist-packages/pandas/_libs/tslibs/conversion.cpython-35m-x86_64-linux-gnu.so in pandas._libs.tslibs.conversion.tz_localize_to_utc()
AmbiguousTimeError: Cannot infer dst time from '2018-11-04 01:00:00', try using the 'ambiguous' argument

The thing is: 2018-11-04 01:31:33-0700 and 2018-11-04 01:31:33-0600 are not ambiguous times, so it makes no sense to require the programmer to use the ambiguous keyword to specify whether they are in standard or daylight savings time. If given a UTC offset and a location like America/Edmonton or America/New_York it is possible to infer whether that time is in standard or daylight savings time. If you look at the traceback from above:

AmbiguousTimeError: Cannot infer dst time from '2018-11-04 01:00:00', try using the 'ambiguous' argument

it seems that pandas strips the time offset while processing the time, which is very strange since the offset is crucial information.

snewhouse · 2021-03-24T15:33:53Z

Hi All - is this still an issue?

It is for us.

This only occurs on our Windows machines though.

I do not get this error on Linux (Ubuntu 20)

ulmag · 2021-03-29T09:11:49Z

Can someone please explain why this has been closed ? It still seems to be an issue (at least, on Windows). In my view, adamkpickering's #23521 (comment) perfectly describes the issue - and is still valid.

torfsen · 2023-02-09T07:58:55Z

I also still run into this on Pandas 1.5.3 (on Linux). I've posted a potential workaround in #49199.

gfyoung added Datetime Datetime data dtype Numeric Operations Arithmetic, Comparison, and Logical operations labels Nov 5, 2018

mroeschke closed this as completed Nov 5, 2018

torfsen mentioned this issue Oct 20, 2022

BUG: Incorrect result for Timestamp.ceil during clock change #49199

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AmbigousTimeError on Timestamp.floor during dst change #23521

AmbigousTimeError on Timestamp.floor during dst change #23521

Safrone commented Nov 5, 2018

INSTALLED VERSIONS

Safrone commented Nov 5, 2018

gfyoung commented Nov 5, 2018

Safrone commented Nov 5, 2018

INSTALLED VERSIONS

mroeschke commented Nov 5, 2018

Safrone commented Nov 5, 2018

adamkpickering commented Nov 22, 2018 •

edited

Loading

snewhouse commented Mar 24, 2021

ulmag commented Mar 29, 2021

torfsen commented Feb 9, 2023

AmbigousTimeError on Timestamp.floor during dst change #23521

AmbigousTimeError on Timestamp.floor during dst change #23521

Comments

Safrone commented Nov 5, 2018

Code Sample, a copy-pastable example if possible

Problem description

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

Safrone commented Nov 5, 2018

gfyoung commented Nov 5, 2018

Safrone commented Nov 5, 2018

INSTALLED VERSIONS

mroeschke commented Nov 5, 2018

Safrone commented Nov 5, 2018

adamkpickering commented Nov 22, 2018 • edited Loading

snewhouse commented Mar 24, 2021

ulmag commented Mar 29, 2021

torfsen commented Feb 9, 2023

Output of `pd.show_versions()`

adamkpickering commented Nov 22, 2018 •

edited

Loading