pd.Timestamp.utctimetuple() raises TypeError on Python 3.8

[st-bender]

Hi there,

it seems that something changed in python 3.8 that makes pandas.Timestamp.utctimetuple() raise a TypeError instead of producing a time.struct_time.

Code Sample

In [1]: date = pd.to_datetime("2000-01-01", utc=True)                                                       
In [2]: date.utctimetuple()                                                                                 
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-e863c650e721> in <module>
----> 1 date.utctimetuple()

pandas/_libs/tslibs/timestamps.pyx in pandas._libs.tslibs.timestamps.Timestamp.__new__()

TypeError: an integer is required

Expected Output

With earlier Python versions the output is:

In [3]: date.utctimetuple()                                                                                  
Out[3]: time.struct_time(tm_year=2000, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=5, tm_yday=1, tm_isdst=0)

An intermediate workaround can be to convert to datetime.datetime (python 3.8):

In [4]: date.to_pydatetime().utctimetuple()                                                                 
Out[4]: time.struct_time(tm_year=2000, tm_mon=1, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=5, tm_yday=1, tm_isdst=0)

Output of pd.show_versions()

commit : None
python : 3.8.1.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-1066-oem
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C
LOCALE : en_GB.UTF-8

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 45.1.0.post20200119
Cython : None
pytest : 5.3.5
hypothesis : 5.5.1
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.1
IPython : 7.12.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.3
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.3.5
pyxlsb : None
s3fs : None
scipy : 1.4.1
sqlalchemy : None
tables : None
tabulate : None
xarray : 0.15.0
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[sarusso]

Is this still open..? No fix yet?

[jreback]

@sarusso you or anyone in the community can contribute a patch

[charlietsai]

I haven't found the root cause yet but it seems to be related to the time zone (on python 3.9 using latest pandas==1.3.5):

>>> import pandas as pd
>>> ts = pd.Timestamp.now()
>>> ts
Timestamp('2022-01-11 14:14:05.936634')
>>> ts.utctimetuple()
time.struct_time(tm_year=2022, tm_mon=1, tm_mday=11, tm_hour=14, tm_min=14, tm_sec=5, tm_wday=1, tm_yday=11, tm_isdst=0)

when adding a timezone:

>>> ts.tz_localize("UTC").utctimetuple()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/_libs/tslibs/timestamps.pyx", line 1332, in pandas._libs.tslibs.timestamps.Timestamp.__new__
TypeError: an integer is required

https://github.com/pandas-dev/pandas/blob/v1.3.5/pandas/_libs/tslibs/timestamps.pyx#L1331-L1332

sorry, I haven't been able to dig any deeper but hopefully this helps others narrow it down

[FObersteiner]

as a side note, I'm not sure why you would need utctimetuple at all for aware UTC Timestamps;

ts = pd.Timestamp.now('UTC')
print(ts.timetuple())
# time.struct_time(tm_year=2022, tm_mon=1, tm_mday=11, tm_hour=13, tm_min=29, tm_sec=56, tm_wday=1, tm_yday=11, tm_isdst=0)
print(ts.to_pydatetime().utctimetuple())
# time.struct_time(tm_year=2022, tm_mon=1, tm_mday=11, tm_hour=13, tm_min=29, tm_sec=56, tm_wday=1, tm_yday=11, tm_isdst=0)

• the output is exactly the same.

Also quoting from the Python datetime docs:

Warning Because naive datetime objects are treated by many datetime methods as local times, it is preferred to use aware datetimes to represent times in UTC; as a result, using utcfromtimetuple may give misleading results. If you have a naive datetime representing UTC, use datetime.replace(tzinfo=timezone.utc) to make it aware, at which point you can use datetime.timetuple().

...so basically the one-liner

tt = ts.utctimetuple() if ts.tzinfo is None else ts.timetuple()

would avoid the error and give the expected result (as with native Python datetime), although it would be only correct if the naive Timestamp (input) would also represent UTC. In which case you could have used a UTC-aware Timestamp in the first place! ;-)

[charlietsai]

I'm not sure why you would need utctimetuple at all for aware Timestamps

I agree! Thanks for summarizing all the additional caveats as well.

In my case, I am not calling utctimetuple in my own code but rather it's being done somewhere in a library that I use (e.g., python protobuf's FromDateTime conversion). I just happened to be using pd.Timestamps and end up with an unexpected runtime error.

I have no problem working around this in my own code, but was curious to see that this is an issue that seems to only happen with python >=3.8.

[st-bender]

as a side note, I'm not sure why you would need utctimetuple at all for aware Timestamps;

I concour and would say that this is exactly the point, to get a timetuple in UTC regardless of the timezone the original dateime is in.

As the OP (almost 2 years ago), I suggest you have a look at the commit I pushed, that references the issue. And also as indicated by @charlietsai, it is called by a package that I used, not my own code. Maybe they changed that in the meantime in that package.

The idea basically is that I'd like to keep all the datetimes as UTC (aware), and in that case I passed that to a package that I have no control over, and I guess they also want to make sure that the time is in UTC. So utctimetuple() should result in the correct UTC timetuple regardless of the timezone used.

This used to work with python < 3.8, but now raises an error, which is a regression imo.

Cheers.