Skip to content

to_datetime broken for Tz-aware datetimes and NaTs in same array #5961

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
cancan101 opened this issue Jan 16, 2014 · 0 comments · Fixed by #5962
Closed

to_datetime broken for Tz-aware datetimes and NaTs in same array #5961

cancan101 opened this issue Jan 16, 2014 · 0 comments · Fixed by #5962
Labels
Bug Datetime Datetime data dtype Timezones Timezone data dtype
Milestone

Comments

@cancan101
Copy link
Contributor

This works:

In [400]: 
pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern'))])

Out[400]:
<class 'pandas.tseries.index.DatetimeIndex'>
[2013-01-01 00:00:00-05:00]
Length: 1, Freq: None, Timezone: US/Eastern

but this (adding an NaT) does not. At the very least, the error message is misleading:

In [401]:
pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern')), pd.NaT])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-401-3b5b8a67d2e1> in <module>()
----> 1 pd.to_datetime([pd.Timestamp("2013-1-1", tz=pytz.timezone('US/Eastern')), pd.NaT])

/usr/local/lib/python2.7/dist-packages/pandas-0.13.0rc1_78_g142ca62-py2.7-linux-x86_64.egg/pandas/tseries/tools.pyc in to_datetime(arg, errors, dayfirst, utc, box, format, coerce, unit)
    137         return Series(values, index=arg.index, name=arg.name)
    138     elif com.is_list_like(arg):
--> 139         return _convert_listlike(arg, box=box)
    140 
    141     return _convert_listlike(np.array([ arg ]), box=box)[0]

/usr/local/lib/python2.7/dist-packages/pandas-0.13.0rc1_78_g142ca62-py2.7-linux-x86_64.egg/pandas/tseries/tools.pyc in _convert_listlike(arg, box)
    127                 return DatetimeIndex._simple_new(values, None, tz=tz)
    128             except (ValueError, TypeError):
--> 129                 raise e
    130 
    131     if arg is None:

ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True

I tracked down this issue to a bug in datetime_to_datetime64

When iterating over the elements in datetime_to_datetime64 the check for nullness is util._checknull(val) which is False for NaT.

The correct null check is to use checknull from lib

PR on the way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype Timezones Timezone data dtype
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant