Skip to content

to_datetime returns NaT for old dates with coerce=True #9107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vfilimonov opened this issue Dec 18, 2014 · 4 comments · Fixed by #28386
Closed

to_datetime returns NaT for old dates with coerce=True #9107

vfilimonov opened this issue Dec 18, 2014 · 4 comments · Fixed by #28386
Labels
Datetime Datetime data dtype Docs
Milestone

Comments

@vfilimonov
Copy link
Contributor

When use coerce=True in pandas.to_datetime old dates (before 1677-09-22) return NaT. The same dates without coerce are parsed correctly:

print pd.to_datetime('13000101', format='%Y%m%d', coerce=True)
print pd.to_datetime('13000101', format='%Y%m%d')

returns

NaT
1300-01-01 00:00:00

pandas: 0.15.2
numpy: 1.9.1

@shoyer shoyer added Datetime Datetime data dtype Usage Question labels Dec 19, 2014
@shoyer
Copy link
Member

shoyer commented Dec 19, 2014

I believe this is intentional, although it is poorly documented. The return value in the second case is a datetime.datetime object rather than a pandas.Timestamp object. @jreback thoughts?

@shoyer
Copy link
Member

shoyer commented Dec 19, 2014

To be clear, this (old dates) is a case where to_datetime cannot turn the date into a Timestamp, which is the normal result. So it's a choice (controlled here by coerce) between returning a pandas compatible NaT object or plain python datetime object.

@jreback
Copy link
Contributor

jreback commented Dec 20, 2014

this is correct, the 2nd is a datetime.datetime (Timestamp is a sub-class of this).

not a bug. let's turn this to a further update to the docs (see #8921).

@vfilimonov want to do this?

@jreback jreback added the Docs label Dec 20, 2014
@jreback jreback added this to the 0.16.0 milestone Dec 20, 2014
vfilimonov pushed a commit to vfilimonov/pandas that referenced this issue Jan 2, 2015
shoyer added a commit that referenced this issue Jan 4, 2015
@jreback jreback modified the milestones: 0.16.0, Next Major Release Mar 6, 2015
@miggec
Copy link
Contributor

miggec commented Sep 11, 2019

as of 1.0.0-dev this looks to be resolved:

pd.to_datetime('13000101', format='%Y%m%d')

...

~/pandas-squigmig/pandas/core/tools/datetimes.py in _convert_listlike_datetimes(arg, box, format, name, tz, unit, errors, infer_datetime_format, dayfirst, yearfirst, exact)
    412                 try:
    413                     result, timezones = array_strptime(
--> 414                         arg, format, exact=exact, errors=errors
    415                     )
    416                     if "%Z" in format or "%z" in format:

~/pandas-squigmig/pandas/_libs/tslibs/strptime.pyx in pandas._libs.tslibs.strptime.array_strptime()
    335                 iresult[i] = NPY_NAT
    336                 continue
--> 337             raise
    338 
    339         result_timezone[i] = timezone

~/pandas-squigmig/pandas/_libs/tslibs/strptime.pyx in pandas._libs.tslibs.strptime.array_strptime()
    330         iresult[i] = dtstruct_to_dt64(&dts)
    331         try:
--> 332             check_dts_bounds(&dts)
    333         except ValueError:
    334             if is_coerce:

~/pandas-squigmig/pandas/_libs/tslibs/np_datetime.pyx in pandas._libs.tslibs.np_datetime.check_dts_bounds()
    116                                                dts.day, dts.hour,
    117                                                dts.min, dts.sec)
--> 118         raise OutOfBoundsDatetime(
    119             'Out of bounds nanosecond timestamp: {fmt}'.format(fmt=fmt))
    120 

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 1300-01-01 00:00:00

miggec pushed a commit to miggec/pandas that referenced this issue Sep 11, 2019
test to ensure OutOfBoundsDatetime exception is raised when calling
pd.to_datetime with out-of-bounds date strings
@miggec miggec mentioned this issue Sep 11, 2019
5 tasks
@jreback jreback modified the milestones: Contributions Welcome, 1.0 Sep 11, 2019
jreback pushed a commit that referenced this issue Sep 12, 2019
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
proost pushed a commit to proost/pandas that referenced this issue Dec 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Datetime Datetime data dtype Docs
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants