to_datetime, inconsistent behavior with invalid dates.

[vincentdavis]

Consider Feb 29 1991 (2291991) and March 32 1991 (3321991), Both are invalid dates.

pd.to_datetime(2291991, format="%m%d%Y", coerce=True, exact=True)

Returns a TypeError: ValueError: day is out of range for month

While

pd.to_datetime(3321991, format="%m%d%Y", coerce=True, exact=True)

Returns NaT. Which is what I would expect.

In any case they should return the same error or value

Joris Van den Bossche pointed out on the mailing list:
"It has something to do with the number of the day, only values above 31 convert to NaT, 31 or lower raises the error (eg also 31 April raise error instead of giving NaT)"

[jorisvandenbossche]

To show it with a little bit simpler example (as the format you show can be a bit ambiguous depending on the month/day):

In [3]: pd.to_datetime('2015-02-29', coerce=True)
Out[3]: NaT

In [4]: pd.to_datetime('2015-02-29', format="%Y-%m-%d", coerce=True)
---------------------------------------------------------------------------
ValueError: day is out of range for month

In [5]: pd.to_datetime('2015-03-32', format="%Y-%m-%d", coerce=True)
Out[5]: NaT

In [6]: pd.to_datetime('2015-02-32', format="%Y-%m-%d", coerce=True)
Out[6]: NaT

In [7]: pd.to_datetime('2015-04-31', format="%Y-%m-%d", coerce=True)
---------------------------------------------------------------------------
ValueError: day is out of range for month

So it is for the code-path with using a specified format, there is a difference in handling out of range days of > 31 and <=31.

[jreback]

@vincentdavis this is pretty straightforward, just need to the catch the exception and if coerce=True then return `NaT. This is done intslib.pyx/array_strptime``

[vincentdavis]

I would like to help but I am having problems getting started. How to you recommend setting up a development environment for pandas, are there docs on how to do this. What needs to be installed?
I am kinda getting lost in the code and might need some help.

[jreback]

http://pandas.pydata.org/pandas-docs/stable/contributing.html

the actual code to look at is in pandas/tslib.pyx; look in array_strptime

[vincentdavis]

Do you want a test for to_datetime() or array_strptime() my vote is to_datetime() as it is really about obeying coerce=True

[jreback]

your example cases can serve as tests

[vincentdavis]

In this case I was expecting a value error
In [4]: pd.to_datetime('2015-02-29', coerce=False)
Out[4]: '2015-02-29'

[jorisvandenbossche]

@vincentdavis te default for errors is 'ignore'. If you do pd.to_datetime('2015-02-29', coerce=False, errors='raise') this will raise a ValueError

[vincentdavis]

@jorisvandenbossche
This looks correct

In [10]: pd.to_datetime('2015-02-29', errors='ignore',  coerce=False)
Out[10]: '2015-02-29'

I would expect the same from this but get a ValueError.

In [12]: pd.to_datetime('2015-02-29', errors='ignore', format="%Y-%m-%d", coerce=False)
....
ValueError: day is out of range for month

adding format="%Y-%m-%d" should not change the output.
Is this correct and also needs to be fixed?

[jorisvandenbossche]

@vincentdavis yes, I think you are correct. Although I don't really like this default of not raising but returning back the original string, so I more like the behaviour of when providing format which raises. But for consistency it should probably be changed ... (@jreback ?)

[jreback]

with errors='ignore' you won't get errors, so this is correct. This was briefly looked at in #8894.

I think it would be nice to change the default (let's make a separate issue for that though).

[jreback]

closed by #10520