PERF: add exact kw to to_datetime to enable faster regex format parsing

[jreback]

closes #8989
closes #8903

Clearly the default is exact=True for back-compat
but allows for a match starting at the beginning (has always been like this),
but doesn't require the string to ONLY match the format, IOW, can be extra stuff after the match.

Avoids having to do a regex replace first.

In [21]: s = Series(['19MAY11','19MAY11:00:00:00']*100000)

In [22]: %timeit pd.to_datetime(s.str.replace(':\S+$',''),format='%d%b%y')
1 loops, best of 3: 828 ms per loop

In [23]: %timeit pd.to_datetime(s,format='%d%b%y',exact=False)
1 loops, best of 3: 603 ms per loop

[jorisvandenbossche]

• is the proposed keyword exact or complete?
• the match can be anywhere? (not the start) but I suppose the most common use case is with / without time part, of subsecond part, timezone part (endings)

[jreback]

yeh, changing back to exact. Also going to change it to do a search rather than a match if exact=False so can find arbitrarily.

Acutally other use-cases for this

In [4]: s = Series(['19MAY11','foobar19MAY11','19MAY11:00:00:00','19MAY11 00:00:00Z'])

In [5]: s
Out[5]: 
0              19MAY11
1        foobar19MAY11
2     19MAY11:00:00:00
3    19MAY11 00:00:00Z
dtype: object

In [6]: pd.to_datetime(s,format='%d%b%y',exact=False)
Out[6]: 
0   2011-05-19
1   2011-05-19
2   2011-05-19
3   2011-05-19
dtype: datetime64[ns]

[jreback]

@jorisvandenbossche ?

[jreback]

@jorisvandenbossche ?

[jorisvandenbossche]

Can you add punctuation and capitals here? (when rendered html, sphinx does not keep the newline, so the visual formatting does not work there)

[jorisvandenbossche]

Does this also work for example what now fails (reading in nanoseconds when specifying a format):

In [11]: pd.to_datetime("2012-01-01 09:00:00.000000001", format="%Y-%m-%d %H:%M:%S.%f")
-> 
ValueError: unconverted data remains: 001

[jorisvandenbossche]

Can you add the new exact keyword as a separate entry in the enhancements section?

[jorisvandenbossche]

for the keyword name: match is also a possibility, but not sure it is better

Out of curiousity, where does the speed-up come from? The extra cython type declerations?

[jreback]

@jorisvandenbossche the speedup comes from not have to do an .str.extract(.....) first, then regular parse it. Currently the regex parsers uses .match (and nothing trailing). This allows it to use .search.

[jreback]

ok with match rather than exact

[jreback]

This works; is their an issue about this?

In [3]: pd.to_datetime("2012-01-01 09:00:00.000000001", format="%Y-%m-%d %H:%M:",exact=False)
Out[3]: Timestamp('2012-01-01 09:00:00')

[jorisvandenbossche]

No, no issue I think, and using this keyword solves the error, but of course not the problem that the nanoseconds are not read in. But I don't know if we can do something about that, as this is defined python formatting strings where f means microseconds

[jreback]

You don't actually need to use a formatting string. But we could easily change the 'f' to allow for nanoseconds I think. let me see.

`I``
n [4]: pd.to_datetime("2012-01-01 09:00:00.000000001")
Out[4]: Timestamp('2012-01-01 09:00:00.000000001')

[jorisvandenbossche]

maybe timeseries_with_format_exact is a better name? (the replace does not really matter for what it is about)

[jorisvandenbossche]

@jreback yes, I know, the format string is not needed in this example because I used a default format, but in general that does not hold always

[jorisvandenbossche]

opened an issue for the nanoseconds #8989

[jorisvandenbossche]

And I copied only part of the line. I meant pd.to_datetime("2012-01-01 09:00:00.000000001", format="%Y-%m-%d %H:%M:%S.%f") above

[jreback]

@jorisvandenbossche ok on the kw?

[jorisvandenbossche]

you mean exact vs match ? No strong opinion, exact seems a bit more self-explanatory, match is a bit more in line with the regex naming.

[jreback]

ok, merging then.