PERF: Some faster date/time string formatting #46759

smarie · 2022-04-12T21:16:00Z

"Easy" fixes related to #44764, taken from the big PR #46116:

Performance improvement in BusinessHour, repr is now 4 times faster
Performance improvement in DatetimeArray and DatetimeIndex: string formatting is now up to 80% faster (as fast as default) when one of the default strftime formats "%Y-%m-%d %H:%M:%S" or "%Y-%m-%d %H:%M:%S.%f" is used (see full bench in PR [WIP] perf improvements for strftime #46116).
Performance improvement in Series.to_excel and DataFrame.to_excel (see ExcelFormatter): processing dates can be up to 4% faster.
Performance improvement in Series.to_sql and DataFrame.to_sql (SQLiteTable): processing time arrays can be up to 65% faster !

I also added a few code comments about datetime formatting sections, to improve maintenance.

closes #xxxx (Replace xxxx with the Github issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

…imes faster ! Related to pandas-dev#44764

smarie · 2022-04-12T21:17:29Z

BusinessHour repr improvements mini-bench:

get_repr = lambda: repr(self.offset1)
glob = globals()
glob.update(locals())
res = Timer("get_repr()", globals=glob).repeat(3, 10000)

before: [0.15039550000000013, 0.16580459999999952, 0.12403839999999988]
after: [0.035295299999999585, 0.0369325000000007, 0.03622369999999897]

= 4 times faster!

…eIndex`: string formatting is now up to 80% faster (as fast as default) when one of the default strftime formats ``"%Y-%m-%d %H:%M:%S"`` or ``"%Y-%m-%d %H:%M:%S.%f"`` is used. See pandas-dev#44764

…me.to_excel` (:class:`ExcelFormatter`): processing dates can be up to 4% faster. (related to :issue:`44764`)

…QLiteTable`): processing time arrays can be up to 65% faster ! (related to pandas-dev#44764)

…ure/44764_strftime_perf_datetimes

smarie · 2022-04-12T23:12:48Z

SQL mini-bench:

import sqlite3
con = sqlite3.connect(':memory:')
dt = pd.date_range('2014-01-01', periods=1000, freq='10h')
df = pd.DataFrame({'time': dt.time})

from timeit import Timer
to_assess = lambda: df.to_sql('test_datetime', con, index=False, if_exists='replace')
glob = globals()
glob.update(locals())
res = Timer("to_assess()", globals=glob).repeat(5, 300)
print(res)

before: [2.667514099999991, 2.711463600000002, 2.7073388000000023, 2.6616616999999962, 2.7392122000000114]
after: [0.9359815999999981, 0.9132108000000017, 0.9482149999999976, 0.9659205999999969, 0.9622573000000045]
65% improvement

smarie · 2022-04-12T23:16:26Z

Ready for review

jreback

do these have specific asv's that test performance? if not pls add. pls show them in the PR

doc/source/whatsnew/v1.5.0.rst

pandas/_libs/tslibs/timedeltas.pyx

pandas/io/sql.py

…formatting" This reverts commit 139f229

…mezone. Made it faster using f-string

smarie · 2022-04-13T08:59:22Z

Thanks to both of you for the quick review ! I took care of all comments.

do these have specific asv's that test performance? if not pls add. pls show them in the PR

Ok. I never used asv, so I'll have to check how this works. It may take a bit of time until I come back with something

pandas/_libs/tslibs/offsets.pyx

jreback

small comments, pls rebase

asv_bench/benchmarks/tslibs/strftime.py

pandas/core/arrays/period.py

…ure/44764_strftime_perf_datetimes � Conflicts: � doc/source/whatsnew/v1.5.0.rst � pandas/_libs/tslib.pyx

smarie · 2022-06-08T06:48:40Z

It seems that everything is ok now. Thanks for this review!

smarie · 2022-06-14T14:42:44Z

@jreback : could we move forward on this ? thanks in advance!

jreback

this looks fine. @jbrockmendel @mroeschke if any comments

…ure/44764_strftime_perf_datetimes � Conflicts: � pandas/_libs/tslib.pyx

…uct`

smarie · 2022-06-16T11:25:24Z

Note after updating the branch to the latest main: a recent mod in format_array_from_datetime introduced a new parameter reso, without documenting it in the docstring :(
I tried my best to guess what I should do for the "basic format day" branch but maybe it is not the right interpretation.

@jbrockmendel @jreback @mroeschke if one of you originated this mod, could you please suggest a docstring sentence and check that my interpretation (of not taking it into account for basic format day branch) is still relevant ?

EDIT: @jbrockmendel answered so this is OK now.

pandas/_libs/tslib.pyx

…ure/44764_strftime_perf_datetimes � Conflicts: � pandas/_libs/tslib.pyx

smarie · 2022-06-20T19:15:37Z

Ping ?
I rebased one more time, and once again it seems that there are some unrelated issues in two runners.
@jbrockmendel , @mroeschke , can you confirm @jreback 's approval and proceed to merge ? Thanks in advance

…ure/44764_strftime_perf_datetimes � Conflicts: � doc/source/whatsnew/v1.5.0.rst

smarie · 2022-07-01T14:53:12Z

I updated the PR to the latest head, it builds ok. @jreback approved.

Could we proceed ? @jreback @mroeschke @jbrockmendel

jreback · 2022-07-01T22:17:25Z

thanks @smarie

smarie · 2022-07-04T10:00:41Z

Great ! Thanks all for the reviews

jbrockmendel · 2022-07-06T21:17:36Z

asv_bench/benchmarks/tslibs/strftime.py

@@ -0,0 +1,64 @@
+import numpy as np
+
+import pandas as pd


sorry for the late review; these benchmarks don't belong in the tslibs directory (unless you can re-write them to not depend on anything outside of tslibs). can you do a follow-up to move them?

yes sure, do you think that moving them one folder up would be enough ?

that would do it, yes. thanks

Done in #47665 . Let's wait for a successful build just in case..

…libs dir (#47665) Code review from #46759 : moved benchmark file Co-authored-by: Sylvain MARIE <[email protected]>

…side of tslibs dir (pandas-dev#47665) Code review from pandas-dev#46759 : moved benchmark file Co-authored-by: Sylvain MARIE <[email protected]>

Performance improvement in :class:BusinessHour, repr is now 4 t…

01b9667

…imes faster ! Related to pandas-dev#44764

Sylvain MARIE added 10 commits April 12, 2022 23:37

For the record: switching the repr in TimeDelta to old-style formatting

139f229

added comment

75e8fd6

Minor: comment on speed

0beed46

added two comments for maintenance, about speed

cdffff8

Performance improvement in :class:DatetimeArray and :class:`Datetim…

903d08c

…eIndex`: string formatting is now up to 80% faster (as fast as default) when one of the default strftime formats ``"%Y-%m-%d %H:%M:%S"`` or ``"%Y-%m-%d %H:%M:%S.%f"`` is used. See pandas-dev#44764

A few comments related to time formatting

f6e4e5a

Performance improvement in :meth:Series.to_excel and :meth:`DataFra…

9851b36

…me.to_excel` (:class:`ExcelFormatter`): processing dates can be up to 4% faster. (related to :issue:`44764`)

Added maintenance comments

d302fe1

Performance improvement in Series.to_sql and DataFrame.to_sql (`S…

54f51f0

…QLiteTable`): processing time arrays can be up to 65% faster ! (related to pandas-dev#44764)

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

378e7f4

…ure/44764_strftime_perf_datetimes

smarie changed the title ~~[WIP] Some faster date/time string formatting~~ [READY] Some faster date/time string formatting Apr 12, 2022

jreback requested changes Apr 13, 2022

View reviewed changes

doc/source/whatsnew/v1.5.0.rst Outdated Show resolved Hide resolved

doc/source/whatsnew/v1.5.0.rst Outdated Show resolved Hide resolved

doc/source/whatsnew/v1.5.0.rst Outdated Show resolved Hide resolved

pandas/_libs/tslibs/timedeltas.pyx Outdated Show resolved Hide resolved

jreback added the Performance Memory or execution speed performance label Apr 13, 2022

mroeschke reviewed Apr 13, 2022

View reviewed changes

pandas/io/sql.py Outdated Show resolved Hide resolved

Sylvain MARIE added 5 commits April 13, 2022 09:52

Revert "For the record: switching the repr in TimeDelta to old-style …

6d60a04

…formatting" This reverts commit 139f229

Updated what's new wrt code review

705471f

Fixed what's new error

a5203ff

Simplified sqlite adapter for time to handle both with and without ti…

1c00b3a

…mezone. Made it faster using f-string

Fixed test

2fde239

smarie commented Apr 13, 2022

View reviewed changes

pandas/_libs/tslibs/offsets.pyx Outdated Show resolved Hide resolved

Update pandas/_libs/tslibs/offsets.pyx

cb8b6d2

smarie mentioned this pull request May 2, 2022

[WIP] perf improvements for strftime #46116

Closed

5 tasks

Sylvain MARIE added 3 commits May 12, 2022 11:49

Fixed Flake8 warning

1e49c05

Added ASVs for strftime related perf

405a5f6

Removed all other offsets from the ASV as they are not relevant

76035d5

As per code review: moved assert outside of for loop

1b991e5

jreback requested changes Jun 5, 2022

View reviewed changes

asv_bench/benchmarks/tslibs/strftime.py Outdated Show resolved Hide resolved

pandas/core/arrays/period.py Outdated Show resolved Hide resolved

Sylvain MARIE added 3 commits June 7, 2022 22:39

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

0a6e8fb

…ure/44764_strftime_perf_datetimes � Conflicts: � doc/source/whatsnew/v1.5.0.rst � pandas/_libs/tslib.pyx

As per code review: removed 2 useless variables and edited a comment

3e42fd4

Fixed wrong merge

fc078b9

smarie requested a review from jreback June 8, 2022 19:35

jreback approved these changes Jun 15, 2022

View reviewed changes

Sylvain MARIE added 2 commits June 16, 2022 13:17

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

e410b31

…ure/44764_strftime_perf_datetimes � Conflicts: � pandas/_libs/tslib.pyx

Aligned the basic format day with new `pandas_datetime_to_datetimestr…

32b03a8

…uct`

smarie commented Jun 16, 2022

View reviewed changes

pandas/_libs/tslib.pyx Show resolved Hide resolved

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

6561e77

…ure/44764_strftime_perf_datetimes � Conflicts: � pandas/_libs/tslib.pyx

Removed useless line

7adff9b

jreback changed the title ~~[READY] Some faster date/time string formatting~~ PERF: Some faster date/time string formatting Jun 21, 2022

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

d104bb4

…ure/44764_strftime_perf_datetimes � Conflicts: � doc/source/whatsnew/v1.5.0.rst

jreback merged commit 55d9dcf into pandas-dev:main Jul 1, 2022

smarie deleted the feature/44764_strftime_perf_datetimes branch July 4, 2022 10:00

jbrockmendel reviewed Jul 6, 2022

View reviewed changes

smarie pushed a commit to smarie/pandas that referenced this pull request Jul 11, 2022

Code review from pandas-dev#46759 : moved benchmark file

96114b1

smarie mentioned this pull request Jul 11, 2022

Code review from #46759 : moved strftime benchmark file outside of tslibs dir #47665

Merged

mroeschke pushed a commit that referenced this pull request Jul 11, 2022

Code review from #46759 : moved strftime benchmark file outside of ts…

6867bab

…libs dir (#47665) Code review from #46759 : moved benchmark file Co-authored-by: Sylvain MARIE <[email protected]>

yehoshuadimarsky pushed a commit to yehoshuadimarsky/pandas that referenced this pull request Jul 13, 2022

PERF: Some faster date/time string formatting (pandas-dev#46759)

ff6613e

smarie mentioned this pull request May 14, 2023

[READY] perf improvements for strftime #51298

Open

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PERF: Some faster date/time string formatting #46759

PERF: Some faster date/time string formatting #46759

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Apr 12, 2022

jreback left a comment

smarie commented Apr 13, 2022

jreback left a comment

smarie commented Jun 8, 2022 •

edited

Loading

smarie commented Jun 14, 2022

jreback left a comment

smarie commented Jun 16, 2022 •

edited

Loading

smarie commented Jun 20, 2022 •

edited

Loading

smarie commented Jul 1, 2022

jreback commented Jul 1, 2022

smarie commented Jul 4, 2022

jbrockmendel Jul 6, 2022

smarie Jul 7, 2022

jbrockmendel Jul 7, 2022

smarie Jul 11, 2022

PERF: Some faster date/time string formatting #46759

PERF: Some faster date/time string formatting #46759

Conversation

smarie commented Apr 12, 2022 • edited Loading

smarie commented Apr 12, 2022 • edited Loading

smarie commented Apr 12, 2022 • edited Loading

smarie commented Apr 12, 2022

jreback left a comment

Choose a reason for hiding this comment

smarie commented Apr 13, 2022

jreback left a comment

Choose a reason for hiding this comment

smarie commented Jun 8, 2022 • edited Loading

smarie commented Jun 14, 2022

jreback left a comment

Choose a reason for hiding this comment

smarie commented Jun 16, 2022 • edited Loading

smarie commented Jun 20, 2022 • edited Loading

smarie commented Jul 1, 2022

jreback commented Jul 1, 2022

smarie commented Jul 4, 2022

jbrockmendel Jul 6, 2022

Choose a reason for hiding this comment

smarie Jul 7, 2022

Choose a reason for hiding this comment

jbrockmendel Jul 7, 2022

Choose a reason for hiding this comment

smarie Jul 11, 2022

Choose a reason for hiding this comment

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Apr 12, 2022 •

edited

Loading

smarie commented Jun 8, 2022 •

edited

Loading

smarie commented Jun 16, 2022 •

edited

Loading

smarie commented Jun 20, 2022 •

edited

Loading