[WIP] perf improvements for strftime #46116

smarie · 2022-02-22T17:12:49Z

Changelog

Added the default format "%Y-%m-%d %H:%M:%S" as basic format (same as format=None), as suggested by @auderson.
closes PERF: strftime is slow #44764 , closes BUG: Period formatting directive %l (millisecond) and %u (microseconds) provide wrong results #46252, closes BUG: UnicodeError when using Period.strftime with non-utf8 locale #46319
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.
Post timeit results showing the perf improvements

@auderson

… format=None), as suggested by @auderson.

jbrockmendel · 2022-02-23T18:13:58Z

Can you post %timeit results showing the perf improvement? (edit OP)

smarie · 2022-02-24T14:07:17Z

Sure. I edited the todo list accordingly.

smarie · 2022-02-24T16:21:35Z

Note: there seem to be a related (but distinct) issue in #29461 , solved by #34668 .

Still I will stick to the current road where I modify the format_array_from_datetime function only.

… to be able to replace strftime faster. The result can be demonstrated using `fast_strftime`.

pep8speaks · 2022-02-24T17:17:16Z

Hello @smarie! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2022-03-27 14:05:05 UTC

smarie · 2022-02-24T17:25:17Z

First milestone reached: all utility functions required to create the fast datetime formatting strings are there.

A fast_strftime util function can be used to demonstrate (and test) their performance on plain old datetime objects. Added relevant tests for this.

Next step: modify format_array_from_datetime so that it relies on the same underlying function convert_dtformat in order to format npy_datetimestruct instances.

…and `UnsupportedDatetimeDirective`. Modified `format_array_from_datetime` to use it when `fast_strftime=True` (new arg). Removed `get_datetime_fmt_dct`

…epts `fast_strftime`. Modified `PeriodArray._format_native_types` so that it benefits from `fast_strftime` too.

…ter`, `Datetime64TZFormatter` and `DatetimeIndex`.

…nce on arrays

…extra directives

…iod_strftime %q format. Also accelerated slightly the _period_strftime.

… code was right, not the docstring. Fixed the docstring

…tting symbols (% or {} depending on formatting style)

…pecial characters. New `Period.fast_strftime` used in `PeriodArray`. Renamed `convert_dtformat` into `convert_strftime_format` and `UnsupportedDatetimeDirective` into `UnsupportedStrFmtDirective`.

smarie · 2022-03-11T20:54:05Z

Noted - I'll at least try to move the two small fixes for Period into separate PRs, before this final big one about fast_strftime is reviewed.

…It was not covered by any test! TODO

…l to `is_dates_only` by an assert that the result is False. Added a corresponding test

smarie · 2022-03-12T14:37:32Z

Quick status on this PR: today I reached a point where all CI checks pass (finally !!!), and therefore I was able to try reproducing the benchmark in #44764 . Results are pretty good !

(blue) the "legacy" curve corresponds to using strftime. It seems even worse on my machine (windows 10) than the result reported in PERF: strftime is slow #44764 !
(orange) the "manual" curve corresponds to the same manual method as in PERF: strftime is slow #44764. It is as good as reported in PERF: strftime is slow #44764
(green) the "new and legacy default" represents the case where date_format=None (default format). It is as good as reported in PERF: strftime is slow #44764
(red) the "new custom" is the main use case why this PR was created: a custom date format. It was improved a lot as compared to the blue curve !
(violet) the "new basic formats" represents the case reported by @auderson where the custom date format is actually identical to the default format (green). We see that both now have the same perf

```python import time import numpy as np import pandas as pd from joblib import Parallel, delayed

def timer(f):
def inner(*args, **kwargs):
s = time.time()
result = f(*args, **kwargs)
e = time.time()
return e - s
return inner

@Timer
def method_legacy(index):
return index.strftime("%Y-%m-%d_%H:%M:%S", fast_strftime=False)

@Timer
def method_manual(index):
attrs = ('year', 'month', 'day', 'hour', 'minute', 'second')
parts = [getattr(index, at) for at in attrs]
b = []
for year, month, day, hour, minute, second in zip(*parts):
b.append(f'{year}-{month:02}-{day:02}_{hour:02}:{minute:02}:{second:02}')
b = pd.Index(b)
return b

@Timer
def method_new_and_legacy_default(index):
return index.strftime(None)

@Timer
def method_new_custom(index):
return index.strftime("%Y-%m-%d_%H:%M:%S")

@Timer
def method_new_basic_formats(index):
return index.strftime("%Y-%m-%d %H:%M:%S")

index = pd.date_range('2000', '2020', freq='1min')

@delayed
def profile(p):
n = int(10 ** p)
time_legacy = method_legacy(index[:n])
time_manual = method_manual(index[:n])
time_new_and_legacy_default = method_new_and_legacy_default(index[:n])
time_new_custom = method_new_custom(index[:n])
time_new_basic_formats = method_new_basic_formats(index[:n])
return n, time_legacy, time_manual, time_new_and_legacy_default,
time_new_custom, time_new_basic_formats

records = Parallel(10, verbose=10, max_nbytes='500M')(
profile(p) for p in np.arange(1, 7.1, 0.1)
)

pd.DataFrame(records, columns=['n', 'time_legacy', 'time_manual',
'time_new_and_legacy_default', 'time_new_custom',
'time_new_basic_formats'
]).set_index(
'n').plot(figsize=(10, 8))

import matplotlib.pyplot as plt
plt.show()

</details>

jreback · 2022-03-12T14:41:26Z

@smarie thins way too large a PR

pls split this up into logical pieces

smarie · 2022-03-12T14:52:28Z

@jreback yes I was going to split it as explained above, but I wanted first to reach a "clean" state where CI is happy and the benchmark shows good results. Now I will try (next week) to split this into pieces, but as already mentioned, there is not much I can do here: strftime was used in many places (Period, Timestamp, csv formatter, excel formatter, sqlite adapter, some repr methods, etc.) and all of them needed replacement. I will do my best do separate between

bug fixes
quick fixes for all repr and the like (not requiring core changes)
the "core" replacement for datetime arrays
the same for periods
... maybe other similar

auderson · 2022-03-13T01:49:11Z

This is impressive😀

…ure/44764_perf_issue

jbrockmendel · 2022-03-15T05:42:01Z

pandas/_libs/tslibs/period.pyx

@@ -2340,14 +2392,70 @@ cdef class _Period(PeriodMixin):
        object_state = None, self.freq, self.ordinal
        return (Period, object_state)

+    def fast_strftime(self, fmt_str: str, loc_s: object) -> str:
+        """A faster alternative to `strftime` using string formatting.


nitpick newline after the """

jbrockmendel · 2022-03-17T16:15:18Z

pandas/_libs/tslib.pyx

@@ -101,7 +105,8 @@ def format_array_from_datetime(
    ndarray[int64_t] values,


if a tslibs.strftime does get implemented, that might be a good home for format_array_from_datetime

…ure/44764_perf_issue � Conflicts: � pandas/_libs/tslibs/period.pyx � pandas/core/arrays/period.py � pandas/tests/io/formats/test_format.py

github-actions · 2022-04-27T00:05:54Z

This pull request is stale because it has been open for thirty days with no activity. Please update and respond to this comment if you're still interested in working on this.

smarie · 2022-05-02T20:07:14Z

bot: please leave this open. #46759 and #46405 are the next bits of this PR to consider

Paul-AUB · 2022-06-17T08:26:36Z

Hello @smarie, it seems there is a flake8 issue (pandas/tests/io/formats/test_format.py:3253:5: F811 redefinition of unused 'test_period_custom' from line 3201) to fix and a conflict to solve before this can be merged. Could you do that please ? Feature is really interesting !

smarie · 2022-06-17T20:34:50Z

Thanks @Paul-AUB ! Actually this PR is just here as a reference, it is split in several pieces. #46759 and #46405 are the next bits of this PR to consider, the first is actually quite close to be merged. The second has a side effect in the CI concerning some locale-dependent setting, that needs to be investigated. Once #46405 is merged, I'll finally update this PR or create a new one with the same contents, so that it contains the remaining bit (the engine able to improve any strftime format)

mroeschke · 2022-12-06T17:54:42Z

Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.

smarie · 2022-12-12T13:56:51Z

Hi @mroeschke, yes I plan to finish this now that most of its "other aspects" have been merged (through other PRs #46361, #46759, #46405, #47570). I'll try to find some time later this month

Added the default format "%Y-%m-%d %H:%M:%S" as basic format (same as…

1062e50

… format=None), as suggested by @auderson.

Created two utils convert_dtformat and get_datetime_fmt_dct so as…

8d07798

… to be able to replace strftime faster. The result can be demonstrated using `fast_strftime`.

Fixed two pep8

4b2380e

jreback changed the title ~~[Work in progress] perf improvements for strftime~~ [WIP] perf improvements for strftime Feb 27, 2022

jreback added Performance Memory or execution speed performance Datetime Datetime data dtype labels Feb 27, 2022

Sylvain MARIE added 19 commits February 28, 2022 15:05

Added new strftime module in tslibs, containing convert_dtformat …

6a3708e

…and `UnsupportedDatetimeDirective`. Modified `format_array_from_datetime` to use it when `fast_strftime=True` (new arg). Removed `get_datetime_fmt_dct`

Modified DatetimeLikeArrayMixin._format_native_types so that it acc…

93f0dbe

…epts `fast_strftime`. Modified `PeriodArray._format_native_types` so that it benefits from `fast_strftime` too.

Added a fast_strftime argument to all formatters: `Datetime64Format…

5b9dd1b

…ter`, `Datetime64TZFormatter` and `DatetimeIndex`.

Added a test in TestDatetimeFastFormatter so that it tests performa…

8328feb

…nce on arrays

Fixed issue in tests happening when format=None

9bba33b

isort

fe8aa3b

Fixed BusinessHour._repr_attrs so that it is faster too.

a3cadec

Added TODO for possible improvement in timedeltas

90c836a

Added maintenance comments concerning using Timestamp.str

d394a1b

Improved/fix periods handling. New c function _period_fast_strftime.

460504d

Fixed period string representation %FQ%q

d700c44

Relaxed perf test for datetimes

1a44d02

Removed 2 useless cdefs and fixed docstring of strftime with missing …

61ec6d6

…extra directives

Fixed issue with BusinessHour formatting

846a0fd

Fixed _period_fast_strftime FR_QTR frequency default format, and _per…

84f4c7f

…iod_strftime %q format. Also accelerated slightly the _period_strftime.

Fixed the quarter issue introduced in previous commit. Apparently the…

e96fe8d

… code was right, not the docstring. Fixed the docstring

Improved convert_dtformat so that it safely escapes remaining forma…

707a084

…tting symbols (% or {} depending on formatting style)

convert_dtformat now supports periods formatting and is robust to s…

37f15c9

…pecial characters. New `Period.fast_strftime` used in `PeriodArray`. Renamed `convert_dtformat` into `convert_strftime_format` and `UnsupportedDatetimeDirective` into `UnsupportedStrFmtDirective`.

Fixed api test

e2e2d70

Sylvain MARIE added 5 commits March 11, 2022 22:16

Fixed _format_datetime64_dateonly used in get_format_datetime64. …

399878a

…It was not covered by any test! TODO

Removed the multi-locale test as it fails on some ci targets

5b6e07e

Fixed var init issue

743106f

Flake8

44724af

Improved Datetime64TZFormatter readability by replacing useless cal…

718a647

…l to `is_dates_only` by an assert that the result is False. Added a corresponding test

smarie mentioned this pull request Mar 12, 2022

PERF: strftime is slow #44764

Open

3 tasks

Flake8 fix !

e6bb843

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

272f0a2

…ure/44764_perf_issue

smarie mentioned this pull request Mar 14, 2022

Fixed Period and PeriodIndex formatting #46361

Merged

5 tasks

jbrockmendel reviewed Mar 15, 2022

View reviewed changes

smarie mentioned this pull request Mar 17, 2022

BUG: Fixed Unicode decoding error in Period.strftime when a locale-specific directive is used #46405

Merged

4 tasks

jbrockmendel reviewed Mar 17, 2022

View reviewed changes

Merge branch 'main' of https://github.com/pandas-dev/pandas into feat…

12d279c

…ure/44764_perf_issue � Conflicts: � pandas/_libs/tslibs/period.pyx � pandas/core/arrays/period.py � pandas/tests/io/formats/test_format.py

smarie mentioned this pull request Apr 12, 2022

PERF: Some faster date/time string formatting #46759

Merged

4 tasks

github-actions bot added the Stale label Apr 27, 2022

simonjayhawkins mentioned this pull request May 20, 2022

BUG: strftime can give incorrect answers #46851

Open

3 tasks

mroeschke closed this Dec 6, 2022

smarie mentioned this pull request Feb 10, 2023

[READY] perf improvements for strftime #51298

Open

5 tasks

smarie mentioned this pull request Apr 8, 2024

ENH: consistent strftime behaviour #58179

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] perf improvements for strftime #46116

[WIP] perf improvements for strftime #46116

smarie commented Feb 22, 2022 •

edited

Loading

jbrockmendel commented Feb 23, 2022

smarie commented Feb 24, 2022

smarie commented Feb 24, 2022 •

edited

Loading

pep8speaks commented Feb 24, 2022 •

edited

Loading

smarie commented Feb 24, 2022

smarie commented Mar 11, 2022 •

edited

Loading

smarie commented Mar 12, 2022 •

edited

Loading

jreback commented Mar 12, 2022

smarie commented Mar 12, 2022

auderson commented Mar 13, 2022

jbrockmendel Mar 15, 2022

jbrockmendel Mar 17, 2022

github-actions bot commented Apr 27, 2022

smarie commented May 2, 2022

Paul-AUB commented Jun 17, 2022

smarie commented Jun 17, 2022 •

edited

Loading

mroeschke commented Dec 6, 2022

smarie commented Dec 12, 2022

		@@ -101,7 +105,8 @@ def format_array_from_datetime(
		ndarray[int64_t] values,

[WIP] perf improvements for strftime #46116

[WIP] perf improvements for strftime #46116

Conversation

smarie commented Feb 22, 2022 • edited Loading

jbrockmendel commented Feb 23, 2022

smarie commented Feb 24, 2022

smarie commented Feb 24, 2022 • edited Loading

pep8speaks commented Feb 24, 2022 • edited Loading

Comment last updated at 2022-03-27 14:05:05 UTC

smarie commented Feb 24, 2022

smarie commented Mar 11, 2022 • edited Loading

smarie commented Mar 12, 2022 • edited Loading

jreback commented Mar 12, 2022

smarie commented Mar 12, 2022

auderson commented Mar 13, 2022

jbrockmendel Mar 15, 2022

Choose a reason for hiding this comment

jbrockmendel Mar 17, 2022

Choose a reason for hiding this comment

github-actions bot commented Apr 27, 2022

smarie commented May 2, 2022

Paul-AUB commented Jun 17, 2022

smarie commented Jun 17, 2022 • edited Loading

mroeschke commented Dec 6, 2022

smarie commented Dec 12, 2022

smarie commented Feb 22, 2022 •

edited

Loading

smarie commented Feb 24, 2022 •

edited

Loading

pep8speaks commented Feb 24, 2022 •

edited

Loading

smarie commented Mar 11, 2022 •

edited

Loading

smarie commented Mar 12, 2022 •

edited

Loading

smarie commented Jun 17, 2022 •

edited

Loading