[WIP] perf improvements for strftime#46116

Closed

smarie wants to merge 68 commits intopandas-dev:mainfrom

smarie:feature/44764_perf_issue

Commits on Feb 22, 2022

Added the default format "%Y-%m-%d %H:%M:%S" as basic format (same as format=None), as suggested by @auderson.
Sylvain MARIE
committed

Commits on Feb 24, 2022

Created two utils `convert_dtformat` and `get_datetime_fmt_dct` so as to be able to replace strftime faster. The result can be demonstrated using `fast_strftime`.
Sylvain MARIE
committed
Fixed two pep8
Sylvain MARIE
committed

Commits on Feb 28, 2022

Commits on Mar 1, 2022

Fixed issue in tests happening when format=None
Sylvain MARIE
committed
isort
Sylvain MARIE
committed
Fixed `BusinessHour._repr_attrs` so that it is faster too.
Sylvain MARIE
committed
Added TODO for possible improvement in timedeltas
Sylvain MARIE
committed
Added maintenance comments concerning using Timestamp.str
Sylvain MARIE
committed
Improved/fix periods handling. New c function `_period_fast_strftime`.
Sylvain MARIE
committed
Fixed period string representation %FQ%q
Sylvain MARIE
committed
Relaxed perf test for datetimes
Sylvain MARIE
committed

Commits on Mar 2, 2022

Commits on Mar 4, 2022

New `Timestamp.fast_strftime`, leveraged by `format_array_from_datetime`. Fixes issue related to timezone-aware timestamp formatting. Added corresponding tests.
Sylvain MARIE
committed
Fixed last failing test
Sylvain MARIE
committed
Fixed `test_missing_public_nat_methods`
Sylvain MARIE
committed
Blackened
Sylvain MARIE
committed
Fixed doctests
Sylvain MARIE
committed

Commits on Mar 7, 2022

Commits on Mar 10, 2022

Added `AM_LOCAL` and `PM_LOCAL` constants fed at initial import time using `strftime`, and reused in fast formatting ops. New function `get_local_ampm` gives an access for inspection.
Sylvain MARIE
committed
Accelerated ODS writer when dates need to be written: at least one of the strftime operations was removed
Sylvain MARIE
committed
Merge branch 'main' of https://github.com/pandas-dev/pandas into feature/44764_perf_issue
Sylvain MARIE
committed
Using `bool_t` instead of `bool` in type hints of core/generic.py
Sylvain MARIE
committed
isort
Sylvain MARIE
committed
flake8 and black
Sylvain MARIE
committed
Docstring fix
Sylvain MARIE
committed
Fixed test namespace
Sylvain MARIE
committed
`AM_LOCAL` and `PM_LOCAL` are now `str` - this should fix the issue on arm and win32 platforms where cython does not seem to intelligently fix the string formatting incompatibility.
Sylvain MARIE
committed
Fixed test_format and removed the performance test, as it would fail on ci when parallel cpus/tests are used
Sylvain MARIE
committed
flake8 black and isort
Sylvain MARIE
committed
Added missing type hint declaration for the `fast_strftime` methods (mypy)
Sylvain MARIE
committed
Now catching the warnings about time zone being dropped in test during dt to period conversion
Sylvain MARIE
committed
Fixed locale-related issue: AM_LOCAL and PM_LOCAL were created at compile time so they were not using the right (runtime) locale. Added a test.
Sylvain MARIE
committed
Replaced `pytest.warns` with pandas `assert_produces_warning`
Sylvain MARIE
committed
Fixed `CSVFormatter` (used in `to_csv`) so that `DatetimeIndex` or `PeriodIndex` are now formatted using the same routine than columns of the same type. In particular `strftime` is not used anymore…
Sylvain MARIE
committed
Fixed locale problem and improved test
Sylvain MARIE
committed
isort
Sylvain MARIE
committed
Reverted the mods as it appears it was not this problem
Sylvain MARIE
committed
Fixed failing test on non-en locales
Sylvain MARIE
committed
Revert "Reverted the mods as it appears it was not this problem"
Sylvain MARIE
committed

Commits on Mar 11, 2022

New `LocaleSpecificDtStrings` objects are returned by `convert_strftime_format` so that the caller has the correct elements to format in other locales.
Sylvain MARIE
committed
Removed useless argument in fast_strftime
Sylvain MARIE
committed
Fixed #46319 by using `PyUnicode_DecodeLocale` to decode the string returned by `c_strftime`.
Sylvain MARIE
committed
flake8 black and isort
Sylvain MARIE
committed
black
Sylvain MARIE
committed
Fixed `_format_datetime64_dateonly` used in `get_format_datetime64`. It was not covered by any test! TODO
Sylvain MARIE
committed
Removed the multi-locale test as it fails on some ci targets
Sylvain MARIE
committed
Fixed var init issue
Sylvain MARIE
committed

Commits on Mar 12, 2022

Flake8
Sylvain MARIE
committed
Improved `Datetime64TZFormatter` readability by replacing useless call to `is_dates_only` by an assert that the result is False. Added a corresponding test
Sylvain MARIE
committed
Flake8 fix !
Sylvain MARIE
committed

Commits on Mar 13, 2022

Merge branch 'main' of https://github.com/pandas-dev/pandas into feature/44764_perf_issue
Sylvain MARIE
committed

Commits on Mar 27, 2022

Merge branch 'main' of https://github.com/pandas-dev/pandas into feature/44764_perf_issue
Sylvain MARIE
committed