BUG: GH29461 Strftime #34668

matteosantama · 2020-06-09T15:29:49Z

closes Timestamp.strftime(): missing support for nanoseconds #29461
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

Closed the other PR while I sorted some things out.

… compatability with datetime.strptime()

…rftime

asv_bench/benchmarks/timeseries.py

asv_bench/benchmarks/tslibs/timestamp.py

pandas/_libs/tslibs/timestamps.pyx

jreback · 2020-06-09T17:28:50Z

pandas/_libs/tslibs/timestamps.pyx

-            result += f'.{self.microsecond:06d}'
-
-        return result
+        fmt = '%H:%M:%S'


does this change any of the benchmarks?

Looks like we suffer a penalty for the additional super() function call in strftime(), and then there's an additional performance impact when we actually need to process nanoseconds.

before after ratio [6efb0b20] [0c4ef36f] <master> <strftime> + 3.61±0.1μs 4.40±0.2μs 1.22 tslibs.timestamp.TimestampStrftimeMethod.time_strftime('%Y-%m-%d %H:%M:%S.%f') + 3.29±0.1μs 3.68±0.05μs 1.12 tslibs.timestamp.TimestampStrftimeMethod.time_strftime('%Y-%m-%d %H:%M:%S') + 8.91±0.2μs 9.90±0.4μs 1.11 tslibs.timestamp.TimestampProperties.time_month_name(None, None) - 409±20ns 308±10ns 0.75 tslibs.timestamp.TimestampProperties.time_days_in_month(<UTC>, 'B')

jreback · 2020-06-09T17:29:11Z

pandas/_libs/tslibs/timestamps.pyx

+        """
+        if '%f' in format and self.nanosecond:
+            replacement = f'{self.microsecond * 1000 + self.nanosecond:09d}'
+            format = re.sub('%f', replacement, format)


can you use string replacement rather than re? doesn't this change perf?

Good call. Tested both and string replace is WAY more performant.

…rftime

TomAugspurger · 2020-07-06T14:52:13Z

@matteosantama can you merge master and fix the conflicts?

jreback · 2020-07-17T11:04:07Z

can u merge master, and show benchmarks again; ping on green

…rftime

matteosantama · 2020-07-20T00:17:11Z

@jreback All checks passing. ASV output is a little confusing. Not entirely sure why so many tests were affected.

       before           after         ratio
     [bfac1362]       [b0de2c4c]
     <master>         <strftime>
+      6.44±0.4μs        34.8±30μs     5.41  tslibs.timestamp.TimestampProperties.time_is_month_end(tzfile('/usr/share/zoneinfo/US/Central'), 'B')
+         204±4ns       1.06±0.8μs     5.18  tslibs.timestamp.TimestampProperties.time_microsecond(tzlocal(), 'B')
+      8.78±0.3μs        42.3±30μs     4.81  tslibs.timestamp.TimestampProperties.time_month_name(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>, None)
+      7.77±0.2μs        19.6±10μs     2.53  tslibs.timestamp.TimestampProperties.time_month_name(<UTC>, 'B')
+      3.52±0.1μs       4.24±0.1μs     1.20  tslibs.timestamp.TimestampStrftimeMethod.time_strftime('%Y-%m-%d %H:%M:%S.%f')
+     3.16±0.06μs       3.60±0.1μs     1.14  tslibs.timestamp.TimestampStrftimeMethod.time_strftime('%Y-%m-%d %H:%M:%S')
-        314±10ns          284±3ns     0.90  tslibs.timestamp.TimestampProperties.time_week(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>, None)
-         277±5ns          250±9ns     0.90  tslibs.timestamp.TimestampProperties.time_days_in_month(<UTC>, None)
-        279±20ns          251±5ns     0.90  tslibs.timestamp.TimestampProperties.time_days_in_month(datetime.timezone(datetime.timedelta(seconds=3600)), 'B')
-        320±20ns          284±2ns     0.89  tslibs.timestamp.TimestampProperties.time_week(tzlocal(), None)
-        279±40ns          248±4ns     0.89  tslibs.timestamp.TimestampProperties.time_is_year_start(tzlocal(), None)
-        283±10ns          250±4ns     0.88  tslibs.timestamp.TimestampProperties.time_dayofweek(<UTC>, None)
-        137±10μs          120±2μs     0.88  tslibs.timestamp.TimestampOps.time_ceil(tzfile('/usr/share/zoneinfo/US/Central'))
-        548±10ns          475±4ns     0.87  tslibs.timestamp.TimestampProperties.time_freqstr(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>, None)
-        79.4±4μs       68.7±0.5μs     0.87  tslibs.timestamp.TimestampOps.time_ceil(<UTC>)
-        227±30ns          195±7ns     0.86  tslibs.timestamp.TimestampProperties.time_microsecond(tzfile('/usr/share/zoneinfo/US/Central'), None)
-      9.13±0.9μs       7.51±0.1μs     0.82  tslibs.timestamp.TimestampConstruction.time_parse_iso8601_tz
-       81.3±20μs       52.9±0.9μs     0.65  tslibs.timestamp.TimestampOps.time_replace_tz(tzlocal())
-        18.1±5μs       11.5±0.3μs     0.63  tslibs.timestamp.TimestampOps.time_to_julian_date(tzutc())
-       43.8±20μs       25.5±0.5μs     0.58  tslibs.timestamp.TimestampOps.time_replace_tz(tzfile('/usr/share/zoneinfo/US/Central'))
-        20.9±7μs       11.5±0.3μs     0.55  tslibs.timestamp.TimestampOps.time_to_julian_date(None)
-       70.4±60μs       31.1±0.5μs     0.44  tslibs.timestamp.TimestampOps.time_replace_None(tzlocal())
-        5.45±5μs      2.28±0.06μs     0.42  tslibs.timestamp.TimestampOps.time_replace_None(tzutc())
-       539±300ns          219±4ns     0.41  tslibs.timestamp.TimestampProperties.time_is_quarter_start(tzfile('/usr/share/zoneinfo/US/Central'), None)
-        15.8±8μs       6.10±0.1μs     0.39  tslibs.timestamp.TimestampProperties.time_is_year_start(tzutc(), 'B')
-        15.2±8μs      5.82±0.08μs     0.38  tslibs.timestamp.TimestampProperties.time_is_month_start(None, 'B')
-        9.42±5μs       3.60±0.1μs     0.38  tslibs.timestamp.TimestampOps.time_normalize(None)
-       52.7±30μs       20.1±0.7μs     0.38  tslibs.timestamp.TimestampOps.time_replace_tz(<UTC>)
-       575±300ns          214±8ns     0.37  tslibs.timestamp.TimestampProperties.time_is_month_start(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>, None)
-        108±70μs       34.8±0.6μs     0.32  tslibs.timestamp.TimestampOps.time_normalize(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>)
-        7.23±5μs      2.33±0.05μs     0.32  tslibs.timestamp.TimestampOps.time_replace_None(<UTC>)
-       142±100μs       40.7±0.6μs     0.29  tslibs.timestamp.TimestampOps.time_normalize(tzlocal())
-       137±100μs         36.8±1μs     0.27  tslibs.timestamp.TimestampOps.time_normalize(tzfile('/usr/share/zoneinfo/US/Central'))
-       13.1±10μs      3.47±0.09μs     0.27  tslibs.timestamp.TimestampOps.time_normalize(<UTC>)
-        1.28±1μs          245±5ns     0.19  tslibs.timestamp.TimestampProperties.time_is_year_start(tzfile('/usr/share/zoneinfo/US/Central'), None)
-       647±500μs          123±1μs     0.19  tslibs.timestamp.TimestampOps.time_floor(tzfile('/usr/share/zoneinfo/US/Central'))
-      1.15±0.9μs          217±5ns     0.19  tslibs.timestamp.TimestampProperties.time_is_month_start(None, None)
-        2.54±2μs          476±3ns     0.19  tslibs.timestamp.TimestampProperties.time_freqstr(tzutc(), None)
-       362±300μs         67.5±2μs     0.19  tslibs.timestamp.TimestampOps.time_floor(<UTC>)
-        1.63±1μs          304±2ns     0.19  tslibs.timestamp.TimestampProperties.time_freqstr(datetime.timezone(datetime.timedelta(seconds=3600)), 'B')
-        1.33±1μs          248±3ns     0.19  tslibs.timestamp.TimestampProperties.time_is_year_start(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>, None)
-        1.29±1μs          240±3ns     0.19  tslibs.timestamp.TimestampOps.time_tz_localize(<UTC>)
-        1.64±1μs          305±6ns     0.19  tslibs.timestamp.TimestampProperties.time_freqstr(tzutc(), 'B')
-        1.28±1μs          236±4ns     0.18  tslibs.timestamp.TimestampProperties.time_dayofweek(None, 'B')
-        1.21±1μs          222±3ns     0.18  tslibs.timestamp.TimestampProperties.time_is_quarter_end(tzlocal(), None)
-        2.57±2μs          470±5ns     0.18  tslibs.timestamp.TimestampProperties.time_freqstr(tzlocal(), None)
-       859±700μs          154±4μs     0.18  tslibs.timestamp.TimestampOps.time_ceil(tzlocal())
-       668±500μs          119±1μs     0.18  tslibs.timestamp.TimestampOps.time_floor(<DstTzInfo 'Europe/Amsterdam' LMT+0:20:00 STD>)
-       386±300μs         67.0±2μs     0.17  tslibs.timestamp.TimestampOps.time_floor(tzutc())
-       564±500μs         91.5±2μs     0.16  tslibs.timestamp.TimestampOps.time_floor(datetime.timezone(datetime.timedelta(seconds=3600)))
-       945±900μs          149±3μs     0.16  tslibs.timestamp.TimestampOps.time_floor(tzlocal())

WillAyd

Looks like a merge conflict - can you move note to 1.2?

WillAyd · 2020-09-01T21:38:23Z

pandas/_libs/tslibs/timestamps.pyx

+        str
+            String representation of Timestamp
+        """
+        if self.nanosecond and '%f' in format:


Is there a reason for this check and the one on L649? Can there just be one check done here?

jreback · 2020-09-01T23:48:22Z

doc/source/whatsnew/v1.1.0.rst

@@ -915,6 +915,7 @@ Datetimelike
 - Bug in :meth:`DatetimeIndex.intersection` and :meth:`TimedeltaIndex.intersection` with results not having the correct ``name`` attribute (:issue:`33904`)
 - Bug in :meth:`DatetimeArray.__setitem__`, :meth:`TimedeltaArray.__setitem__`, :meth:`PeriodArray.__setitem__` incorrectly allowing values with ``int64`` dtype to be silently cast (:issue:`33717`)
 - Bug in subtracting :class:`TimedeltaIndex` from :class:`Period` incorrectly raising ``TypeError`` in some cases where it should succeed and ``IncompatibleFrequency`` in some cases where it should raise ``TypeError`` (:issue:`33883`)
+- Bug in :meth:`Timestamp.strftime` did not display full nanosecond precision (:issue:`29461`)


move to 1.2

MarcoGorelli · 2020-09-25T06:58:37Z

Hi @matteosantama - is this still active? If so, could you address Jeff's and Will's comments please?

matteosantama and others added 30 commits May 19, 2020 23:24

TST: GH28813 test .diff() on Sparse dtype

12d0b4d

TST: GH28813 test .diff() on Sparse dtype

02c4a85

TST: GH28813 pull sparse diff() test into its own function

7e3256b

Merge branch 'sparse_diff'

f00e48a

Merge branch 'master' of https://github.com/pandas-dev/pandas

a90f928

BUG: GH29461 display nanoseconds with strftime()

26a920c

BUG: GH29461 don't display nanoseconds in strftime if none exists for…

3a529ac

… compatability with datetime.strptime()

update whatsnew doc

d0124aa

black formatting

0c0aaf2

remove trailing whitespace to conform to CI

abdbe4e

still trying to pass linting checks

97ae6b3

Merge branch 'master' of https://github.com/pandas-dev/pandas

1132aba

Merge branch 'master' of github.com:matteosantama/pandas into strftime

0e42026

Add strftime benchmarks

ab0c9d4

Early exit strftime if no nanoseconds

532b19c

Make loop more pythonic

88fba5e

Fix benchmark test

2ac690c

Remove whitespace

3b533c9

Remove extra function call

f6ba1c9

Benchmark series.strftime()

07b27e2

Merge branch 'master' of https://github.com/pandas-dev/pandas into st…

4b57c4f

…rftime

Use explicitly named parameters in testing

d72222a

Use regex for replacing %f

e920d20

Clean up Timestamp._time_repr to use new strftime functionality

34db469

Commiting so I can merge master

fbe286e

Merge branch 'master' of https://github.com/pandas-dev/pandas into st…

54d30eb

…rftime

Use regex for replacing %f

90629c5

Clean up _time_repr to use new strftime functionality

821dfbb

Test for all datetime strftime directives

16b0f9f

Test for all datetime strftime directives

b8565f2

matteosantama added 8 commits June 9, 2020 09:28

Test for all datetime strftime directives

6866a3d

Merge branch 'master' of https://github.com/pandas-dev/pandas into st…

3806567

…rftime

Call super strftime instead of time strftime

a4ddcbb

Improve docstring

7fe0a5e

Remove whitespace in docstring

ce9ef3d

Only show fractional seconds if they exist

7b69abb

Docstring not building correctly

858b3fb

Fixed typo

e7b8525

jreback requested changes Jun 9, 2020

View reviewed changes

jreback added Output-Formatting __repr__ of pandas objects, to_string Datetime Datetime data dtype labels Jun 9, 2020

matteosantama added 6 commits June 9, 2020 12:48

Check nanoseconds first

6a2f3d2

Merge branch 'master' of https://github.com/pandas-dev/pandas into st…

d62ff1d

…rftime

Rename testing classes

cbb735e

Use string replace instead of re package

0c4ef36

Fix test parametrization

62c1126

Fix test parametrization

4468168

matteosantama added 2 commits July 7, 2020 10:01

Resolve merge conflicts

88c7e26

Update docstring for NaT to match Timestamp

4d433f9

Merge branch 'master' of https://github.com/pandas-dev/pandas into st…

b0de2c4

…rftime

WillAyd reviewed Sep 1, 2020

View reviewed changes

jreback requested changes Sep 1, 2020

View reviewed changes

matteosantama closed this Sep 25, 2020

smarie mentioned this pull request Feb 24, 2022

[WIP] perf improvements for strftime #46116

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: GH29461 Strftime #34668

BUG: GH29461 Strftime #34668

matteosantama commented Jun 9, 2020

jreback Jun 9, 2020

matteosantama Jun 11, 2020 •

edited

Loading

jreback Jun 9, 2020

matteosantama Jun 11, 2020

TomAugspurger commented Jul 6, 2020

jreback commented Jul 17, 2020

matteosantama commented Jul 20, 2020

WillAyd left a comment

WillAyd Sep 1, 2020

jreback Sep 1, 2020

MarcoGorelli commented Sep 25, 2020

BUG: GH29461 Strftime #34668

BUG: GH29461 Strftime #34668

Conversation

matteosantama commented Jun 9, 2020

jreback Jun 9, 2020

Choose a reason for hiding this comment

matteosantama Jun 11, 2020 • edited Loading

Choose a reason for hiding this comment

jreback Jun 9, 2020

Choose a reason for hiding this comment

matteosantama Jun 11, 2020

Choose a reason for hiding this comment

TomAugspurger commented Jul 6, 2020

jreback commented Jul 17, 2020

matteosantama commented Jul 20, 2020

WillAyd left a comment

Choose a reason for hiding this comment

WillAyd Sep 1, 2020

Choose a reason for hiding this comment

jreback Sep 1, 2020

Choose a reason for hiding this comment

MarcoGorelli commented Sep 25, 2020

matteosantama Jun 11, 2020 •

edited

Loading