BUG: Add fix for hashing timestamps with folds

[jonwiggins]

• closes BUG: hash of Timestamp on fold=1 create a Segfault #33931, BUG: Reindexing a DatetimeIndex during Daylight Saving Transition Causes Segmentation Fault with dateutil tz #40817
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

Edit:

#42305 (comment)
This comment identifies the problem as being upstream in datetime's construction of timestamps which fails during some DST changes.

The datetime hash function then being used tries to unset the fold representing the DST change, (https://github.com/python/cpython/blob/main/Lib/datetime.py#L2117). But it does this using the datetime replace function - this change fixes that problem by removing the fold prior to calling the datetime hash function using timestamp's replace function instead.

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[mroeschke]

Could you use pytest.mark.parameterize over the timezone and TImestamp inputs?

[jbrockmendel]

try tests.scalar.timestamp.test_timestamp, there's another test with "hash" in the name that this can go right next to

[jreback]

can you add a whatsnew note abou this 1.4., datetime bug fix section.

also we'd like to assure that this isn't impacting perf very much. if you can run the datetime asv's would be great

[jonwiggins]

@jreback I don't see an asv set datetime specifically, so I tried to run the all of the ones for tslibs, let me know if you want me to run a different set.

-> asv continuous -f 1.1 -E virtualenv upstream/master fold_hash_fix -b ^tslibs
...

     [95123939]       [c6400d66]
     <master>         <fold_hash_fix>
-     1.47±0.02μs      1.34±0.03μs     0.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 1000, tzlocal())
-     4.40±0.03μs      3.99±0.09μs     0.91  tslibs.tslib.TimeIntsToPydatetime.time_ints_to_pydatetime('time', 0, <DstTzInfo 'US/Pacific' LMT-1 day, 16:07:00 STD>)
-     1.49±0.03μs      1.35±0.04μs     0.91  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 2011, tzlocal())
-     1.47±0.08μs      1.32±0.02μs     0.90  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(0, 3000, None)
-     1.53±0.07μs      1.37±0.02μs     0.89  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 2000, tzlocal())
-     1.50±0.08μs      1.33±0.02μs     0.88  tslibs.period.TimeDT64ArrToPeriodArr.time_dt64arr_to_periodarr(1, 1000, None)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

[jbrockmendel]

There aren't any asvs that specifically hit this. If you wanted to add one, it would go in benchmarks.tslibs.timestamp. Ideally in TimestampOps, but i dont know if it the ts defined in setup there makes a difference.

(an asv for this is likely overkill)

[jreback]

can you add the test here: #40817 as well. ping on green for merge

[jonwiggins]

@jreback Sounds good, I've added the reindex unit test and mentioned it in whatsnew

[jonwiggins]

@jreback Any more thoughts?

[jreback]

thanks @jonwiggins very nice!