BUG: get_indexer_non_unique with np.datetime64("NaT") and np.timedelta64("NaT")

[alexhlim]

• closes BUG: get_indexer_non_unique does not handle np.datetime64("NaT") and np.timedelta64("NaT") #43869
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[alexhlim]

@jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)

[jbrockmendel]

@jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)

first guess is to check if numpy/numpy#12550 is biting you. i.e. when you construct Index([np_nat_fixture], dtype=object), some numpy versions might be silently casting that np_nat_fixture to an integer.

[alexhlim]

@jbrockmendel any idea why tests might be failing some machines but not others? right now, they're passing on my local machine. any tips on debugging would be appreciated :)

first guess is to check if numpy/numpy#12550 is biting you. i.e. when you construct Index([np_nat_fixture], dtype=object), some numpy versions might be silently casting that np_nat_fixture to an integer.

You were right -- it was because the older version of numpy was down-casting the NaT. I was able to get the CI passing except for Windows (timed-out) -- is this a consequence of performance?

[jbrockmendel]

I was able to get the CI passing except for Windows (timed-out) -- is this a consequence of performance?

windows timeouts are pretty common on the CI; almost certainly unrelated

[jreback]

can you run asv's which might hit this (conversions) and see if anything is changed

[alexhlim]

@jbrockmendel @jreback I was able to simplify all the na checks into one (float/complex, Decimal, np.datetime64(“NaT”) and np.timedelta64(“NaT”)) - and looks like performance increased as well!

asv continuous -f 1.1 upstream/master HEAD -b ^indexing


       before           after         ratio
     [c4cce9b7]       [da4300cc]
     <master>         <nui-dt64nat-td64nat>
-      97.9±0.4ms       85.7±0.7ms     0.88  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.Float64Index'>, 'nonunique_monotonic_inc')
-      99.9±0.5ms         87.4±2ms     0.87  indexing.NumericSeriesIndexing.time_getitem_array(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-     1.41±0.03μs      1.22±0.03μs     0.87  indexing_engines.NumericEngineIndexing.time_get_loc((<class 'pandas._libs.index.Float32Engine'>, <class 'numpy.float32'>), 'monotonic_incr', True, 2000000)
-         102±1ms         83.8±1ms     0.82  indexing.NumericSeriesIndexing.time_getitem_lists(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-        99.3±1ms       80.7±0.7ms     0.81  indexing.NumericSeriesIndexing.time_getitem_array(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-         101±1ms       82.0±0.7ms     0.81  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-       101±0.9ms       82.2±0.9ms     0.81  indexing.NumericSeriesIndexing.time_loc_array(<class 'pandas.core.indexes.numeric.UInt64Index'>, 'nonunique_monotonic_inc')
-         102±2ms         81.4±1ms     0.80  indexing.NumericSeriesIndexing.time_getitem_lists(<class 'pandas.core.indexes.numeric.Int64Index'>, 'nonunique_monotonic_inc')
-      29.8±0.5ms       22.6±0.8ms     0.76  indexing.DataFrameNumericIndexing.time_loc_dups

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

[jreback]

lgtm. @jbrockmendel good here?

[jbrockmendel]

LGTM really nice work here

[alexhlim]

fixed up some typos in the comments + adding test cases (failing on master, but this PR should resolve). I'll ping on green @jbrockmendel @jreback

[alexhlim]

ping @jreback @jbrockmendel

[jreback]

thanks @alexhlim really nice!

[alexhlim]

LGTM really nice work here

your suggestion of using sets + is_matching_na really helped me out!!

[alexhlim]

thanks for taking the time to review @jreback @jbrockmendel !