ENH: Compile Factorizer class for all numeric dtypes

[phofl]

The current implementation has a couple of ugly side effects in merge:

• uint is cast to int, which causes overflow in some cases
• float is cast to object

This does not enable the classes yet. Will do as a follow up.

[phofl]

cc @jbrockmendel would you mind having a look?

[phofl]

Adding factorize to the base class simplifies the follow up

[jbrockmendel]

simplifies the follow up

i guess the follow-up will change how klass is found in merge.py?

[jbrockmendel]

is there any particular reason for this to be in the pxi.in instead of the pyx?

[phofl]

Had some failures initially, but looks like this was caused by something else. Moved it back

[jbrockmendel]

a few questions, generally looks good

[phofl]

simplifies the follow up

i guess the follow-up will change how klass is found in merge.py?

Yeah and also avoids having a million typing errors

[mroeschke]

LGTM merge when ready @jbrockmendel

[jbrockmendel]

thanks @phofl

[rhshadrach]

This patch may have induced a potential regression. Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected appear as subbullets. This is a partially automated message.

• https://asv-runner.github.io/asv-collection/pandas/#algos.isin.IsInForObjects.time_isin
  • series_type='long'; vals_type='nans'
• https://asv-runner.github.io/asv-collection/pandas/#algos.isin.IsInLongSeriesLookUpDominates.time_isin
  • dtype='Float64'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='float64'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='object'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='object'; MaxNumber=1000; series_type='random_misses'
  • dtype='object'; MaxNumber=5; series_type='monotone_hits'
• https://asv-runner.github.io/asv-collection/pandas/#algos.isin.IsinWithArange.time_isin
  • dtype=<class 'numpy.object_'>; M=1000; offset_factor=-2
  • dtype=<class 'numpy.object_'>; M=2000; offset_factor=-2
  • dtype=<class 'numpy.object_'>; M=8000; offset_factor=-2
  • dtype=<class 'numpy.uint64'>; M=1000; offset_factor=-2
  • dtype=<class 'numpy.uint64'>; M=1000; offset_factor=0
  • dtype=<class 'numpy.uint64'>; M=2000; offset_factor=0
  • dtype=<class 'numpy.uint64'>; M=2000; offset_factor=2
  • dtype=<class 'numpy.uint64'>; M=8000; offset_factor=0

[phofl]

I think this is a false positive, looks to me as this is impacted by the klib quadratic probing and seems to go back to normal when this pr was reverted

[rhshadrach]

This patch may have induced a potential regression. Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected appear as subbullets. This is a partially automated message.

Subsequent benchmarks may have skipped some commits. See the link below to see which commits are between the two benchmark runs where the regression was identified.

c04bba4...f6d3cb2

• algos.isin.IsInForObjects.time_isin
  • series_type='long'; vals_type='nans'
• algos.isin.IsInLongSeriesLookUpDominates.time_isin
  • dtype='Float64'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='float64'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='object'; MaxNumber=1000; series_type='monotone_hits'
  • dtype='object'; MaxNumber=1000; series_type='random_misses'
  • dtype='object'; MaxNumber=5; series_type='monotone_hits'
• algos.isin.IsinWithArange.time_isin
  • dtype=<class 'numpy.object_'>; M=1000; offset_factor=-2
  • dtype=<class 'numpy.object_'>; M=2000; offset_factor=-2
  • dtype=<class 'numpy.object_'>; M=8000; offset_factor=-2
  • dtype=<class 'numpy.uint64'>; M=1000; offset_factor=-2
  • dtype=<class 'numpy.uint64'>; M=1000; offset_factor=0
  • dtype=<class 'numpy.uint64'>; M=2000; offset_factor=0
  • dtype=<class 'numpy.uint64'>; M=2000; offset_factor=2
  • dtype=<class 'numpy.uint64'>; M=8000; offset_factor=0

[phofl]

The links don't work. But this pr did not touch the code paths that the isin case takes

[rhshadrach]

Reworked my scripts and updated the links, the subbullets now contain links to single parameterization of the benchmark where my code is identifying a regression. But certainly could be a false positive. For example, the first subbullet of algos.isin.IsInLongSeriesLookUpDominates.time_isin is an interesting case:

The first step up points to this PR whereas the second step up is the revert of quadratic programming. Both look like "regressions" but the revert of quadratic programming puts us back to where we started. I'm going to run asvs between certain commits; I'm just curious what's going on here.