PERF: Float64Index to use Float hashtable backend

[jreback]

https://groups.google.com/forum/m/#!topic/pydata/zUxl7rOHVNY

Currently Float64Index uses the Object back-end for ease of implementation.
See Here: https://github.com/pydata/pandas/blob/master/pandas/core/index.py#L1881

• change the backend to use Float64Engine - the cythob code needs fixin in order to make it work.
• needs vbench for perf indications
• needs pickle validation that it is backward compat (it should be w/o changes)

[cpcloud]

@jreback can I take this or do u already have something worked up?

[jreback]

go for it!

its very close....I just didn't have time to debug it....most of the code was already there from @wesm originally

[cpcloud]

Cool Ill take a look

[jreback]

@cpcloud any progress on this?

[cpcloud]

Yep ... should be able to put up a WIP PR this weekend

[jreback]

gr8 thanks

[cpcloud]

nan isn't really a singleton, probably because of its internal representation as some large integer. Small integers are singletons in Python, whereas larger ones are new instances of int every time and so their id changes. The first set membership test checks the object nan whose reference I put in to the array whereas flt has a new instance of nan. What's weird is that hash(nan) == 0 and yet the membership check fails. I thought the order of lookup for sets was hash -> eq (if no __eq__ method then compare id), but since nan is an instance of float it obviously has that method. @jreback Am I missing something here?

[jreback]

nope that's why you can lookup object dtyped floats/nan pretty easiky
but actual Nan's that are floats need special logic
in essence you have to segregate Nan's and keep a reference to their locations and not hash them
so keep the float hash table for non-nan then getitem checks for nan before you test the hash

[jreback]

@cpcloud coming along?

[cpcloud]

Yep coming along. Couple of errors left to go. Mostly just renaming and keeping those refs to nan locs around.

[jreback]

gr8