Allow for near misses when doing alignment with Float64Index?

[shoyer]

The "exact match" requirement for index alignment doesn't work very well when using Float64Index if the index values may result from computations.

Here's an example of how the current behavior results in a poor user experience:

In [22]: s1 = pd.Series(range(3), np.arange(3.0))

In [23]: s2 = pd.Series(range(3), s1.index + 0.15 - 0.15)

In [24]: s1 + s2
Out[24]:
0     0
1   NaN
1   NaN
2     4
dtype: float64

In [25]: s1.index
Out[25]: Float64Index([0.0, 1.0, 2.0], dtype='float64')

In [26]: s2.index
Out[26]: Float64Index([0.0, 1.0, 2.0], dtype='float64')

In [27]: s1.index[1]
Out[27]: 1.0

In [28]: s2.index[1]
Out[28]: 0.99999999999999989

reindex/joins with method='nearest' will help (#9258), but automatic alignment will still remain broken. Ideally, we should figure out some way for array alignment to ignore very small differences that are almost certainly due to the precision limits of floating point arithmetic.

I'm not entirely sure what the right solution here looks like, but I wanted to open this to discussion.

CC @hugadams who I'm guessing has almost certainly encountered these sort of issues when doing computations on indexes for scikit-spectra.

[jreback]

There is some code here, which is currently not used anywhere (and I don't recall when / how was used). To do something like this.

I agree that perhaps align should support method='nearest' (maybe with a rtol for float indexes, that defaults to machine precision).

[hughesadam87]

Thanks for CCing me Stephan. I absolutely have encountered. I still havent' gotten around to updating scikit-spectra to pandas > 0.14 (lots of other projects, grad school), so in 0.14, I can't even test some of this because it would result in an error. Therefore, I'm pleased to see that you at least get output in these operations.

s2 = pd.Series(range(3), s1.index + 0.15 - 0.15)

TypeError                                 Traceback (most recent call last)
<ipython-input-3-dbbad8e288fd> in <module>()
      4 s1 = pd.Series(range(3), np.arange(3.0))
      5 
----> 6 s2 = pd.Series(range(3), s1.index + 0.15 - 0.15)

/home/adam/Enthought/Canopy_32bit/User/lib/python2.7/site-packages/pandas/core/index.pyc in __sub__(self, other)
    959 
    960     def __sub__(self, other):
--> 961         return self.diff(other)
    962 
    963     def __and__(self, other):

/home/adam/Enthought/Canopy_32bit/User/lib/python2.7/site-packages/pandas/core/index.pyc in diff(self, other)
   1112 
   1113         if not hasattr(other, '__iter__'):
-> 1114             raise TypeError('Input must be iterable!')
   1115 
   1116         if self.equals(other):

TypeError: Input must be iterable!

Still using 0.14.1 on this machine. For the most part, I've just avoided dealing with this issue, but am glad to see it's not just me losing my marbles!

[shoyer]

@hugadams I think you can make this example work on older versions of pandas by changing s1.index to s1.index.values.

[mroeschke]

Seems like there hasn't been much movement on this feature request in years so closing