cc @immerrr
@jorisvandenbossche
@shoyer

I think this should resolve things and make .loc strict, while matching .ix (notably in that .ix tries for positional access when it can, but .loc will refuse to guess).

First of all, big thanks for working on this, @jreback!

However, I don't think we need to enforce distinctions between float and int for slice bounds, because there is no ambiguity in the ordering between integers and floats (whereas the ordering with respect to strings, for example, is indeed problematic). Users shouldn't have to check the type of their index (if they know it's numeric) before doing indexing.

In particular:

1. I think it should be OK to index an Int64Index with floats, like .loc[0.5:2.5] or .loc[-1.0:2].
2. I think it should be OK to index a Float64Index with integers, like .loc[0:10]. (Of course, these are still to be interpreted as labels, not integer positions)

The binary search logic for determining slice bound locations should already work for both of these cases.

To give a more concrete example from my experience: an index corresponding to a variable like longitude might variously consist of integer (e.g., if it's in multiples of whole degrees) or as floating point numbers (e.g., if it's in multiples of half degrees). It's a bad user experience to need to cast their slice bounds to the type of index. Code like s.loc[min_lon:max_lon] should work regardless of dtypes.

@shoyer

I added in these checks, mainly because numpy is deprecating float indexers (IIRC it has a warning now), and we have been warning on it.

Its easy to take out, but I think that indexing with a float on an integer index is really not in the .loc domain. This is moving back to the .ix functionaility (which actually doesn't work in 0.15.2 or atm), though .loc did (which is odd).

though it IS convenient.

currently we don't allow this for [],.ix (but allow for .loc in 0.15.2.

simple example from 0.15.2

In [1]: s = Series(range(5))

In [2]: s[1.5:3]
TypeError: the slice start value [1.5] is not a proper indexer for this index type (Int64Index)

In [3]: s.ix[1.5:3]
TypeError: the slice start value [1.5] is not a proper indexer for this index type (Int64Index)

In [4]: s.loc[1.5:3]
Out[4]: 
2    2
3    3
dtype: int64

These all fail with
TypeError: cannot do slice start value indexing on <class 'pandas.core.index.Int64Index'> with these indexers [1.5] of <type 'float'> in master/0.16.0

@jorisvandenbossche what do you think?

I agree that using float indexers is a based idea for position based indexing (.iloc). Position based indexing should always use integers. We should be consistent with numpy for this.

But indexing with .loc uses labels. Here (e.g., with searchsorted), numpy is more flexible -- the important thing is that there is a well defined sort order.

this should then also work for .ix/[]?

To elaborate:

There is never a good reason to use floats for position based indexing. Numpy warns and/or raises because using a float is an indication that your code probably has a bug.

In contrast, there certainly are good reasons to use floats for label based indexing, because the labels might include floats themselves (or might not, depending on dtypes).

.ix and [] are an unpredictable mess! We hadn't even written down the rules for exactly how indexing with [] works last time I checked.

If I had my way, we would make some hard choices and deprecate/remove fall-back indexing all together (#9213). In the mean-time, I suspect the best approach is to simply try to break/change as little existing code as possible.

ok, here is the change (plus making some testing more robust)

In [3]:  index = tm.makeIntIndex()
In [4]:  s = Series(np.arange(len(index))+10,index+5)
In [5]: s
Out[5]: 
5     10
6     11
7     12
8     13
9     14
10    15
11    16
12    17
13    18
14    19
dtype: int64

In [6]: s.loc[6.0:8.5]
Out[6]: 
6    11
7    12
8    13
dtype: int64

In [7]: s.loc[5.5:8.5]
Out[7]: 
6    11
7    12
8    13
dtype: int64

In [8]: s.loc[5.5:8.0]
Out[8]: 
6    11
7    12
8    13
dtype: int64

I could enable for []/ix, but then would s[5.0:8.0] become a fallback scenario (yes I think)
so what would s[5.5,8.5] do (non-fallback I think), so they become very sensitive to the actual values, hence bad

I agree with the logic of allowing float indexers for integer indexes when using loc (so as you now implemented)

Ideally, I think it should also work for ix (to keep ix as much as possible to the logic of try: s.loc[]; except: s.iloc[]. In principle , we say that for an integer index axis, we do never integer location fallback (http://pandas.pydata.org/pandas-docs/stable/indexing.html#different-choices-for-indexing), so following the loc behaviour should not be ambiguous in this case I think.

For [], it is something else. For that, we should first have a better documented overview of it behaviour (I started with that last week, will try to put it up as a PR)

ok, I added the .ix support for float indexers on an Int index. It is trivial to make this work for []. I am thinking that this should have the same semantics as .ix (e.g. label based with an int index). It works the same.

But for [] slicing is 'integer-location' based (an exception on the other types of slicing), so this should follow the semantics of iloc in this case, disallowing floats.

Example:

In [46]: s = Series(range(4), index=[-1,0,1,2])

In [47]: s
Out[47]:
-1    0
 0    1
 1    2
 2    3
dtype: int64

In [48]: s[0:2]    # <--- slicing = integer location based
Out[48]:
-1    0
 0    1
dtype: int64

In [49]: s[0]    # <--- single label = label based
Out[49]: 1

ok, is updated, any more comments
@jorisvandenbossche @shoyer

Generally seems fine by me.

indexing is such a rabbit hole :)

I cleaned up _convert_slice_indexer a bit. I tried to use _maybe_cast_slice_bound directly there (I added the kind argument which is the type of indexing, e.g. ix/loc/iloc/getitem). But so many cases are handled that its really hard to get this right. Because the different kinds are handled in different places

its tricky because you have the kinds of indexers (ix/loc/iloc/getitem) x index_types (Index,Float,Int,Datetime....) x kinds of indexing (slice/scalar)

So its better (meaning slightly cleaner), with the semantics we discussed above.

Thanks a lot for this!

Indeed, greatly appreciated!

That's great that you guys (and especially @jreback) actually went through this 👍

trying to make indexing slightly less of a rabbit hole :)