Skip to content

Int64Index with dtype=float and slicing issues #9966

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
josteinbf opened this issue Apr 22, 2015 · 3 comments · Fixed by #10638
Closed

Int64Index with dtype=float and slicing issues #9966

josteinbf opened this issue Apr 22, 2015 · 3 comments · Fixed by #10638
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions Indexing Related to indexing on series/frames, not to indexes themselves
Milestone

Comments

@josteinbf
Copy link

I have stumbled onto problems with slicing a pandas DataFrame when an index that originally contained integers has been manipulated such that it contains floats. Consider the following example (somewhat contrived to make it stand on its own; in practice, data are loaded from file):

df = pd.DataFrame({'i': np.linspace(0, 5, 6).astype(np.int64), 
               'j': np.linspace(2, 3, 6)})
df.set_index('i', inplace=True)
print(df.index)   # Int64Index([0, 1, 2, 3, 4, 5], dtype='int64')
df.index = df.index / 10.
print(df.index)   # Int64Index([0.0, 0.1, 0.2, 0.3, 0.4, 0.5], dtype='float64')
df[:0.3]

The final line raises this exception:

TypeError: the slice stop value [None] is not a proper indexer 
for this index type (Int64Index)

While it was relatively easy to work around this using an explicit cast along the lines of

df.index = np.asarray(df.index, dtype=np.float64) / 10.

it took me quite a while to figure out what was going on. What I expected to happen in the original code was that the index would become a Float64Index or something similar. When I first noticed the Int64Index with dtype=float I thought that was wierd. Is it a good idea to change this behaviour so that there is not this mismatch between index type and dtype?

My pandas version is 0.15.2. User EdChum at StackOverflow[1] has kindly tested this on 0.16, finding the same behaviour with the code above. Slicing with .ix works on 0.16 but not on 0.15.2; slicing with .loc works for both versions.

[1] http://stackoverflow.com/questions/29795225/pandas-int64index-with-dtype-float

@josteinbf josteinbf changed the title Int64Index with Int64Index with dtype=float and slicing issues Apr 22, 2015
@jreback
Copy link
Contributor

jreback commented Apr 22, 2015

This is a bug (but not with the slicing, rather the index construction).

This should be return Index(op(values, other), **self._get_attributes_dict()

(I think the unary ops are ok, e.g. they will always be the same type, but could change that as well).

These are not really tested all that well, so need a comprehensive test.

Interested in doing a PR?

@jreback jreback added Bug Indexing Related to indexing on series/frames, not to indexes themselves Dtype Conversions Unexpected or buggy dtype conversions labels Apr 22, 2015
@jreback jreback added this to the 0.16.1 milestone Apr 22, 2015
@josteinbf
Copy link
Author

@jreback I'm interested in doing a PR, but as I have never done any work on pandas before, I'm unsure if I will be able to. I hope it's ok for me to take a couple of days to look into it to see if I'm able to understand the code?

Also, to make sure I understand correctly: We should fix the issue that Int64Index can sometimes contain floats by making sure we create a Float64Index where appropriate. As a consequence of that fix the slicing issues I found should go away, right?

@jreback
Copy link
Contributor

jreback commented Apr 23, 2015

see contributing guidelines

The issue is that certain numeric ops return the wrong index type; all else follows from there.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Dtype Conversions Unexpected or buggy dtype conversions Indexing Related to indexing on series/frames, not to indexes themselves
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants