pd.DataFrame converts np.uint64 greater than 2**63-1 to objects

[ily9000]

My question is based on this post I found:
http://stackoverflow.com/questions/34283319/pandas-converts-large-unsigned-integers-to-object-types

The maximum value for np.uint64 is (2^64)-1 so why does pandas convert np.uint64 types greater than 2^63-1 to objects when converting arrays to dataframes. pd.to_numeric() also fails to convert these values back to np.uint64.

[jreback]

this is a dupe of this: #4471

essentially uint64 support in pandas is not well tested / buggy.

pull-request are for sure welcome. Not really sure what actual use-cases in the wild are for this type of integer.as IMHO it doesn't buy you much.

[aoz]

@jreback FYI: I've just hit this issue while working on a real life, commercial project. My largish (few GB) datasets consist of uint64 elements, runs failed on some of them, and I tracked down the issue to this. Trying to find a workaround...

[pwaller]

@jreback Re: you can't think of a use case — unfortunately I'm importing a dataset which already has uint64's in it. Not being able to use them is quite a big inconvenience, as I keep hitting this, even if I work around it!

I think this is really surprising behaviour, but I guess that's because DataFrame's storage is somehow different from that of a Series in a way I don't understand...

In [1]: import pandas as pd, numpy as np

In [2]: s = pd.Series(np.array([2**64 - 1]))

In [3]: s.dtype
Out[3]: dtype('uint64')

In [4]: df = pd.DataFrame({"s": s})

In [5]: df.dtypes
Out[5]: 
s    object
dtype: object