Skip to content

Series[dt64].astype(int64) vs Series[Sparse[dt64]].astype(int64) #49631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jbrockmendel opened this issue Nov 10, 2022 · 0 comments
Open

Series[dt64].astype(int64) vs Series[Sparse[dt64]].astype(int64) #49631

jbrockmendel opened this issue Nov 10, 2022 · 0 comments

Comments

@jbrockmendel
Copy link
Member

jbrockmendel commented Nov 10, 2022

dti = pd.date_range("2016-01-01", periods=3)
ser = pd.Series(dti)
ser[0] = pd.NaT

dense = ser._values
sparse = pd.core.arrays.SparseArray(ser.values)

>>> dense.astype("int64")
array([-9223372036854775808,  1451692800000000000,  1451779200000000000])

>>> sparse.astype("int64")
[...]
ValueError: Cannot convert NaT values to integer

>>> sparse.astype("Sparse[int64]")
[0, 1451692800000000000, 1451779200000000000]
Fill: 0
IntIndex
Indices: array([1, 2], dtype=int32)

The dense version goes through DatetimeArray.astype, for which .astype to int64 is basically a view (xref #45034). The Sparse version goes through astype_nansafe which specifically checks for NaTs when going from dt64->int64. I expected this to match the non-sparse behavior.

When converting to Sparse[int64], we only call astype_nansafe on the non-NaT elements so it doesn't raise, but when converting the fill_value from NaT it somehow gets 0, whereas I expected that to raise.

Side-notes:
ser.astype(pd.SparseDtype(ser.dtype)) raises, as does dense.astype("Sparse[int64]")

@jbrockmendel jbrockmendel added Bug Needs Triage Issue that has not been reviewed by a pandas team member Astype and removed Needs Triage Issue that has not been reviewed by a pandas team member labels Nov 10, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant