datetime64[ns] astype datetime64[s]

[tboquet]

Hello all,
I'm trying to convert a datetime64[ns] object to a unix timestamp without an apply.
I was wondering if it was the desired behavior to throw an error when trying to do:

import pandas as pd
import datetime as dt
import numpy as np

my_time = np.datetime64(dt.datetime.utcnow())
my_df = pd.DataFrame({"time": [my_time]*3})
my_df.time = my_df.time.astype("datetime64[s]")

The error:

TypeError                                 Traceback (most recent call last)
<ipython-input-150-0b63b4529623> in <module>()
----> 1 my_df.time = my_df.time.astype("datetime64[s]")

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/generic.py in astype(self, dtype, copy, raise_on_error, **kwargs)
   2409 
   2410         mgr = self._data.astype(
-> 2411             dtype=dtype, copy=copy, raise_on_error=raise_on_error, **kwargs)
   2412         return self._constructor(mgr).__finalize__(self)
   2413 

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in astype(self, dtype, **kwargs)
   2502 
   2503     def astype(self, dtype, **kwargs):
-> 2504         return self.apply('astype', dtype=dtype, **kwargs)
   2505 
   2506     def convert(self, **kwargs):

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in apply(self, f, axes, filter, do_integrity_check, **kwargs)
   2457                                                  copy=align_copy)
   2458 
-> 2459             applied = getattr(b, f)(**kwargs)
   2460 
   2461             if isinstance(applied, list):

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in astype(self, dtype, copy, raise_on_error, values, **kwargs)
    371     def astype(self, dtype, copy=False, raise_on_error=True, values=None, **kwargs):
    372         return self._astype(dtype, copy=copy, raise_on_error=raise_on_error,
--> 373                             values=values, **kwargs)
    374 
    375     def _astype(self, dtype, copy=False, raise_on_error=True, values=None,

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/internals.py in _astype(self, dtype, copy, raise_on_error, values, klass, **kwargs)
    401             if values is None:
    402                 # _astype_nansafe works fine with 1-d only
--> 403                 values = com._astype_nansafe(self.values.ravel(), dtype, copy=True)
    404                 values = values.reshape(self.values.shape)
    405             newb = make_block(values,

/home/thomas/anaconda3/lib/python3.4/site-packages/pandas/core/common.py in _astype_nansafe(arr, dtype, copy)
   2699         elif dtype != _NS_DTYPE:
   2700             raise TypeError("cannot astype a datetimelike from [%s] to [%s]" %
-> 2701                             (arr.dtype, dtype))
   2702         return arr.astype(_NS_DTYPE)
   2703     elif is_timedelta64_dtype(arr):

TypeError: cannot astype a datetimelike from [datetime64[ns]] to [datetime64[s]]

But if I only take the values and work on this numpy array:

print(my_df.time.values.astype("datetime64[s]").dtype)
datetime64[s]
my_df.time = my_df.time.values.astype("datetime64[s]")
print(my_df.time)
0   2015-09-07 18:56:44
1   2015-09-07 18:56:44
2   2015-09-07 18:56:44
Name: time, dtype: datetime64[ns]

The type is correctly changed and pandas reconvert it to a datetime64[ns] object.
Could we control this behavior?
So I can convert my datetime64[ns] object into a unix timestamp like this:

my_df.time = my_df.time.values.astype("datetime64[s]").astype(int)

Second question why not have the same behavior for pandas.Series?

[shoyer]

The problem is that we don't support the datetime64[s] type on pandas objects, even though it's a valid type for NumPy arrays. So I think the current behavior (and your work around) is about the best we can do.

[jreback]

this is a dupe of #6741

your work-around is fine. You can also have a look at to_json which does this type of conversion (when going to JSON).

[tboquet]

Sorry for the dupe and thank you for the confirmation!