Merge pull request #9257 from jreback/td

API: restore full datetime.timedelta compat with Timedelta w.r.t. seconds/microseconds accessors (GH9185, GH9139)

Author: jreback

doc/source/timedeltas.rst

@@ -251,8 +251,13 @@ yields another ``timedelta64[ns]`` dtypes Series.
 Attributes
 ----------
 
-You can access various components of the ``Timedelta`` or ``TimedeltaIndex`` directly using the attributes ``days,hours,minutes,seconds,milliseconds,microseconds,nanoseconds``.
-These operations can be directly accessed via the ``.dt`` property of the ``Series`` as well. These return an integer representing that interval (which is signed according to whether the ``Timedelta`` is signed).
+You can access various components of the ``Timedelta`` or ``TimedeltaIndex`` directly using the attributes ``days,seconds,microseconds,nanoseconds``. These are identical to the values returned by ``datetime.timedelta``, in that, for example, the ``.seconds`` attribute represents the number of seconds >= 0 and < 1 day. These are signed according to whether the ``Timedelta`` is signed.
+
+These operations can also be directly accessed via the ``.dt`` property of the ``Series`` as well.
+
+.. note::
+
+   Note that the attributes are NOT the displayed values of the ``Timedelta``. Use ``.components`` to retrieve the displayed values.
 
 For a ``Series``
 
@@ -271,29 +276,12 @@ You can access the component field for a scalar ``Timedelta`` directly.
    (-tds).seconds
 
 You can use the ``.components`` property to access a reduced form of the timedelta. This returns a ``DataFrame`` indexed
-similarly to the ``Series``
+similarly to the ``Series``. These are the *displayed* values of the ``Timedelta``.
 
 .. ipython:: python
 
    td.dt.components
-
-.. _timedeltas.attribues_warn:
-
-.. warning::
-
-   ``Timedelta`` scalars (and ``TimedeltaIndex``) component fields are *not the same* as the component fields on a ``datetime.timedelta`` object. For example, ``.seconds`` on a ``datetime.timedelta`` object returns the total number of seconds combined between ``hours``, ``minutes`` and ``seconds``. In contrast, the pandas ``Timedelta`` breaks out hours, minutes, microseconds and nanoseconds separately.
-
-   .. ipython:: python
-
-      # Timedelta accessor
-      tds = Timedelta('31 days 5 min 3 sec')
-      tds.minutes
-      tds.seconds
-
-      # datetime.timedelta accessor
-      # this is 5 minutes * 60 + 3 seconds
-      tds.to_pytimedelta().seconds
-
+   td.dt.components.seconds
 
 .. _timedeltas.index:
 

doc/source/whatsnew/v0.15.0.txt

@@ -29,7 +29,7 @@ users upgrade to this version.
   - Split out string methods documentation into :ref:`Working with Text Data <text>`
 
 - Check the :ref:`API Changes <whatsnew_0150.api>` and :ref:`deprecations <whatsnew_0150.deprecations>` before updating
-  
+
 - :ref:`Other Enhancements <whatsnew_0150.enhancements>`
 
 - :ref:`Performance Improvements <whatsnew_0150.performance>`
@@ -403,7 +403,7 @@ Rolling/Expanding Moments improvements
 
     rolling_window(s, window=3, win_type='triang', center=True)
 
-- Removed ``center`` argument from all :func:`expanding_ <expanding_apply>` functions (see :ref:`list <api.functions_expanding>`), 
+- Removed ``center`` argument from all :func:`expanding_ <expanding_apply>` functions (see :ref:`list <api.functions_expanding>`),
   as the results produced when ``center=True`` did not make much sense. (:issue:`7925`)
 
 - Added optional ``ddof`` argument to :func:`expanding_cov` and :func:`rolling_cov`.
@@ -574,20 +574,20 @@ for more details):
   .. code-block:: python
 
     In [2]: pd.Categorical.from_codes([0,1,0,2,1], categories=['a', 'b', 'c'])
-    Out[2]: 
+    Out[2]:
     [a, b, a, c, b]
     Categories (3, object): [a, b, c]
 
 API changes related to the introduction of the ``Timedelta`` scalar (see
 :ref:`above <whatsnew_0150.timedeltaindex>` for more details):
-  
+
 - Prior to 0.15.0 :func:`to_timedelta` would return a ``Series`` for list-like/Series input,
   and a ``np.timedelta64`` for scalar input. It will now return a ``TimedeltaIndex`` for
   list-like input, ``Series`` for Series input, and ``Timedelta`` for scalar input.
 
 For API changes related to the rolling and expanding functions, see detailed overview :ref:`above <whatsnew_0150.roll>`.
 
-Other notable API changes:  
+Other notable API changes:
 
 - Consistency when indexing with ``.loc`` and a list-like indexer when no values are found.
 
@@ -872,7 +872,7 @@ Enhancements in the importing/exporting of Stata files:
   objects and columns containing missing values have ``object`` data type. (:issue:`8045`)
 
 Enhancements in the plotting functions:
-  
+
 - Added ``layout`` keyword to ``DataFrame.plot``. You can pass a tuple of ``(rows, columns)``, one of which can be ``-1`` to automatically infer (:issue:`6667`, :issue:`8071`).
 - Allow to pass multiple axes to ``DataFrame.plot``, ``hist`` and ``boxplot`` (:issue:`5353`, :issue:`6970`, :issue:`7069`)
 - Added support for ``c``, ``colormap`` and ``colorbar`` arguments for ``DataFrame.plot`` with ``kind='scatter'`` (:issue:`7780`)

doc/source/whatsnew/v0.16.0.txt

@@ -27,6 +27,41 @@ Backwards incompatible API changes
 
 .. _whatsnew_0160.api_breaking:
 
+- In v0.15.0 a new scalar type ``Timedelta`` was introduced, that is a sub-class of ``datetime.timedelta``. Mentioned :ref:`here <whatsnew_0150.timedeltaindex>` was a notice of an API change w.r.t. the ``.seconds`` accessor. The intent was to provide a user-friendly set of accessors that give the 'natural' value for that unit, e.g. if you had a ``Timedelta('1 day, 10:11:12')``, then ``.seconds`` would return 12. However, this is at odds with the definition of ``datetime.timedelta``, which defines ``.seconds`` as ``10 * 3600 + 11 * 60 + 12 == 36672``.
+
+So in v0.16.0, we are restoring the API to match that of ``datetime.timedelta``. However, the component values are still available through the ``.components`` accessor. This affects the ``.seconds`` and ``.microseconds`` accessors, and removes the ``.hours``, ``.minutes``, ``.milliseconds`` accessors. These changes affect ``TimedeltaIndex`` and the Series ``.dt`` accessor as well. (:issue:`9185`, :issue:`9139`)
+
+Previous Behavior
+
+.. code-block:: python
+
+   In [2]: t = pd.Timedelta('1 day, 10:11:12.100123')
+
+   In [3]: t.days
+   Out[3]: 1
+
+   In [4]: t.seconds
+   Out[4]: 12
+
+   In [5]: t.microseconds
+   Out[5]: 123
+
+New Behavior
+
+.. ipython:: python
+
+   t = pd.Timedelta('1 day, 10:11:12.100123')
+   t.days
+   t.seconds
+   t.microseconds
+
+Using ``.components`` allows the full component access
+
+.. ipython:: python
+
+   t.components
+   t.components.seconds
+
 - ``Index.duplicated`` now returns `np.array(dtype=bool)` rather than `Index(dtype=object)` containing `bool` values. (:issue:`8875`)
 - ``DataFrame.to_json`` now returns accurate type serialisation for each column for frames of mixed dtype (:issue:`9037`)
 

pandas/io/tests/test_excel.py

@@ -1151,6 +1151,29 @@ def test_swapped_columns(self):
             tm.assert_series_equal(write_frame['A'], read_frame['A'])
             tm.assert_series_equal(write_frame['B'], read_frame['B'])
 
+    def test_datetimes(self):
+
+        # Test writing and reading datetimes. For issue #9139. (xref #9185)
+        _skip_if_no_xlrd()
+
+        datetimes = [datetime(2013, 1, 13, 1, 2, 3),
+                     datetime(2013, 1, 13, 2, 45, 56),
+                     datetime(2013, 1, 13, 4, 29, 49),
+                     datetime(2013, 1, 13, 6, 13, 42),
+                     datetime(2013, 1, 13, 7, 57, 35),
+                     datetime(2013, 1, 13, 9, 41, 28),
+                     datetime(2013, 1, 13, 11, 25, 21),
+                     datetime(2013, 1, 13, 13, 9, 14),
+                     datetime(2013, 1, 13, 14, 53, 7),
+                     datetime(2013, 1, 13, 16, 37, 0),
+                     datetime(2013, 1, 13, 18, 20, 52)]
+
+        with ensure_clean(self.ext) as path:
+            write_frame = DataFrame.from_items([('A', datetimes)])
+            write_frame.to_excel(path, 'Sheet1')
+            read_frame = read_excel(path, 'Sheet1', header=0)
+
+            tm.assert_series_equal(write_frame['A'], read_frame['A'])
 
 def raise_wrapper(major_ver):
     def versioned_raise_wrapper(orig_method):

pandas/tests/test_series.py

@@ -84,7 +84,7 @@ def test_dt_namespace_accessor(self):
         ok_for_dt = ok_for_base + ['date','time','microsecond','nanosecond', 'is_month_start', 'is_month_end', 'is_quarter_start',
                                    'is_quarter_end', 'is_year_start', 'is_year_end', 'tz']
         ok_for_dt_methods = ['to_period','to_pydatetime','tz_localize','tz_convert']
-        ok_for_td = ['days','hours','minutes','seconds','milliseconds','microseconds','nanoseconds']
+        ok_for_td = ['days','seconds','microseconds','nanoseconds']
         ok_for_td_methods = ['components','to_pytimedelta']
 
         def get_expected(s, name):

pandas/tseries/tdi.py

@@ -118,8 +118,8 @@ def _join_i8_wrapper(joinf, **kwargs):
     _left_indexer_unique = _join_i8_wrapper(
         _algos.left_join_indexer_unique_int64, with_indexers=False)
     _arrmap = None
-    _datetimelike_ops = ['days','hours','minutes','seconds','milliseconds','microseconds',
-                         'nanoseconds','freq','components']
+    _datetimelike_ops = ['days','seconds','microseconds','nanoseconds',
+                         'freq','components']
 
     __eq__ = _td_index_cmp('__eq__')
     __ne__ = _td_index_cmp('__ne__', nat_result=True)
@@ -349,37 +349,22 @@ def _get_field(self, m):
 
     @property
     def days(self):
-        """ The number of integer days for each element """
+        """ Number of days for each element. """
         return self._get_field('days')
 
-    @property
-    def hours(self):
-        """ The number of integer hours for each element """
-        return self._get_field('hours')
-
-    @property
-    def minutes(self):
-        """ The number of integer minutes for each element """
-        return self._get_field('minutes')
-
     @property
     def seconds(self):
-        """ The number of integer seconds for each element """
+        """ Number of seconds (>= 0 and less than 1 day) for each element. """
         return self._get_field('seconds')
 
-    @property
-    def milliseconds(self):
-        """ The number of integer milliseconds for each element """
-        return self._get_field('milliseconds')
-
     @property
     def microseconds(self):
-        """ The number of integer microseconds for each element """
+        """ Number of microseconds (>= 0 and less than 1 second) for each element. """
         return self._get_field('microseconds')
 
     @property
     def nanoseconds(self):
-        """ The number of integer nanoseconds for each element """
+        """ Number of nanoseconds (>= 0 and less than 1 microsecond) for each element. """
         return self._get_field('nanoseconds')
 
     @property

pandas/tseries/tests/test_timedeltas.py

@@ -301,30 +301,33 @@ class Other:
         self.assertTrue(td.__floordiv__(td) is NotImplemented)
 
     def test_fields(self):
+
+        # compat to datetime.timedelta
         rng = to_timedelta('1 days, 10:11:12')
         self.assertEqual(rng.days,1)
-        self.assertEqual(rng.hours,10)
-        self.assertEqual(rng.minutes,11)
-        self.assertEqual(rng.seconds,12)
-        self.assertEqual(rng.milliseconds,0)
+        self.assertEqual(rng.seconds,10*3600+11*60+12)
         self.assertEqual(rng.microseconds,0)
         self.assertEqual(rng.nanoseconds,0)
 
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
+
         td = Timedelta('-1 days, 10:11:12')
         self.assertEqual(abs(td),Timedelta('13:48:48'))
         self.assertTrue(str(td) == "-1 days +10:11:12")
         self.assertEqual(-td,Timedelta('0 days 13:48:48'))
         self.assertEqual(-Timedelta('-1 days, 10:11:12').value,49728000000000)
         self.assertEqual(Timedelta('-1 days, 10:11:12').value,-49728000000000)
 
-        rng = to_timedelta('-1 days, 10:11:12')
+        rng = to_timedelta('-1 days, 10:11:12.100123456')
         self.assertEqual(rng.days,-1)
-        self.assertEqual(rng.hours,10)
-        self.assertEqual(rng.minutes,11)
-        self.assertEqual(rng.seconds,12)
-        self.assertEqual(rng.milliseconds,0)
-        self.assertEqual(rng.microseconds,0)
-        self.assertEqual(rng.nanoseconds,0)
+        self.assertEqual(rng.seconds,10*3600+11*60+12)
+        self.assertEqual(rng.microseconds,100*1000+123)
+        self.assertEqual(rng.nanoseconds,456)
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
 
         # components
         tup = pd.to_timedelta(-1, 'us').components
@@ -830,22 +833,22 @@ def test_astype(self):
         self.assert_numpy_array_equal(result, rng.asi8)
 
     def test_fields(self):
-        rng = timedelta_range('1 days, 10:11:12', periods=2, freq='s')
+        rng = timedelta_range('1 days, 10:11:12.100123456', periods=2, freq='s')
         self.assert_numpy_array_equal(rng.days, np.array([1,1],dtype='int64'))
-        self.assert_numpy_array_equal(rng.hours, np.array([10,10],dtype='int64'))
-        self.assert_numpy_array_equal(rng.minutes, np.array([11,11],dtype='int64'))
-        self.assert_numpy_array_equal(rng.seconds, np.array([12,13],dtype='int64'))
-        self.assert_numpy_array_equal(rng.milliseconds, np.array([0,0],dtype='int64'))
-        self.assert_numpy_array_equal(rng.microseconds, np.array([0,0],dtype='int64'))
-        self.assert_numpy_array_equal(rng.nanoseconds, np.array([0,0],dtype='int64'))
+        self.assert_numpy_array_equal(rng.seconds, np.array([10*3600+11*60+12,10*3600+11*60+13],dtype='int64'))
+        self.assert_numpy_array_equal(rng.microseconds, np.array([100*1000+123,100*1000+123],dtype='int64'))
+        self.assert_numpy_array_equal(rng.nanoseconds, np.array([456,456],dtype='int64'))
+
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
 
         # with nat
         s = Series(rng)
         s[1] = np.nan
 
         tm.assert_series_equal(s.dt.days,Series([1,np.nan],index=[0,1]))
-        tm.assert_series_equal(s.dt.hours,Series([10,np.nan],index=[0,1]))
-        tm.assert_series_equal(s.dt.milliseconds,Series([0,np.nan],index=[0,1]))
+        tm.assert_series_equal(s.dt.seconds,Series([10*3600+11*60+12,np.nan],index=[0,1]))
 
     def test_components(self):
         rng = timedelta_range('1 days, 10:11:12', periods=2, freq='s')

pandas/tslib.pyx

@@ -1896,45 +1896,43 @@ class Timedelta(_Timedelta):
 
     @property
     def days(self):
-        """ The days for the Timedelta """
+        """
+        Number of Days
+
+        .components will return the shown components
+        """
         self._ensure_components()
         if self._sign < 0:
             return -1*self._d
         return self._d
 
-    @property
-    def hours(self):
-        """ The hours for the Timedelta """
-        self._ensure_components()
-        return self._h
-
-    @property
-    def minutes(self):
-        """ The minutes for the Timedelta """
-        self._ensure_components()
-        return self._m
-
     @property
     def seconds(self):
-        """ The seconds for the Timedelta """
-        self._ensure_components()
-        return self._s
+        """
+        Number of seconds (>= 0 and less than 1 day).
 
-    @property
-    def milliseconds(self):
-        """ The milliseconds for the Timedelta """
+        .components will return the shown components
+        """
         self._ensure_components()
-        return self._ms
+        return self._h*3600 + self._m*60 + self._s
 
     @property
     def microseconds(self):
-        """ The microseconds for the Timedelta """
+        """
+        Number of microseconds (>= 0 and less than 1 second).
+
+        .components will return the shown components
+        """
         self._ensure_components()
-        return self._us
+        return self._ms*1000 + self._us
 
     @property
     def nanoseconds(self):
-        """ The nanoseconds for the Timedelta """
+        """
+        Number of nanoseconds (>= 0 and less than 1 microsecond).
+
+        .components will return the shown components
+        """
         self._ensure_components()
         return self._ns