API: Change the sum of all-NA / all-Empty sum / prod

Author: TomAugspurger

doc/source/whatsnew/v0.22.0.txt

@@ -3,12 +3,185 @@
 v0.22.0
 -------
 
-This is a major release from 0.21.1 and includes a number of API changes,
-deprecations, new features, enhancements, and performance improvements along
-with a large number of bug fixes. We recommend that all users upgrade to this
-version.
+This is a major release from 0.21.1 and includes a single, API-breaking change.
+We recommend that all users upgrade to this version after carefully reading the
+release note (singular!).
 
 .. _whatsnew_0220.api_breaking:
 
 Backwards incompatible API changes
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pandas 0.22.0 changes the handling of empty and all-NA sums and products. The
+summary is that
+
+* The sum of an all-NA or empty series is now 0
+* The product of an all-NA or empty series is now 1
+* We've added a ``min_count`` parameter to ``.sum`` and ``.prod`` to control
+  the minimum number of valid values for the result to be valid. If fewer than
+  ``min_count`` valid values are present, the result is NA. The default is
+  ``0``. To return ``NaN``, the 0.21 behavior, use ``min_count=1``.
+
+Some background: In pandas 0.21, we fixed a long-standing inconsistency
+in the return value of all-NA series depending on whether or not bottleneck
+was installed. See :ref:`whatsnew_0210.api_breaking.bottleneck`_. At the same
+time, we changed the sum and prod of an empty Series to also be ``NaN``.
+
+Based on feedback, we've partially reverted those changes. The default sum for
+all-NA and empty series is now 0 (1 for ``prod``).
+
+*pandas 0.21*
+
+.. code-block:: ipython
+
+   In [3]: pd.Series([]).sum()
+   Out[3]: nan
+
+   In [4]: pd.Series([np.nan]).sum()
+   Out[4]: nan
+
+*pandas 0.22.0*
+
+.. ipython:: python
+
+   pd.Series([]).sum()
+   pd.Series([np.nan]).sum()
+
+The default behavior is the same as pandas 0.20.3 with bottleneck installed. It
+also matches the behavior of ``np.nansum`` and ``np.nanprod`` on empty and
+all-NA arrays.
+
+To have the sum of an empty series return ``NaN``, use the ``min_count``
+keyword. Thanks to the ``skipna`` parameter, the ``.sum`` on an all-NA
+series is conceptually the same as on an empty. The ``min_count`` parameter
+refers to the minimum number of *valid* values required for a non-NA sum
+or product.
+
+.. ipython:: python
+
+   pd.Series([]).sum(min_count=1)
+   pd.Series([np.nan]).sum(min_count=1)
+
+Returning ``NaN`` was the default behavior for pandas 0.20.3 without bottleneck
+installed.
+
+Note that this affects some other places in the library:
+
+1. Grouping by a Categorical with some unobserved categories and computing the
+``sum`` / ``prod``.
+
+*pandas 0.21*
+
+.. code-block:: ipython
+
+   In [5]: grouper = pd.Categorical(['a', 'a'], categories=['a', 'b'])
+
+   In [6]: pd.Series([1, 2]).groupby(grouper).sum()
+   Out[6]:
+   a    3.0
+   b    NaN
+   dtype: float64
+
+*pandas 0.22*
+
+.. ipython:: python
+
+   grouper = pd.Categorical(['a', 'a'], categories=['a', 'b'])
+   pd.Series([1, 2]).groupby(grouper).sum()
+
+   pd.Series([1, 2]).groupby(grouper).sum(min_count=1)
+
+2. Resampling and taking the ``sum`` / ``prod``.
+
+The output for an all-NaN bin will change:
+
+*pandas 0.21.0*
+
+.. code-block:: ipython
+
+   In [7]: s = pd.Series([1, 1, np.nan, np.nan],
+      ...:               index=pd.date_range('2017', periods=4))
+      ...:
+
+   In [8]: s
+   Out[8]:
+   2017-01-01    1.0
+   2017-01-02    1.0
+   2017-01-03    NaN
+   2017-01-04    NaN
+   Freq: D, dtype: float64
+
+   In [9]: s.resample('2d').sum()
+   Out[9]:
+   2017-01-01    2.0
+   2017-01-03    NaN
+   Freq: 2D, dtype: float64
+
+*pandas 0.22.0*
+
+.. ipython:: python
+
+   s = pd.Series([1, 1, np.nan, np.nan],
+                 index=pd.date_range('2017', periods=4))
+   s.resample('2d').sum()
+
+To restore the 0.21 behavior of returning ``NaN``, use ``min_count>=1``.
+
+.. ipython:: python
+
+   s.resample('2d').sum(min_count=1)
+
+In particular, upsampling and taking the sum or product is affected, as
+upsampling introduces all-NaN series even if your original series was
+entirely valid.
+
+*pandas 0.21.0*
+
+.. code-block:: ipython
+
+   In [10]: idx = pd.DatetimeIndex(['2017-01-01', '2017-01-02'])
+
+   In [10]: pd.Series([1, 2], index=idx).resample('12H').sum()
+   Out[10]:
+   2017-01-01 00:00:00    1.0
+   2017-01-01 12:00:00    NaN
+   2017-01-02 00:00:00    2.0
+   Freq: 12H, dtype: float64
+
+*pandas 0.22.0*
+
+.. ipython:: python
+
+   idx = pd.DatetimeIndex(['2017-01-01', '2017-01-02'])
+   pd.Series([1, 2], index=idx).resample("12H").sum()
+
+Once again, the ``min_count`` keyword is available to restore the 0.21 behavior.
+
+   pd.Series([1, 2], index=idx).resample("12H").sum(min_count=1)
+
+3. Rolling / Expanding window operations and taking the ``sum`` / ``prod``.
+
+Rolling and expanding already have a ``min_periods`` keyword that behaves
+similarly to ``min_count``. The only case that changes is when doing a rolling
+or expanding sum on an all-NaN series with ``min_periods=0``. Previously this
+returned ``NaN``, now it will return ``0`` (or ``1`` for ``prod``).
+
+*pandas 0.21.1*
+
+.. ipython:: python
+
+   In [11]: s = pd.Series([np.nan, np.nan])
+
+   In [12]: s.rolling(2, min_periods=0).sum()
+   Out[12]:
+   0   NaN
+   1   NaN
+   dtype: float64
+
+*pandas 0.22.0*
+
+.. ipython:: python
+
+   In [2]: s = pd.Series([np.nan, np.nan])
+
+   In [3]: s.rolling(2, min_periods=0).sum()

pandas/_libs/groupby_helper.pxi.in

@@ -37,7 +37,7 @@ def group_add_{{name}}(ndarray[{{dest_type2}}, ndim=2] out,
                        ndarray[int64_t] counts,
                        ndarray[{{c_type}}, ndim=2] values,
                        ndarray[int64_t] labels,
-                       Py_ssize_t min_count=1):
+                       Py_ssize_t min_count=0):
     """
     Only aggregates on axis=0
     """
@@ -101,7 +101,7 @@ def group_prod_{{name}}(ndarray[{{dest_type2}}, ndim=2] out,
                         ndarray[int64_t] counts,
                         ndarray[{{c_type}}, ndim=2] values,
                         ndarray[int64_t] labels,
-                        Py_ssize_t min_count=1):
+                        Py_ssize_t min_count=0):
     """
     Only aggregates on axis=0
     """

pandas/_libs/window.pyx

@@ -409,10 +409,11 @@ def roll_count(ndarray[double_t] input, int64_t win, int64_t minp,
 # Rolling sum
 
 
-cdef inline double calc_sum(int64_t minp, int64_t nobs, double sum_x) nogil:
+cdef inline double calc_sum(int64_t minp, int64_t nobs, double sum_x,
+                            bint no_min=False) nogil:
     cdef double result
 
-    if nobs >= minp:
+    if no_min or nobs >= minp:
         result = sum_x
     else:
         result = NaN
@@ -443,10 +444,14 @@ def roll_sum(ndarray[double_t] input, int64_t win, int64_t minp,
         double val, prev_x, sum_x = 0
         int64_t s, e
         int64_t nobs = 0, i, j, N
-        bint is_variable
+        bint is_variable, no_min
         ndarray[int64_t] start, end
         ndarray[double_t] output
 
+    if minp == 0:
+        no_min = True
+    else:
+        no_min = False
     start, end, N, win, minp, is_variable = get_window_indexer(input, win,
                                                                minp, index,
                                                                closed)
@@ -483,7 +488,7 @@ def roll_sum(ndarray[double_t] input, int64_t win, int64_t minp,
                     for j in range(end[i - 1], e):
                         add_sum(input[j], &nobs, &sum_x)
 
-                output[i] = calc_sum(minp, nobs, sum_x)
+                output[i] = calc_sum(minp, nobs, sum_x, no_min)
 
     else:
 
@@ -503,7 +508,7 @@ def roll_sum(ndarray[double_t] input, int64_t win, int64_t minp,
                     prev_x = input[i - win]
                     remove_sum(prev_x, &nobs, &sum_x)
 
-                output[i] = calc_sum(minp, nobs, sum_x)
+                output[i] = calc_sum(minp, nobs, sum_x, no_min)
 
     return output
 

pandas/core/generic.py

@@ -7619,48 +7619,48 @@ def _doc_parms(cls):
 _sum_examples = """\
 Examples
 --------
-By default, the sum of an empty series is ``NaN``.
+By default, the sum of an empty series is ``0``.
 
->>> pd.Series([]).sum()  # min_count=1 is the default
-nan
+>>> pd.Series([]).sum()  # min_count=0 is the default
+0.0
 
 This can be controlled with the ``min_count`` parameter. For example, if
-you'd like the sum of an empty series to be 0, pass ``min_count=0``.
+you'd like the sum of an empty series to be NaN, pass ``min_count=1``.
 
->>> pd.Series([]).sum(min_count=0)
-0.0
+>>> pd.Series([]).sum(min_count=1)
+nan
 
 Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
 empty series identically.
 
 >>> pd.Series([np.nan]).sum()
-nan
-
->>> pd.Series([np.nan]).sum(min_count=0)
 0.0
+
+>>> pd.Series([np.nan]).sum(min_count=1)
+nan
 """
 
 _prod_examples = """\
 Examples
 --------
-By default, the product of an empty series is ``NaN``
+By default, the product of an empty series is ``1``
 
 >>> pd.Series([]).prod()
-nan
+1.0
 
 This can be controlled with the ``min_count`` parameter
 
->>> pd.Series([]).prod(min_count=0)
-1.0
+>>> pd.Series([]).prod(min_count=1)
+nan
 
 Thanks to the ``skipna`` parameter, ``min_count`` handles all-NA and
 empty series identically.
 
 >>> pd.Series([np.nan]).prod()
-nan
-
->>> pd.Series([np.nan]).sum(min_count=0)
 1.0
+
+>>> pd.Series([np.nan]).sum(min_count=1)
+nan
 """
 
 
@@ -7683,7 +7683,7 @@ def _make_min_count_stat_function(cls, name, name1, name2, axis_descr, desc,
                   examples=examples)
     @Appender(_num_doc)
     def stat_func(self, axis=None, skipna=None, level=None, numeric_only=None,
-                  min_count=1,
+                  min_count=0,
                   **kwargs):
         nv.validate_stat_func(tuple(), kwargs, fname=name)
         if skipna is None:

pandas/core/groupby.py

@@ -1286,8 +1286,8 @@ def last(x):
             else:
                 return last(x)
 
-        cls.sum = groupby_function('sum', 'add', np.sum, min_count=1)
-        cls.prod = groupby_function('prod', 'prod', np.prod, min_count=1)
+        cls.sum = groupby_function('sum', 'add', np.sum, min_count=0)
+        cls.prod = groupby_function('prod', 'prod', np.prod, min_count=0)
         cls.min = groupby_function('min', 'min', np.min, numeric_only=False)
         cls.max = groupby_function('max', 'max', np.max, numeric_only=False)
         cls.first = groupby_function('first', 'first', first_compat,

pandas/core/nanops.py

@@ -109,6 +109,8 @@ def f(values, axis=None, skipna=True, **kwds):
             try:
                 if values.size == 0 and kwds.get('min_count') is None:
                     # We are empty, returning NA for our type
+                    # Only applies for the default `min_count` of None
+                    # since that affects how empty arrays are handled.
                     return _na_for_min_count(values, axis)
 
                 if (_USE_BOTTLENECK and skipna and
@@ -308,7 +310,7 @@ def nanall(values, axis=None, skipna=True):
 
 @disallow('M8')
 @bottleneck_switch()
-def nansum(values, axis=None, skipna=True, min_count=1):
+def nansum(values, axis=None, skipna=True, min_count=0):
     values, mask, dtype, dtype_max = _get_values(values, skipna, 0)
     dtype_sum = dtype_max
     if is_float_dtype(dtype):
@@ -645,7 +647,7 @@ def nankurt(values, axis=None, skipna=True):
 
 
 @disallow('M8', 'm8')
-def nanprod(values, axis=None, skipna=True, min_count=1):
+def nanprod(values, axis=None, skipna=True, min_count=0):
     mask = isna(values)
     if skipna and not is_any_int_dtype(values):
         values = values.copy()

pandas/core/resample.py

@@ -605,7 +605,7 @@ def size(self):
 # downsample methods
 for method in ['sum', 'prod']:
 
-    def f(self, _method=method, min_count=1, *args, **kwargs):
+    def f(self, _method=method, min_count=0, *args, **kwargs):
         nv.validate_resampler_func(_method, args, kwargs)
         return self._downsample(_method, min_count=min_count)
     f.__doc__ = getattr(GroupBy, method).__doc__