API: unique labels

Author: TomAugspurger

doc/source/index.rst.template

@@ -71,6 +71,7 @@ See the :ref:`overview` for more detail about what's in the library.
   * :doc:`user_guide/reshaping`
   * :doc:`user_guide/text`
   * :doc:`user_guide/missing_data`
+  * :doc:`user_guide/duplicates`
   * :doc:`user_guide/categorical`
   * :doc:`user_guide/integer_na`
   * :doc:`user_guide/visualization`

doc/source/reference/frame.rst

@@ -23,6 +23,7 @@ Attributes and underlying data
 
    DataFrame.index
    DataFrame.columns
+   DataFrame.allows_duplicate_labels
 
 .. autosummary::
    :toctree: api/

doc/source/reference/series.rst

@@ -22,6 +22,7 @@ Attributes
    :toctree: api/
 
    Series.index
+   Series.allows_duplicate_labels
 
 .. autosummary::
    :toctree: api/

doc/source/user_guide/duplicates.rst

@@ -0,0 +1,174 @@
+.. _duplicates:
+
+****************
+Duplicate Labels
+****************
+
+:class:`Index` objects are not required to be unique; you can have duplicate row
+or column labels. This may be a bit confusing at first. If you're familiar with
+SQL, you know that row labels are similar to a primary key on a table, and you
+would never want duplicates in a SQL table. But one of pandas' roles is to clean
+messy, real-world data before it goes to some downstream system. And real-world
+data has duplicates, even in fields that are supposed to be unique.
+
+This section describes how duplicate labels change the behavior of certain
+operations, and how prevent duplicates from arising during operations, or to
+detect them if they do.
+
+.. ipython:: python
+
+   import pandas as pd
+   import numpy as np
+
+Consequences of Duplicate Labels
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Some pandas methods (:meth:`Series.reindex` for example) just don't work with
+duplicates present. The output can't be determined, and so pandas raises.
+
+.. ipython:: python
+   :okexcept:
+
+   s1 = pd.Series([0, 1, 2], index=['a', 'b', 'b'])
+   s1.reindex(['a', 'b', 'c'])
+
+Other methods, like indexing, can give very surprising results. Typically
+indexing with a scalar will *reduce dimensionality*. Slicing a ``DataFrame``
+with a scalar will return a ``Series``. Slicing a ``Series`` with a scalar will
+return a scalar. But with duplicates, this isn't the case.
+
+.. ipython:: python
+
+   df1 = pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=['A', 'A', 'B'])
+   df1
+
+We have duplicates in the columns. If we slice ``'B'``, we get back a ``Series``
+
+.. ipython:: python
+
+   df1['B']  # a series
+
+But slicing ``'A'`` returns a ``DataFrame``
+
+
+.. ipython:: python
+
+   df1['A']  # a DataFrame
+
+This applies to row labels as well
+
+.. ipython:: python
+
+   df2 = pd.DataFrame({"A": [0, 1, 2]}, index=['a', 'a', 'b'])
+   df2
+   df2.loc['b', 'A']  # a scalar
+
+   df2.loc['a', 'A']  # a Series
+
+Duplicate Label Detection
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can check with an :class:`Index` (storing the row or column labels) is
+unique with :attr:`Index.is_unique`:
+
+.. ipython:: python
+
+   df2
+   df2.index.is_unique
+   df2.columns.is_unique
+
+.. note::
+
+   Checking whether an index is unique is somewhat expensive for large datasets.
+   Pandas does cache this result, so re-checking on the same index is very fast.
+
+:meth:`Index.duplicated` will return a boolean ndarray indicating whether a
+label is a repeat.
+
+.. ipython:: python
+
+   df2.index.duplicated()
+
+Which can be used as a boolean filter to drop duplicate rows.
+
+.. ipython:: python
+
+   df2.loc[~df2.index.duplicated(), :]
+
+If you need additional logic to handle duplicate labels, rather than just
+dropping the repeats, using :meth:`~DataFrame.groupby` on the index is a common
+trick. For example, we'll resolve duplicates by taking the average of all rows
+with the same label.
+
+.. ipython:: python
+
+   df2.groupby(level=0).mean()
+
+.. _duplicates.disallow:
+
+Disallowing Duplicate Labels
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+As noted above, handling duplicates is an important feature when reading in raw
+data. That said, you may want to avoid introducing duplicates as part of a data
+processing pipeline (from methods like :meth:`pandas.concat`,
+:meth:`~DataFrame.rename`, etc.). Both :class:`Series` and :class:`DataFrame`
+can be created with the argument ``allow_duplicate_labels=False`` to *disallow*
+duplicate labels (the default is to allow them). If there are duplicate labels,
+an exception will be raised.
+
+.. ipython:: python
+   :okexcept:
+
+   pd.Series([0, 1, 2], index=['a', 'b', 'b'], allow_duplicate_labels=False)
+
+This applies to both row and column labels for a :class:`DataFrame`
+
+.. ipython:: python
+   :okexcept:
+
+   pd.DataFrame([[0, 1, 2], [3, 4, 5]], columns=["A", "B", "C"],
+                allow_duplicate_labels=False)
+
+This attribute can be checked with :attr:`~DataFrame.allows_duplicate_labels`,
+which indicates whether that object can have duplicate labels.
+
+.. ipython:: python
+
+   df = pd.DataFrame({"A": [0, 1, 2, 3]}, index=['x', 'y', 'X', 'Y'],
+                     allow_duplicate_labels=False)
+   df
+   df.allows_duplicate_labels
+
+Performing an operation that introduces duplicate labels on a ``Series`` or
+``DataFrame`` that disallows duplicates will raise an
+:class:`errors.DuplicateLabelError`.
+
+.. ipython:: python
+   :okexcept:
+
+   df.rename(str.upper)
+
+Duplicate Label Propagation
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In general, disallowing duplicates is "sticky". It's preserved through
+operations.
+
+.. ipython:: python
+   :okexcept:
+
+   s1 = pd.Series(0, index=['a', 'b'], allow_duplicate_labels=False)
+   s1
+   abs(s1).rename({"a": "b"})
+
+When multiple Series or DataFrames are involved in an operation,
+duplictes are disallowed if *any* of the inputs disallow duplicates.
+
+.. ipython:: python
+   :okexcept:
+
+   df1 = pd.Series(0, index=['a', 'b'], allow_duplicate_labels=False)
+   df2 = pd.Series(1, index=['b', 'c'], allow_duplicate_labels=True)
+
+   pd.concat([df1, df2])

doc/source/whatsnew/v1.0.0.rst

@@ -406,6 +406,9 @@ Other
 - Trying to set the ``display.precision``, ``display.max_rows`` or ``display.max_columns`` using :meth:`set_option` to anything but a ``None`` or a positive int will raise a ``ValueError`` (:issue:`23348`)
 - Using :meth:`DataFrame.replace` with overlapping keys in a nested dictionary will no longer raise, now matching the behavior of a flat dictionary (:issue:`27660`)
 - :meth:`DataFrame.to_csv` and :meth:`Series.to_csv` now support dicts as ``compression`` argument with key ``'method'`` being the compression method and others as additional compression options when the compression method is ``'zip'``. (:issue:`26023`)
+- Metadata is now finalized for the following methods on ``Series`` and ``DataFrame`` (:issue:``)
+   * :meth:`~DataFrame.abs`
+   * :meth:`Series.to_frame`
 - Bug in :meth:`Series.diff` where a boolean series would incorrectly raise a ``TypeError`` (:issue:`17294`)
 - :meth:`Series.append` will no longer raise a ``TypeError`` when passed a tuple of ``Series`` (:issue:`28410`)
 - Fix corrupted error message when calling ``pandas.libs._json.encode()`` on a 0d array (:issue:`18878`)

pandas/core/frame.py

@@ -1,4 +1,4 @@
-"""
+""""
 DataFrame
 ---------
 An efficient 2D container for potentially mixed-type time series or other
@@ -338,6 +338,13 @@ class DataFrame(NDFrame):
         Data type to force. Only a single dtype is allowed. If None, infer.
     copy : bool, default False
         Copy data from inputs. Only affects DataFrame / 2d ndarray input.
+    allow_duplicate_labels : bool, default True
+        Whether to allow duplicate row or column labels in this DataFrame.
+        By default, duplicte labels are permitted. Setting this to ``False``
+        will cause an :class:`errors.DuplicateLabelError` to be raised when
+        `index` or `columns` are not unique, or when any subsequent operation
+        on this DataFrame introduces duplicates. See :ref:`duplictes.disallow`
+        for more.
 
     See Also
     --------
@@ -407,6 +414,7 @@ def __init__(
         columns: Optional[Axes] = None,
         dtype: Optional[Dtype] = None,
         copy: bool = False,
+        allow_duplicate_labels: bool = True,
     ):
         if data is None:
             data = {}
@@ -497,7 +505,9 @@ def __init__(
             else:
                 raise ValueError("DataFrame constructor not properly called!")
 
-        NDFrame.__init__(self, mgr, fastpath=True)
+        NDFrame.__init__(
+            self, mgr, fastpath=True, allow_duplicate_labels=allow_duplicate_labels
+        )
 
     # ----------------------------------------------------------------------
 
@@ -2770,6 +2780,8 @@ def _ixs(self, i: int, axis: int = 0):
         If slice passed, the resulting data will be a view.
         """
         # irow
+        # TODO: Figure out if this is the right place to finalize.
+        #   Does it make sense to do here, or higher-level (like `LocationIndexer`)?
         if axis == 0:
             label = self.index[i]
             new_values = self._data.fast_xs(i)
@@ -2781,7 +2793,7 @@ def _ixs(self, i: int, axis: int = 0):
                 index=self.columns,
                 name=self.index[i],
                 dtype=new_values.dtype,
-            )
+            ).__finalize__(self, method="ixs")
             result._set_is_copy(self, copy=copy)
             return result
 
@@ -2798,6 +2810,8 @@ def _ixs(self, i: int, axis: int = 0):
             if len(self.index) and not len(values):
                 values = np.array([np.nan] * len(self.index), dtype=object)
             result = self._box_col_values(values, label)
+            if isinstance(result, NDFrame):
+                result.__finalize__(self, method="ixs")
 
             # this is a cached value, mark it so
             result._set_as_cached(label, self)
@@ -2859,6 +2873,8 @@ def __getitem__(self, key):
             if data.shape[1] == 1 and not isinstance(self.columns, ABCMultiIndex):
                 data = data[key]
 
+        if isinstance(data, NDFrame):
+            data.__finalize__(self, method="dataframe_getitem")
         return data
 
     def _getitem_bool_array(self, key):
@@ -5300,6 +5316,7 @@ def _arith_op(left, right):
             with np.errstate(all="ignore"):
                 res_values = _arith_op(this.values, other.values)
             new_data = dispatch_fill_zeros(func, this.values, other.values, res_values)
+        # XXX: pass them here.
         return this._construct_result(new_data)
 
     def _combine_match_index(self, other, func):