Merge branch 'master' of https://github.com/pandas-dev/pandas into libmore

Author: jbrockmendel

doc/source/advanced.rst

@@ -503,6 +503,47 @@ method, allowing you to permute the hierarchical index levels in one step:
 
    df[:5].reorder_levels([1,0], axis=0)
 
+.. _advanced.index_names:
+
+Renaming names of an ``Index`` or ``MultiIndex``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The :meth:`~DataFrame.rename` method is used to rename the labels of a
+``MultiIndex``, and is typically used to rename the columns of a ``DataFrame``.
+The ``columns`` argument of ``rename`` allows a dictionary to be specified
+that includes only the columns you wish to rename.
+
+.. ipython:: python
+
+   df.rename(columns={0: "col0", 1: "col1"})
+
+This method can also be used to rename specific labels of the main index
+of the ``DataFrame``.
+
+.. ipython:: python
+
+   df.rename(index={"one" : "two", "y" : "z"})
+
+The :meth:`~DataFrame.rename_axis` method is used to rename the name of a
+``Index`` or ``MultiIndex``. In particular, the names of the levels of a
+``MultiIndex`` can be specified, which is useful if ``reset_index()`` is later
+used to move the values from the ``MultiIndex`` to a column.
+
+.. ipython:: python
+
+   df.rename_axis(index=['abc', 'def'])
+   
+Note that the columns of a ``DataFrame`` are an index, so that using
+``rename_axis`` with the ``columns`` argument will change the name of that
+index.
+
+.. ipython:: python
+
+   df.rename_axis(columns="Cols").columns
+
+Both ``rename`` and ``rename_axis`` support specifying a dictionary,
+``Series`` or a mapping function to map labels/names to new values.
+
 Sorting a ``MultiIndex``
 ------------------------
 

doc/source/basics.rst

@@ -1466,8 +1466,21 @@ for altering the ``Series.name`` attribute.
 
 .. _basics.rename_axis:
 
-The Panel class has a related :meth:`~Panel.rename_axis` class which can rename
-any of its three axes.
+.. versionadded:: 0.24.0
+
+The methods :meth:`~DataFrame.rename_axis` and :meth:`~Series.rename_axis`
+allow specific names of a `MultiIndex` to be changed (as opposed to the
+labels).
+
+.. ipython:: python
+
+   df = pd.DataFrame({'x': [1, 2, 3, 4, 5, 6],
+                      'y': [10, 20, 30, 40, 50, 60]},
+                     index=pd.MultiIndex.from_product([['a', 'b', 'c'], [1, 2]],
+                     names=['let', 'num']))
+   df
+   df.rename_axis(index={'let': 'abc'})
+   df.rename_axis(index=str.upper)
 
 .. _basics.iteration:
 

doc/source/whatsnew/v0.24.0.txt

@@ -180,6 +180,29 @@ array, but rather an ``ExtensionArray``:
 This is the same behavior as ``Series.values`` for categorical data. See
 :ref:`whatsnew_0240.api_breaking.interval_values` for more.
 
+.. _whatsnew_0240.enhancements.rename_axis:
+
+Renaming names in a MultiIndex
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:func:`DataFrame.rename_axis` now supports ``index`` and ``columns`` arguments
+and :func:`Series.rename_axis` supports ``index`` argument (:issue:`19978`)
+
+This change allows a dictionary to be passed so that some of the names 
+of a ``MultiIndex`` can be changed.
+
+Example:
+
+.. ipython:: python
+
+    mi = pd.MultiIndex.from_product([list('AB'), list('CD'), list('EF')],
+                                    names=['AB', 'CD', 'EF'])
+    df = pd.DataFrame([i for i in range(len(mi))], index=mi, columns=['N'])
+    df
+    df.rename_axis(index={'CD': 'New'})
+
+See the :ref:`advanced docs on renaming<advanced.index_names>` for more details.
+
 .. _whatsnew_0240.enhancements.other:
 
 Other Enhancements
@@ -930,6 +953,7 @@ Deprecations
 - :func:`DatetimeIndex.shift` and :func:`PeriodIndex.shift` now accept ``periods`` argument instead of ``n`` for consistency with :func:`Index.shift` and :func:`Series.shift`. Using ``n`` throws a deprecation warning (:issue:`22458`, :issue:`22912`)
 - The ``fastpath`` keyword of the different Index constructors is deprecated (:issue:`23110`).
 - :meth:`Timestamp.tz_localize`, :meth:`DatetimeIndex.tz_localize`, and :meth:`Series.tz_localize` have deprecated the ``errors`` argument in favor of the ``nonexistent`` argument (:issue:`8917`)
+- The class ``FrozenNDArray`` has been deprecated. When unpickling, ``FrozenNDArray`` will be unpickled to ``np.ndarray`` once this class is removed (:issue:`9031`)
 
 .. _whatsnew_0240.prior_deprecations:
 
@@ -942,7 +966,7 @@ Removal of prior version deprecations/changes
 - Removal of the previously deprecated module ``pandas.core.datetools`` (:issue:`14105`, :issue:`14094`)
 - Strings passed into :meth:`DataFrame.groupby` that refer to both column and index levels will raise a ``ValueError`` (:issue:`14432`)
 - :meth:`Index.repeat` and :meth:`MultiIndex.repeat` have renamed the ``n`` argument to ``repeats`` (:issue:`14645`)
-- The ``Series`` constructor and ``.astype`` method will now raise a ``ValueError`` if timestamp dtypes are passed in without a frequency (e.g. ``np.datetime64``) for the ``dtype`` parameter (:issue:`15987`)
+- The ``Series`` constructor and ``.astype`` method will now raise a ``ValueError`` if timestamp dtypes are passed in without a unit (e.g. ``np.datetime64``) for the ``dtype`` parameter (:issue:`15987`)
 - Removal of the previously deprecated ``as_indexer`` keyword completely from ``str.match()`` (:issue:`22356`, :issue:`6581`)
 - The modules ``pandas.types``, ``pandas.computation``, and ``pandas.util.decorators`` have been removed (:issue:`16157`, :issue:`16250`)
 - Removed the ``pandas.formats.style`` shim for :class:`pandas.io.formats.style.Styler` (:issue:`16059`)
@@ -1030,6 +1054,7 @@ Datetimelike
 - Bug in :func:`to_datetime` with an :class:`Index` argument that would drop the ``name`` from the result (:issue:`21697`)
 - Bug in :class:`PeriodIndex` where adding or subtracting a :class:`timedelta` or :class:`Tick` object produced incorrect results (:issue:`22988`)
 - Bug in :func:`date_range` when decrementing a start date to a past end date by a negative frequency (:issue:`23270`)
+- Bug in :meth:`Series.min` which would return ``NaN`` instead of ``NaT`` when called on a series of ``NaT`` (:issue:`23282`)
 - Bug in :func:`DataFrame.combine` with datetimelike values raising a TypeError (:issue:`23079`)
 - Bug in :func:`date_range` with frequency of ``Day`` or higher where dates sufficiently far in the future could wrap around to the past instead of raising ``OutOfBoundsDatetime`` (:issue:`14187`)
 - Bug in :class:`PeriodIndex` with attribute ``freq.n`` greater than 1 where adding a :class:`DateOffset` object would return incorrect results (:issue:`23215`)
@@ -1108,6 +1133,7 @@ Interval
 - Bug in the ``IntervalIndex`` repr where a trailing comma was missing after the list of intervals (:issue:`20611`)
 - Bug in :class:`Interval` where scalar arithmetic operations did not retain the ``closed`` value (:issue:`22313`)
 - Bug in :class:`IntervalIndex` where indexing with datetime-like values raised a ``KeyError`` (:issue:`20636`)
+- Bug in ``IntervalTree`` where data containing ``NaN`` triggered a warning and resulted in incorrect indexing queries with :class:`IntervalIndex` (:issue:`23352`)
 
 Indexing
 ^^^^^^^^

pandas/_libs/intervaltree.pxi.in

@@ -72,6 +72,12 @@ cdef class IntervalTree(IntervalMixin):
 
         self.closed = closed
 
+        # GH 23352: ensure no nan in nodes
+        mask = ~np.isnan(self.left)
+        self.left = self.left[mask]
+        self.right = self.right[mask]
+        indices = indices[mask]
+
         node_cls = NODE_CLASSES[str(self.dtype), closed]
         self.root = node_cls(self.left, self.right, indices, leaf_size)
 

pandas/_version.py

@@ -12,6 +12,7 @@
 import re
 import subprocess
 import sys
+
 from pandas.compat import PY3
 
 

pandas/compat/pickle_compat.py

@@ -60,6 +60,17 @@ def load_reduce(self):
         ('pandas.core.arrays', 'SparseArray'),
 
     # 15477
+    #
+    # TODO: When FrozenNDArray is removed, add
+    # the following lines for compat:
+    #
+    # ('pandas.core.base', 'FrozenNDArray'):
+    #     ('numpy', 'ndarray'),
+    # ('pandas.core.indexes.frozen', 'FrozenNDArray'):
+    #     ('numpy', 'ndarray'),
+    #
+    # Afterwards, remove the current entry
+    # for `pandas.core.base.FrozenNDArray`.
     ('pandas.core.base', 'FrozenNDArray'):
         ('pandas.core.indexes.frozen', 'FrozenNDArray'),
     ('pandas.core.base', 'FrozenList'):

pandas/conftest.py

@@ -1,15 +1,15 @@
-import os
 import importlib
+import os
 
+import hypothesis
+from hypothesis import strategies as st
+import numpy as np
 import pytest
 
-import pandas
-import numpy as np
-import pandas as pd
 from pandas.compat import PY3
 import pandas.util._test_decorators as td
-import hypothesis
 
+import pandas as pd
 
 hypothesis.settings.register_profile(
     "ci",
@@ -285,7 +285,7 @@ def deco(*args):
 @pytest.fixture
 def iris(datapath):
     """The iris dataset as a DataFrame."""
-    return pandas.read_csv(datapath('data', 'iris.csv'))
+    return pd.read_csv(datapath('data', 'iris.csv'))
 
 
 @pytest.fixture(params=['nlargest', 'nsmallest'])
@@ -512,7 +512,6 @@ def mock():
 # ----------------------------------------------------------------
 # Global setup for tests using Hypothesis
 
-from hypothesis import strategies as st
 
 # Registering these strategies makes them globally available via st.from_type,
 # which is use for offsets in tests/tseries/offsets/test_offsets_properties.py

pandas/core/arrays/sparse.py

@@ -10,35 +10,35 @@
 
 import numpy as np
 
-import pandas._libs.sparse as splib
-import pandas.core.algorithms as algos
-import pandas.core.common as com
-import pandas.io.formats.printing as printing
-from pandas import compat
 from pandas._libs import index as libindex, lib
+import pandas._libs.sparse as splib
 from pandas._libs.sparse import BlockIndex, IntIndex
 from pandas._libs.tslibs import NaT
+import pandas.compat as compat
 from pandas.compat.numpy import function as nv
-from pandas.core.accessor import PandasDelegate, delegate_names
-from pandas.core.arrays import ExtensionArray, ExtensionOpsMixin
-from pandas.core.base import PandasObject
+from pandas.errors import PerformanceWarning
+
 from pandas.core.dtypes.base import ExtensionDtype
 from pandas.core.dtypes.cast import (
     astype_nansafe, construct_1d_arraylike_from_scalar, find_common_type,
-    infer_dtype_from_scalar, maybe_convert_platform
-)
+    infer_dtype_from_scalar, maybe_convert_platform)
 from pandas.core.dtypes.common import (
     is_array_like, is_bool_dtype, is_datetime64_any_dtype, is_dtype_equal,
     is_integer, is_list_like, is_object_dtype, is_scalar, is_string_dtype,
-    pandas_dtype
-)
+    pandas_dtype)
 from pandas.core.dtypes.dtypes import register_extension_dtype
 from pandas.core.dtypes.generic import (
-    ABCIndexClass, ABCSeries, ABCSparseSeries
-)
+    ABCIndexClass, ABCSeries, ABCSparseSeries)
 from pandas.core.dtypes.missing import isna, na_value_for_dtype, notna
+
+from pandas.core.accessor import PandasDelegate, delegate_names
+import pandas.core.algorithms as algos
+from pandas.core.arrays import ExtensionArray, ExtensionOpsMixin
+from pandas.core.base import PandasObject
+import pandas.core.common as com
 from pandas.core.missing import interpolate_2d
-from pandas.errors import PerformanceWarning
+
+import pandas.io.formats.printing as printing
 
 
 # ----------------------------------------------------------------------------

pandas/core/common.py

@@ -454,3 +454,21 @@ def _pipe(obj, func, *args, **kwargs):
         return func(*args, **kwargs)
     else:
         return func(obj, *args, **kwargs)
+
+
+def _get_rename_function(mapper):
+    """
+    Returns a function that will map names/labels, dependent if mapper
+    is a dict, Series or just a function.
+    """
+    if isinstance(mapper, (compat.Mapping, ABCSeries)):
+
+        def f(x):
+            if x in mapper:
+                return mapper[x]
+            else:
+                return x
+    else:
+        f = mapper
+
+    return f

pandas/core/dtypes/cast.py

@@ -667,7 +667,7 @@ def astype_nansafe(arr, dtype, copy=True, skipna=False):
     Raises
     ------
     ValueError
-        The dtype was a datetime /timedelta dtype, but it had no frequency.
+        The dtype was a datetime64/timedelta64 dtype, but it had no unit.
     """
 
     # dispatch on extension dtype if needed
@@ -749,7 +749,7 @@ def astype_nansafe(arr, dtype, copy=True, skipna=False):
             return astype_nansafe(to_timedelta(arr).values, dtype, copy=copy)
 
     if dtype.name in ("datetime64", "timedelta64"):
-        msg = ("The '{dtype}' dtype has no frequency. "
+        msg = ("The '{dtype}' dtype has no unit. "
                "Please pass in '{dtype}[ns]' instead.")
         raise ValueError(msg.format(dtype=dtype.name))
 
@@ -1021,7 +1021,7 @@ def maybe_cast_to_datetime(value, dtype, errors='raise'):
         if is_datetime64 or is_datetime64tz or is_timedelta64:
 
             # Force the dtype if needed.
-            msg = ("The '{dtype}' dtype has no frequency. "
+            msg = ("The '{dtype}' dtype has no unit. "
                    "Please pass in '{dtype}[ns]' instead.")
 
             if is_datetime64 and not is_dtype_equal(dtype, _NS_DTYPE):

pandas/core/frame.py

@@ -4944,7 +4944,7 @@ def _combine_match_columns(self, other, func, level=None):
         assert left.columns.equals(right.index)
         return ops.dispatch_to_series(left, right, func, axis="columns")
 
-    def _combine_const(self, other, func, errors='raise'):
+    def _combine_const(self, other, func):
         assert lib.is_scalar(other) or np.ndim(other) == 0
         return ops.dispatch_to_series(self, other, func)