Merge branch 'pandas-dev:master' into value_counts-with-duplicate-labels

Author: johnzangwill

.pre-commit-config.yaml

@@ -9,7 +9,7 @@ repos:
     -   id: absolufy-imports
         files: ^pandas/
 -   repo: https://github.com/python/black
-    rev: 21.11b1
+    rev: 21.12b0
     hooks:
     -   id: black
 -   repo: https://github.com/codespell-project/codespell
@@ -19,7 +19,7 @@ repos:
         types_or: [python, rst, markdown]
         files: ^(pandas|doc)/
 -   repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v4.0.1
+    rev: v4.1.0
     hooks:
     -   id: debug-statements
     -   id: end-of-file-fixer
@@ -49,7 +49,7 @@ repos:
     hooks:
     -   id: isort
 -   repo: https://github.com/asottile/pyupgrade
-    rev: v2.29.1
+    rev: v2.31.0
     hooks:
     -   id: pyupgrade
         args: [--py38-plus]

doc/source/reference/indexing.rst

@@ -170,7 +170,6 @@ Numeric Index
    :toctree: api/
    :template: autosummary/class_without_autosummary.rst
 
-   NumericIndex
    RangeIndex
    Int64Index
    UInt64Index

doc/source/user_guide/advanced.rst

@@ -852,10 +852,9 @@ Int64Index and RangeIndex
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. deprecated:: 1.4.0
-    In pandas 2.0, :class:`NumericIndex` will become the default index type for numeric types
+    In pandas 2.0, :class:`Index` will become the default index type for numeric types
     instead of ``Int64Index``, ``Float64Index`` and ``UInt64Index`` and those index types
     are therefore deprecated and will be removed in a futire version.
-    See :ref:`here <advanced.numericindex>` for more.
     ``RangeIndex`` will not be removed, as it represents an optimized version of an integer index.
 
 :class:`Int64Index` is a fundamental basic index in pandas. This is an immutable array
@@ -870,10 +869,9 @@ Float64Index
 ~~~~~~~~~~~~
 
 .. deprecated:: 1.4.0
-    :class:`NumericIndex` will become the default index type for numeric types in the future
+    :class:`Index` will become the default index type for numeric types in the future
     instead of ``Int64Index``, ``Float64Index`` and ``UInt64Index`` and those index types
     are therefore deprecated and will be removed in a future version of Pandas.
-    See :ref:`here <advanced.numericindex>` for more.
     ``RangeIndex`` will not be removed as it represents an optimized version of an integer index.
 
 By default a :class:`Float64Index` will be automatically created when passing floating, or mixed-integer-floating values in index creation.
@@ -971,37 +969,6 @@ If you need integer based selection, you should use ``iloc``:
    dfir.iloc[0:5]
 
 
-.. _advanced.numericindex:
-
-NumericIndex
-~~~~~~~~~~~~
-
-.. versionadded:: 1.4.0
-
-.. note::
-
-    In pandas 2.0, :class:`NumericIndex` will become the default index type for numeric types
-    instead of :class:`Int64Index`, :class:`Float64Index` and :class:`UInt64Index` and those index types
-    are therefore deprecated and will be removed in a future version.
-    :class:`RangeIndex` will not be removed as it represents an optimized version of an integer index.
-
-:class:`NumericIndex` is an index type that can hold data of any numpy int/uint/float dtype. For example:
-
-.. ipython:: python
-
-   idx = pd.NumericIndex([1, 2, 4, 5], dtype="int8")
-   idx
-   ser = pd.Series(range(4), index=idx)
-   ser
-
-``NumericIndex`` works the same way as the existing ``Int64Index``, ``Float64Index`` and
-``UInt64Index`` except that it can hold any numpy int, uint or float dtype.
-
-Until Pandas 2.0, you will have to call ``NumericIndex`` explicitly in order to use it, like in the example above.
-In the future, ``NumericIndex`` will become the default pandas numeric index type and will automatically be used where appropriate.
-
-Please notice that ``NumericIndex`` *can not* hold Pandas numeric dtypes (:class:`Int64Dtype`, :class:`Int32Dtype` etc.).
-
 .. _advanced.intervalindex:
 
 IntervalIndex

doc/source/whatsnew/v0.16.2.rst

@@ -62,6 +62,7 @@ When the function you wish to apply takes its data anywhere other than the first
 of ``(function, keyword)`` indicating where the DataFrame should flow. For example:
 
 .. ipython:: python
+   :okwarning:
 
    import statsmodels.formula.api as sm
 

doc/source/whatsnew/v1.4.0.rst

@@ -40,55 +40,43 @@ This made it difficult to determine where the warning was being generated from.
     A value is trying to be set on a copy of a slice from a DataFrame.
 
 
-.. _whatsnew_140.enhancements.numeric_index:
 
-More flexible numeric dtypes for indexes
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Until now, it has only been possible to create numeric indexes with int64/float64/uint64 dtypes.
-It is now possible to create an index of any numpy int/uint/float dtype using the new :class:`NumericIndex` index type (:issue:`41153`):
+.. _whatsnew_140.enhancements.ExtensionIndex:
 
-.. ipython:: python
+Index can hold arbitrary ExtensionArrays
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-    pd.NumericIndex([1, 2, 3], dtype="int8")
-    pd.NumericIndex([1, 2, 3], dtype="uint32")
-    pd.NumericIndex([1, 2, 3], dtype="float32")
+Until now, passing a custom :class:`ExtensionArray` to ``pd.Index`` would cast the
+array to ``object`` dtype. Now :class:`Index` can directly hold arbitrary ExtensionArrays (:issue:`43930`).
 
-In order to maintain backwards compatibility, calls to the base :class:`Index` will currently
-return :class:`Int64Index`, :class:`UInt64Index` and :class:`Float64Index`, where relevant.
-For example, the code below returns an ``Int64Index`` with dtype ``int64``:
+*Previous behavior*:
 
-.. code-block:: ipython
+.. ipython:: python
 
-    In [1]: pd.Index([1, 2, 3], dtype="int8")
-    Int64Index([1, 2, 3], dtype='int64')
+   arr = pd.array([1, 2, pd.NA])
+   idx = pd.Index(arr)
 
-but will in a future version return a :class:`NumericIndex` with dtype ``int8``.
+In the old behavior, ``idx`` would be object-dtype:
 
-More generally, currently, all operations that until now have
-returned :class:`Int64Index`, :class:`UInt64Index` and :class:`Float64Index` will
-continue to so. This means, that in order to use ``NumericIndex`` in the current version, you
-will have to call ``NumericIndex`` explicitly. For example the below series will have an ``Int64Index``:
+*Previous behavior*:
 
 .. code-block:: ipython
 
-    In [2]: ser = pd.Series([1, 2, 3], index=[1, 2, 3])
-    In [3]: ser.index
-    Int64Index([1, 2, 3], dtype='int64')
+   In [1]: idx
+   Out[1]: Index([1, 2, <NA>], dtype='object')
 
-Instead, if you want to use a ``NumericIndex``, you should do:
+With the new behavior, we keep the original dtype:
 
-.. ipython:: python
+*New behavior*:
 
-    idx = pd.NumericIndex([1, 2, 3], dtype="int8")
-    ser = pd.Series([1, 2, 3], index=idx)
-    ser.index
+.. ipython:: python
 
-In a future version of Pandas, :class:`NumericIndex` will become the default numeric index type and
-``Int64Index``, ``UInt64Index`` and ``Float64Index`` are therefore deprecated and will
-be removed in the future, see :ref:`here <whatsnew_140.deprecations.int64_uint64_float64index>` for more.
+   idx
 
-See :ref:`here <advanced.numericindex>` for more about :class:`NumericIndex`.
+One exception to this is ``SparseArray``, which will continue to cast to numpy
+dtype until pandas 2.0. At that point it will retain its dtype like other
+ExtensionArrays.
 
 .. _whatsnew_140.enhancements.styler:
 
@@ -236,7 +224,7 @@ Other enhancements
 - :meth:`is_list_like` now identifies duck-arrays as list-like unless ``.ndim == 0`` (:issue:`35131`)
 - :class:`ExtensionDtype` and :class:`ExtensionArray` are now (de)serialized when exporting a :class:`DataFrame` with :meth:`DataFrame.to_json` using ``orient='table'`` (:issue:`20612`, :issue:`44705`).
 - Add support for `Zstandard <http://facebook.github.io/zstd/>`_ compression to :meth:`DataFrame.to_pickle`/:meth:`read_pickle` and friends (:issue:`43925`)
--
+- :meth:`DataFrame.to_sql` now returns an ``int`` of the number of written rows (:issue:`23998`)
 
 
 .. ---------------------------------------------------------------------------
@@ -504,12 +492,33 @@ Deprecations
 
 Deprecated Int64Index, UInt64Index & Float64Index
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
 :class:`Int64Index`, :class:`UInt64Index` and :class:`Float64Index` have been deprecated
-in favor of the new :class:`NumericIndex` and will be removed in Pandas 2.0 (:issue:`43028`).
+in favor of the base :class:`Index` class and will be removed in Pandas 2.0 (:issue:`43028`).
+
+For constructing a numeric index, you can use the base :class:`Index` class instead
+specifying the data type (which will also work on older pandas releases):
+
+.. code-block:: python
+
+    # replace
+    pd.Int64Index([1, 2, 3])
+    # with
+    pd.Index([1, 2, 3], dtype="int64")
+
+For checking the data type of an index object, you can replace ``isinstance``
+checks with checking the ``dtype``:
+
+.. code-block:: python
+
+    # replace
+    isinstance(idx, pd.Int64Index)
+    # with
+    idx.dtype == "int64"
 
 Currently, in order to maintain backward compatibility, calls to
 :class:`Index` will continue to return :class:`Int64Index`, :class:`UInt64Index` and :class:`Float64Index`
-when given numeric data, but in the future, a :class:`NumericIndex` will be returned.
+when given numeric data, but in the future, an :class:`Index` will be returned.
 
 *Current behavior*:
 
@@ -525,9 +534,9 @@ when given numeric data, but in the future, a :class:`NumericIndex` will be retu
 .. code-block:: ipython
 
     In [3]: pd.Index([1, 2, 3], dtype="int32")
-    Out [3]: NumericIndex([1, 2, 3], dtype='int32')
+    Out [3]: Index([1, 2, 3], dtype='int32')
     In [4]: pd.Index([1, 2, 3], dtype="uint64")
-    Out [4]: NumericIndex([1, 2, 3], dtype='uint64')
+    Out [4]: Index([1, 2, 3], dtype='uint64')
 
 
 .. _whatsnew_140.deprecations.frame_series_append:
@@ -603,6 +612,7 @@ Other Deprecations
 - Deprecated passing non boolean argument to sort in :func:`concat` (:issue:`41518`)
 - Deprecated passing arguments as positional for :func:`read_fwf` other than ``filepath_or_buffer`` (:issue:`41485`):
 - Deprecated passing ``skipna=None`` for :meth:`DataFrame.mad` and :meth:`Series.mad`, pass ``skipna=True`` instead (:issue:`44580`)
+- Deprecated the behavior of :func:`to_datetime` with the string "now" with ``utc=False``; in a future version this will match ``Timestamp("now")``, which in turn matches :meth:`Timestamp.now` returning the local time (:issue:`18705`)
 - Deprecated :meth:`DateOffset.apply`, use ``offset + other`` instead (:issue:`44522`)
 - Deprecated parameter ``names`` in :meth:`Index.copy` (:issue:`44916`)
 - A deprecation warning is now shown for :meth:`DataFrame.to_latex` indicating the arguments signature may change and emulate more the arguments to :meth:`.Styler.to_latex` in future versions (:issue:`44411`)
@@ -619,8 +629,10 @@ Other Deprecations
 - Deprecated ``numeric_only=None`` in :meth:`DataFrame.rank`; in a future version ``numeric_only`` must be either ``True`` or ``False`` (the default) (:issue:`45036`)
 - Deprecated the behavior of :meth:`Timestamp.utcfromtimestamp`, in the future it will return a timezone-aware UTC :class:`Timestamp` (:issue:`22451`)
 - Deprecated :meth:`NaT.freq` (:issue:`45071`)
+- Deprecated behavior of :class:`Series` and :class:`DataFrame` construction when passed float-dtype data containing ``NaN`` and an integer dtype ignoring the dtype argument; in a future version this will raise (:issue:`40110`)
 -
 
+
 .. ---------------------------------------------------------------------------
 
 .. _whatsnew_140.performance:
@@ -718,6 +730,8 @@ Timedelta
 ^^^^^^^^^
 - Bug in division of all-``NaT`` :class:`TimeDeltaIndex`, :class:`Series` or :class:`DataFrame` column with object-dtype arraylike of numbers failing to infer the result as timedelta64-dtype (:issue:`39750`)
 - Bug in floor division of ``timedelta64[ns]`` data with a scalar returning garbage values (:issue:`44466`)
+- Bug in :class:`Timedelta` now properly taking into account any nanoseconds contribution of any kwarg (:issue:`43764`)
+-
 
 Timezones
 ^^^^^^^^^
@@ -800,6 +814,7 @@ Indexing
 - Bug in :meth:`IntervalIndex.get_indexer_non_unique` not handling targets of ``dtype`` 'object' with NaNs correctly (:issue:`44482`)
 - Fixed regression where a single column ``np.matrix`` was no longer coerced to a 1d ``np.ndarray`` when added to a :class:`DataFrame` (:issue:`42376`)
 - Bug in :meth:`Series.__getitem__` with a :class:`CategoricalIndex` of integers treating lists of integers as positional indexers, inconsistent with the behavior with a single scalar integer (:issue:`15470`, :issue:`14865`)
+- Bug in :meth:`Series.__setitem__` when setting floats or integers into integer-dtype series failing to upcast when necessary to retain precision (:issue:`45121`)
 -
 
 Missing
@@ -870,6 +885,8 @@ Period
 - Bug in :meth:`PeriodIndex.to_timestamp` when the index has ``freq="B"`` inferring ``freq="D"`` for its result instead of ``freq="B"`` (:issue:`44105`)
 - Bug in :class:`Period` constructor incorrectly allowing ``np.timedelta64("NaT")`` (:issue:`44507`)
 - Bug in :meth:`PeriodIndex.to_timestamp` giving incorrect values for indexes with non-contiguous data (:issue:`44100`)
+- Bug in :meth:`Series.where` with ``PeriodDtype`` incorrectly raising when the ``where`` call should not replace anything (:issue:`45135`)
+
 -
 
 Plotting
@@ -899,6 +916,7 @@ Groupby/resample/rolling
 - Bug in :meth:`GroupBy.nth` failing on ``axis=1`` (:issue:`43926`)
 - Fixed bug in :meth:`Series.rolling` and :meth:`DataFrame.rolling` not respecting right bound on centered datetime-like windows, if the index contain duplicates (:issue:`3944`)
 - Bug in :meth:`Series.rolling` and :meth:`DataFrame.rolling` when using a :class:`pandas.api.indexers.BaseIndexer` subclass that returned unequal start and end arrays would segfault instead of raising a ``ValueError`` (:issue:`44470`)
+- Bug in :meth:`Groupby.nunique` not respecting ``observed=True`` for Categorical grouping columns (:issue:`45128`)
 - Bug in :meth:`GroupBy.head` and :meth:`GroupBy.tail` not dropping groups with ``NaN`` when ``dropna=True`` (:issue:`45089`)
 - Fixed bug in :meth:`GroupBy.__iter__` after selecting a subset of columns in a :class:`GroupBy` object, which returned all columns instead of the chosen subset (:issue:`#44821`)
 - Bug in :meth:`Groupby.rolling` when non-monotonic data passed, fails to correctly raise ``ValueError`` (:issue:`43909`)
@@ -924,6 +942,7 @@ Reshaping
 - Bug in :meth:`Series.unstack` with object doing unwanted type inference on resulting columns (:issue:`44595`)
 - Bug in :class:`MultiIndex` failing join operations with overlapping ``IntervalIndex`` levels (:issue:`44096`)
 - Bug in :meth:`DataFrame.replace` and :meth:`Series.replace` results is different ``dtype`` based on ``regex`` parameter (:issue:`44864`)
+- Bug in :meth:`DataFrame.pivot` with ``index=None`` when the :class:`DataFrame` index was a :class:`MultiIndex` (:issue:`23955`)
 
 Sparse
 ^^^^^^
@@ -940,6 +959,7 @@ ExtensionArray
 - Bug in :func:`array` failing to preserve :class:`PandasArray` (:issue:`43887`)
 - NumPy ufuncs ``np.abs``, ``np.positive``, ``np.negative`` now correctly preserve dtype when called on ExtensionArrays that implement ``__abs__, __pos__, __neg__``, respectively. In particular this is fixed for :class:`TimedeltaArray` (:issue:`43899`, :issue:`23316`)
 - NumPy ufuncs ``np.minimum.reduce`` ``np.maximum.reduce``, ``np.add.reduce``, and ``np.prod.reduce`` now work correctly instead of raising ``NotImplementedError`` on :class:`Series` with ``IntegerDtype`` or ``FloatDtype`` (:issue:`43923`, :issue:`44793`)
+- NumPy ufuncs with ``out`` keyword are now supported by arrays with ``IntegerDtype`` and ``FloatingDtype`` (:issue:`45122`)
 - Avoid raising ``PerformanceWarning`` about fragmented DataFrame when using many columns with an extension dtype (:issue:`44098`)
 - Bug in :class:`IntegerArray` and :class:`FloatingArray` construction incorrectly coercing mismatched NA values (e.g. ``np.timedelta64("NaT")``) to numeric NA (:issue:`44514`)
 - Bug in :meth:`BooleanArray.__eq__` and :meth:`BooleanArray.__ne__` raising ``TypeError`` on comparison with an incompatible type (like a string). This caused :meth:`DataFrame.replace` to sometimes raise a ``TypeError`` if a nullable boolean column was included (:issue:`44499`)

environment.yml

@@ -120,6 +120,9 @@ dependencies:
   - tabulate>=0.8.3  # DataFrame.to_markdown
   - natsort  # DataFrame.sort_values
   - pip:
-    - git+https://github.com/pydata/pydata-sphinx-theme.git@master
+      #issue with building environment in conda on windows. Issue: https://github.com/pandas-dev/pandas/issues/45123
+      #issue with pydata-sphix-theme on windows. Issue: https://github.com/pydata/pydata-sphinx-theme/issues/523
+      #using previous stable version as workaround
+    - git+https://github.com/pydata/pydata-sphinx-theme.git@41764f5
     - pandas-dev-flaker==0.2.0
     - pytest-cython

pandas/__init__.py

@@ -73,7 +73,6 @@
     Index,
     CategoricalIndex,
     RangeIndex,
-    NumericIndex,
     MultiIndex,
     IntervalIndex,
     TimedeltaIndex,
@@ -199,7 +198,7 @@ def __getattr__(name):
         warnings.warn(
             f"pandas.{name} is deprecated "
             "and will be removed from pandas in a future version. "
-            "Use pandas.NumericIndex with the appropriate dtype instead.",
+            "Use pandas.Index with the appropriate dtype instead.",
             FutureWarning,
             stacklevel=2,
         )
@@ -335,7 +334,6 @@ def __getattr__(name):
     "NA",
     "NaT",
     "NamedAgg",
-    "NumericIndex",
     "Period",
     "PeriodDtype",
     "PeriodIndex",

pandas/_libs/index.pyx

@@ -33,7 +33,9 @@ from pandas._libs import (
     hashtable as _hash,
 )
 
+from pandas._libs.lib cimport eq_NA_compat
 from pandas._libs.missing cimport (
+    C_NA as NA,
     checknull,
     is_matching_na,
 )
@@ -62,7 +64,7 @@ cdef ndarray _get_bool_indexer(ndarray values, object val):
     if values.descr.type_num == cnp.NPY_OBJECT:
         # i.e. values.dtype == object
         if not checknull(val):
-            indexer = values == val
+            indexer = eq_NA_compat(values, val)
 
         else:
             # We need to check for _matching_ NA values

pandas/_libs/lib.pxd

@@ -1 +1,6 @@
+from numpy cimport ndarray
+
+
 cdef bint c_is_list_like(object, bint) except -1
+
+cpdef ndarray eq_NA_compat(ndarray[object] arr, object key)

pandas/_libs/lib.pyx

@@ -3050,6 +3050,27 @@ def is_bool_list(obj: list) -> bool:
     return True
 
 
+cpdef ndarray eq_NA_compat(ndarray[object] arr, object key):
+    """
+    Check for `arr == key`, treating all values as not-equal to pd.NA.
+
+    key is assumed to have `not isna(key)`
+    """
+    cdef:
+        ndarray[uint8_t, cast=True] result = np.empty(len(arr), dtype=bool)
+        Py_ssize_t i
+        object item
+
+    for i in range(len(arr)):
+        item = arr[i]
+        if item is C_NA:
+            result[i] = False
+        else:
+            result[i] = item == key
+
+    return result
+
+
 def dtypes_all_equal(list types not None) -> bool:
     """
     Faster version for: