Merge remote-tracking branch 'upstream/master' into bisect

Author: simonjayhawkins

.github/workflows/ci.yml

@@ -153,12 +153,17 @@ jobs:
       run: |
         source activate pandas-dev
         pytest pandas/tests/frame/methods --array-manager
+        pytest pandas/tests/frame/test_reductions.py --array-manager
+        pytest pandas/tests/reductions/ --array-manager
+        pytest pandas/tests/generic/test_generic.py --array-manager
         pytest pandas/tests/arithmetic/ --array-manager
+        pytest pandas/tests/groupby/aggregate/ --array-manager
+        pytest pandas/tests/reshape/merge --array-manager
 
         # indexing subset (temporary since other tests don't pass yet)
         pytest pandas/tests/frame/indexing/test_indexing.py::TestDataFrameIndexing::test_setitem_boolean --array-manager
         pytest pandas/tests/frame/indexing/test_where.py --array-manager
-        pytest pandas/tests/frame/indexing/test_indexing.py::TestDataFrameIndexing::test_setitem_multi_index --array-manager
+        pytest pandas/tests/frame/indexing/test_setitem.py::TestDataFrameSetItem::test_setitem_multi_index --array-manager
         pytest pandas/tests/frame/indexing/test_setitem.py::TestDataFrameSetItem::test_setitem_listlike_indexer_duplicate_columns --array-manager
         pytest pandas/tests/indexing/multiindex/test_setitem.py::TestMultiIndexSetItem::test_astype_assignment_with_dups --array-manager
         pytest pandas/tests/indexing/multiindex/test_setitem.py::TestMultiIndexSetItem::test_frame_setitem_multi_column --array-manager

.pre-commit-config.yaml

@@ -163,6 +163,11 @@ repos:
         entry: np\.bool[^_8]
         language: pygrep
         types_or: [python, cython, rst]
+    -   id: np-object
+        name: Check for use of np.object instead of np.object_
+        entry: np\.object[^_8]
+        language: pygrep
+        types_or: [python, cython, rst]
     -   id: no-os-remove
         name: Check code for instances of os.remove
         entry: os\.remove

asv_bench/benchmarks/algos/isin.py

@@ -104,7 +104,7 @@ def time_isin(self, dtype, exponent, title):
 
 class IsinWithRandomFloat:
     params = [
-        [np.float64, np.object],
+        [np.float64, np.object_],
         [
             1_300,
             2_000,
@@ -134,7 +134,7 @@ def time_isin(self, dtype, size, title):
 
 class IsinWithArangeSorted:
     params = [
-        [np.float64, np.int64, np.uint64, np.object],
+        [np.float64, np.int64, np.uint64, np.object_],
         [
             1_000,
             2_000,
@@ -155,7 +155,7 @@ def time_isin(self, dtype, size):
 
 class IsinWithArange:
     params = [
-        [np.float64, np.int64, np.uint64, np.object],
+        [np.float64, np.int64, np.uint64, np.object_],
         [
             1_000,
             2_000,
@@ -273,6 +273,7 @@ class IsInLongSeriesLookUpDominates:
     def setup(self, dtype, MaxNumber, series_type):
         N = 10 ** 7
 
+        # https://github.com/pandas-dev/pandas/issues/39844
         if not np_version_under1p20 and dtype in ("Int64", "Float64"):
             raise NotImplementedError
 
@@ -303,6 +304,11 @@ class IsInLongSeriesValuesDominate:
 
     def setup(self, dtype, series_type):
         N = 10 ** 7
+
+        # https://github.com/pandas-dev/pandas/issues/39844
+        if not np_version_under1p20 and dtype in ("Int64", "Float64"):
+            raise NotImplementedError
+
         if series_type == "random":
             np.random.seed(42)
             vals = np.random.randint(0, 10 * N, N)

asv_bench/benchmarks/rolling.py

@@ -114,7 +114,7 @@ def time_ewm(self, constructor, window, dtype, method):
         getattr(self.ewm, method)()
 
     def time_ewm_times(self, constructor, window, dtype, method):
-        self.ewm.mean()
+        self.ewm_times.mean()
 
 
 class VariableWindowMethods(Methods):

doc/source/ecosystem.rst

@@ -156,7 +156,7 @@ A good implementation for Python users is `has2k1/plotnine <https://github.com/h
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 `IPython Vega <https://github.com/vega/ipyvega>`__ leverages `Vega
-<https://github.com/trifacta/vega>`__ to create plots within Jupyter Notebook.
+<https://github.com/vega/vega>`__ to create plots within Jupyter Notebook.
 
 `Plotly <https://plot.ly/python>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/getting_started/install.rst

@@ -255,47 +255,52 @@ For example, :func:`pandas.read_hdf` requires the ``pytables`` package, while
 optional dependency is not installed, pandas will raise an ``ImportError`` when
 the method requiring that dependency is called.
 
+Visualization
+^^^^^^^^^^^^^
+
 ========================= ================== =============================================================
 Dependency                Minimum Version    Notes
 ========================= ================== =============================================================
-BeautifulSoup4            4.6.0              HTML parser for read_html (see :ref:`note <optional_html>`)
+matplotlib                2.2.3              Plotting library
 Jinja2                    2.10               Conditional formatting with DataFrame.style
-PyQt4                                        Clipboard I/O
-PyQt5                                        Clipboard I/O
-PyTables                  3.5.1              HDF5-based reading / writing
-SQLAlchemy                1.3.0              SQL support for databases other than sqlite
+tabulate                  0.8.7              Printing in Markdown-friendly format (see `tabulate`_)
+========================= ================== =============================================================
+
+Computation
+^^^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
 SciPy                     1.12.0             Miscellaneous statistical functions
-xlsxwriter                1.0.2              Excel writing
-blosc                     1.17.0             Compression for HDF5
-fsspec                    0.7.4              Handling files aside from local and HTTP
-fastparquet               0.4.0              Parquet reading / writing
-gcsfs                     0.6.0              Google Cloud Storage access
-html5lib                  1.0.1              HTML parser for read_html (see :ref:`note <optional_html>`)
-lxml                      4.3.0              HTML parser for read_html (see :ref:`note <optional_html>`)
-matplotlib                2.2.3              Visualization
 numba                     0.46.0             Alternative execution engine for rolling operations
+                                             (see :ref:`Enhancing Performance <enhancingperf.numba>`)
+xarray                    0.12.3             pandas-like API for N-dimensional data
+========================= ================== =============================================================
+
+Excel files
+^^^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+xlrd                      1.2.0              Reading Excel
+xlwt                      1.3.0              Writing Excel
+xlsxwriter                1.0.2              Writing Excel
 openpyxl                  3.0.0              Reading / writing for xlsx files
-pandas-gbq                0.12.0             Google Big Query access
-psycopg2                  2.7                PostgreSQL engine for sqlalchemy
-pyarrow                   0.15.0             Parquet, ORC, and feather reading / writing
-pymysql                   0.8.1              MySQL engine for sqlalchemy
-pyreadstat                                   SPSS files (.sav) reading
 pyxlsb                    1.0.6              Reading for xlsb files
-qtpy                                         Clipboard I/O
-s3fs                      0.4.0              Amazon S3 access
-tabulate                  0.8.7              Printing in Markdown-friendly format (see `tabulate`_)
-xarray                    0.12.3             pandas-like API for N-dimensional data
-xclip                                        Clipboard I/O on linux
-xlrd                      1.2.0              Excel reading
-xlwt                      1.3.0              Excel writing
-xsel                                         Clipboard I/O on linux
-zlib                                         Compression for HDF5
 ========================= ================== =============================================================
 
-.. _optional_html:
+HTML
+^^^^
 
-Optional dependencies for parsing HTML
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+BeautifulSoup4            4.6.0              HTML parser for read_html
+html5lib                  1.0.1              HTML parser for read_html
+lxml                      4.3.0              HTML parser for read_html
+========================= ================== =============================================================
 
 One of the following combinations of libraries is needed to use the
 top-level :func:`~pandas.read_html` function:
@@ -320,3 +325,52 @@ top-level :func:`~pandas.read_html` function:
 .. _BeautifulSoup4: https://www.crummy.com/software/BeautifulSoup
 .. _lxml: https://lxml.de
 .. _tabulate: https://github.com/astanin/python-tabulate
+
+SQL databases
+^^^^^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+SQLAlchemy                1.3.0              SQL support for databases other than sqlite
+psycopg2                  2.7                PostgreSQL engine for sqlalchemy
+pymysql                   0.8.1              MySQL engine for sqlalchemy
+========================= ================== =============================================================
+
+Other data sources
+^^^^^^^^^^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+PyTables                  3.5.1              HDF5-based reading / writing
+blosc                     1.17.0             Compression for HDF5
+zlib                                         Compression for HDF5
+fastparquet               0.4.0              Parquet reading / writing
+pyarrow                   0.15.0             Parquet, ORC, and feather reading / writing
+pyreadstat                                   SPSS files (.sav) reading
+========================= ================== =============================================================
+
+Access data in the cloud
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+fsspec                    0.7.4              Handling files aside from simple local and HTTP
+gcsfs                     0.6.0              Google Cloud Storage access
+pandas-gbq                0.12.0             Google Big Query access
+s3fs                      0.4.0              Amazon S3 access
+========================= ================== =============================================================
+
+Clipboard
+^^^^^^^^^
+
+========================= ================== =============================================================
+Dependency                Minimum Version    Notes
+========================= ================== =============================================================
+PyQt4/PyQt5                                  Clipboard I/O
+qtpy                                         Clipboard I/O
+xclip                                        Clipboard I/O on linux
+xsel                                         Clipboard I/O on linux
+========================= ================== =============================================================

doc/source/user_guide/cookbook.rst

@@ -1410,7 +1410,7 @@ Often it's useful to obtain the lower (or upper) triangular form of a correlatio
 
     corr_mat.where(mask)
 
-The ``method`` argument within ``DataFrame.corr`` can accept a callable in addition to the named correlation types.  Here we compute the ``distance correlation <https://en.wikipedia.org/wiki/Distance_correlation>``__ matrix for a ``DataFrame`` object.
+The ``method`` argument within ``DataFrame.corr`` can accept a callable in addition to the named correlation types.  Here we compute the `distance correlation <https://en.wikipedia.org/wiki/Distance_correlation>`__ matrix for a ``DataFrame`` object.
 
 .. ipython:: python
 

doc/source/user_guide/io.rst

@@ -29,7 +29,7 @@ The pandas I/O API is a set of top level ``reader`` functions accessed like
     binary;`Feather Format <https://github.com/wesm/feather>`__;:ref:`read_feather<io.feather>`;:ref:`to_feather<io.feather>`
     binary;`Parquet Format <https://parquet.apache.org/>`__;:ref:`read_parquet<io.parquet>`;:ref:`to_parquet<io.parquet>`
     binary;`ORC Format <https://orc.apache.org/>`__;:ref:`read_orc<io.orc>`;
-    binary;`Msgpack <https://msgpack.org/index.html>`__;:ref:`read_msgpack<io.msgpack>`;:ref:`to_msgpack<io.msgpack>`
+    binary;`Msgpack <https://msgpack.org/>`__;:ref:`read_msgpack<io.msgpack>`;:ref:`to_msgpack<io.msgpack>`
     binary;`Stata <https://en.wikipedia.org/wiki/Stata>`__;:ref:`read_stata<io.stata_reader>`;:ref:`to_stata<io.stata_writer>`
     binary;`SAS <https://en.wikipedia.org/wiki/SAS_(software)>`__;:ref:`read_sas<io.sas_reader>`;
     binary;`SPSS <https://en.wikipedia.org/wiki/SPSS>`__;:ref:`read_spss<io.spss_reader>`;

doc/source/user_guide/style.ipynb

@@ -1154,7 +1154,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "from IPython.html import widgets\n",
+    "from ipywidgets import widgets\n",
     "@widgets.interact\n",
     "def f(h_neg=(0, 359, 1), h_pos=(0, 359), s=(0., 99.9), l=(0., 99.9)):\n",
     "    return df.style.background_gradient(\n",

doc/source/whatsnew/v0.24.0.rst

@@ -1755,8 +1755,8 @@ Missing
 
 - Bug in :func:`DataFrame.fillna` where a ``ValueError`` would raise when one column contained a ``datetime64[ns, tz]`` dtype (:issue:`15522`)
 - Bug in :func:`Series.hasnans` that could be incorrectly cached and return incorrect answers if null elements are introduced after an initial call (:issue:`19700`)
-- :func:`Series.isin` now treats all NaN-floats as equal also for ``np.object``-dtype. This behavior is consistent with the behavior for float64 (:issue:`22119`)
-- :func:`unique` no longer mangles NaN-floats and the ``NaT``-object for ``np.object``-dtype, i.e. ``NaT`` is no longer coerced to a NaN-value and is treated as a different entity. (:issue:`22295`)
+- :func:`Series.isin` now treats all NaN-floats as equal also for ``np.object_``-dtype. This behavior is consistent with the behavior for float64 (:issue:`22119`)
+- :func:`unique` no longer mangles NaN-floats and the ``NaT``-object for ``np.object_``-dtype, i.e. ``NaT`` is no longer coerced to a NaN-value and is treated as a different entity. (:issue:`22295`)
 - :class:`DataFrame` and :class:`Series` now properly handle numpy masked arrays with hardened masks. Previously, constructing a DataFrame or Series from a masked array with a hard mask would create a pandas object containing the underlying value, rather than the expected NaN. (:issue:`24574`)
 - Bug in :class:`DataFrame` constructor where ``dtype`` argument was not honored when handling numpy masked record arrays. (:issue:`24874`)
 

doc/source/whatsnew/v1.2.3.rst

@@ -15,8 +15,15 @@ including other versions of pandas.
 Fixed regressions
 ~~~~~~~~~~~~~~~~~
 
-- Fixed regression in :func:`pandas.to_excel` raising ``KeyError`` when giving duplicate columns with ``columns`` attribute (:issue:`39695`)
--
+- Fixed regression in :meth:`~DataFrame.to_excel` raising ``KeyError`` when giving duplicate columns with ``columns`` attribute (:issue:`39695`)
+- Fixed regression in nullable integer unary ops propagating mask on assignment (:issue:`39943`)
+- Fixed regression in :meth:`DataFrame.__setitem__` not aligning :class:`DataFrame` on right-hand side for boolean indexer (:issue:`39931`)
+- Fixed regression in :meth:`~DataFrame.to_json` failing to use ``compression`` with URL-like paths that are internally opened in binary mode or with user-provided file objects that are opened in binary mode (:issue:`39985`)
+- Fixed regression in :meth:`~Series.sort_index` and :meth:`~DataFrame.sort_index`,
+  which exited with an ungraceful error when having kwarg ``ascending=None`` passed (:issue:`39434`).
+  Passing ``ascending=None`` is still considered invalid,
+  and the new error message suggests a proper usage
+  (``ascending`` must be a boolean or a list-like boolean).
 
 .. ---------------------------------------------------------------------------