Merge remote-tracking branch 'upstream/master' into pandas-devgh-16454

Author: dwhu

.pre-commit-config.yaml

@@ -11,7 +11,7 @@ repos:
         language: python_venv
         additional_dependencies: [flake8-comprehensions>=3.1.0]
 -   repo: https://github.com/pre-commit/mirrors-isort
-    rev: v4.3.20
+    rev: v4.3.21
     hooks:
     -   id: isort
         language: python_venv

asv_bench/benchmarks/indexing.py

@@ -131,6 +131,7 @@ def setup(self):
         self.col_scalar = columns[10]
         self.bool_indexer = self.df[self.col_scalar] > 0
         self.bool_obj_indexer = self.bool_indexer.astype(object)
+        self.boolean_indexer = (self.df[self.col_scalar] > 0).astype("boolean")
 
     def time_loc(self):
         self.df.loc[self.idx_scalar, self.col_scalar]
@@ -144,6 +145,9 @@ def time_boolean_rows(self):
     def time_boolean_rows_object(self):
         self.df[self.bool_obj_indexer]
 
+    def time_boolean_rows_boolean(self):
+        self.df[self.boolean_indexer]
+
 
 class DataFrameNumericIndexing:
     def setup(self):

ci/deps/travis-36-cov.yaml

@@ -30,8 +30,6 @@ dependencies:
   - openpyxl<=3.0.1
   # https://github.com/pandas-dev/pandas/pull/30009 openpyxl 3.0.2 broke
   - pandas-gbq
-  # https://github.com/pydata/pandas-gbq/issues/271
-  - google-cloud-bigquery<=1.11
   - psycopg2
   - pyarrow>=0.12.0
   - pymysql

doc/source/development/code_style.rst

@@ -0,0 +1,129 @@
+.. _code_style:
+
+{{ header }}
+
+=======================
+pandas code style guide
+=======================
+
+.. contents:: Table of contents:
+   :local:
+
+Patterns
+========
+
+foo.__class__
+-------------
+
+*pandas* uses 'type(foo)' instead 'foo.__class__' as it is making the code more
+readable.
+
+For example:
+
+**Good:**
+
+.. code-block:: python
+
+    foo = "bar"
+    type(foo)
+
+**Bad:**
+
+.. code-block:: python
+
+    foo = "bar"
+    foo.__class__
+
+
+String formatting
+=================
+
+Concatenated strings
+--------------------
+
+f-strings
+~~~~~~~~~
+
+*pandas* uses f-strings formatting instead of '%' and '.format()' string formatters.
+
+The convention of using f-strings on a string that is concatenated over serveral lines,
+is to prefix only the lines containing the value needs to be interpeted.
+
+For example:
+
+**Good:**
+
+.. code-block:: python
+
+    foo = "old_function"
+    bar = "new_function"
+
+    my_warning_message = (
+        f"Warning, {foo} is deprecated, "
+        "please use the new and way better "
+        f"{bar}"
+    )
+
+**Bad:**
+
+.. code-block:: python
+
+    foo = "old_function"
+    bar = "new_function"
+
+    my_warning_message = (
+        f"Warning, {foo} is deprecated, "
+        f"please use the new and way better "
+        f"{bar}"
+    )
+
+White spaces
+~~~~~~~~~~~~
+
+Putting the white space only at the end of the previous line, so
+there is no whitespace at the beggining of the concatenated string.
+
+For example:
+
+**Good:**
+
+.. code-block:: python
+
+    example_string = (
+        "Some long concatenated string, "
+        "with good placement of the "
+        "whitespaces"
+    )
+
+**Bad:**
+
+.. code-block:: python
+
+    example_string = (
+        "Some long concatenated string,"
+        " with bad placement of the"
+        " whitespaces"
+    )
+
+Representation function (aka 'repr()')
+--------------------------------------
+
+*pandas* uses 'repr()' instead of '%r' and '!r'.
+
+The use of 'repr()' will only happend when the value is not an obvious string.
+
+For example:
+
+**Good:**
+
+.. code-block:: python
+
+    value = str
+    f"Unknown recived value, got: {repr(value)}"
+
+**Good:**
+
+.. code-block:: python
+
+    value = str
+    f"Unknown recived type, got: '{type(value).__name__}'"

doc/source/development/contributing.rst

@@ -569,8 +569,7 @@ do not make sudden changes to the code that could have the potential to break
 a lot of user code as a result, that is, we need it to be as *backwards compatible*
 as possible to avoid mass breakages.
 
-Additional standards are outlined on the `code style wiki
-page <https://github.com/pandas-dev/pandas/wiki/Code-Style-and-Conventions>`_.
+Additional standards are outlined on the `pandas code style guide <code_style>`_
 
 Optional dependencies
 ---------------------

doc/source/development/index.rst

@@ -13,6 +13,7 @@ Development
     :maxdepth: 2
 
     contributing
+    code_style
     maintaining
     internals
     extending

doc/source/ecosystem.rst

@@ -327,6 +327,21 @@ PyTables, h5py, and pymongo to move data between non pandas formats. Its graph
 based approach is also extensible by end users for custom formats that may be
 too specific for the core of odo.
 
+`Pandarallel <https://github.com/nalepae/pandarallel>`__
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Pandarallel provides a simple way to parallelize your pandas operations on all your CPUs by changing only one line of code.
+If also displays progress bars.
+
+.. code:: python
+
+    from pandarallel import pandarallel
+
+    pandarallel.initialize(progress_bar=True)
+
+    # df.apply(func)
+    df.parallel_apply(func)
+
 `Ray <https://ray.readthedocs.io/en/latest/pandas_on_ray.html>`__
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -380,4 +395,3 @@ Library        Accessor   Classes
 
 .. _cyberpandas: https://cyberpandas.readthedocs.io/en/latest
 .. _pdvega: https://altair-viz.github.io/pdvega/
-

doc/source/getting_started/basics.rst

@@ -1937,21 +1937,36 @@ See :ref:`extending.extension-types` for how to write your own extension that
 works with pandas. See :ref:`ecosystem.extensions` for a list of third-party
 libraries that have implemented an extension.
 
-The following table lists all of pandas extension types. See the respective
+The following table lists all of pandas extension types. For methods requiring ``dtype``
+arguments, strings can be specified as indicated. See the respective
 documentation sections for more on each type.
 
-=================== ========================= ================== ============================= =============================
-Kind of Data        Data Type                 Scalar             Array                         Documentation
-=================== ========================= ================== ============================= =============================
-tz-aware datetime   :class:`DatetimeTZDtype`  :class:`Timestamp` :class:`arrays.DatetimeArray` :ref:`timeseries.timezone`
-Categorical         :class:`CategoricalDtype` (none)             :class:`Categorical`          :ref:`categorical`
-period (time spans) :class:`PeriodDtype`      :class:`Period`    :class:`arrays.PeriodArray`   :ref:`timeseries.periods`
-sparse              :class:`SparseDtype`      (none)             :class:`arrays.SparseArray`   :ref:`sparse`
-intervals           :class:`IntervalDtype`    :class:`Interval`  :class:`arrays.IntervalArray` :ref:`advanced.intervalindex`
-nullable integer    :class:`Int64Dtype`, ...  (none)             :class:`arrays.IntegerArray`  :ref:`integer_na`
-Strings             :class:`StringDtype`      :class:`str`       :class:`arrays.StringArray`   :ref:`text`
-Boolean (with NA)   :class:`BooleanDtype`     :class:`bool`      :class:`arrays.BooleanArray`  :ref:`api.arrays.bool`
-=================== ========================= ================== ============================= =============================
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| Kind of Data      | Data Type                 | Scalar             | Array                         | String Aliases                          | Documentation                 |
++===================+===========================+====================+===============================+=========================================+===============================+
+| tz-aware datetime | :class:`DatetimeTZDtype`  | :class:`Timestamp` | :class:`arrays.DatetimeArray` | ``'datetime64[ns, <tz>]'``              | :ref:`timeseries.timezone`    |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| Categorical       | :class:`CategoricalDtype` | (none)             | :class:`Categorical`          | ``'category'``                          | :ref:`categorical`            |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| period            | :class:`PeriodDtype`      | :class:`Period`    | :class:`arrays.PeriodArray`   | ``'period[<freq>]'``,                   | :ref:`timeseries.periods`     |
+| (time spans)      |                           |                    |                               | ``'Period[<freq>]'``                    |                               |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| sparse            | :class:`SparseDtype`      | (none)             | :class:`SparseArray`          | ``'Sparse'``, ``'Sparse[int]'``,        | :ref:`sparse`                 |
+|                   |                           |                    |                               | ``'Sparse[float]'``                     |                               |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| intervals         | :class:`IntervalDtype`    | :class:`Interval`  | :class:`arrays.IntervalArray` | ``'interval'``, ``'Interval'``,         | :ref:`advanced.intervalindex` |
+|                   |                           |                    |                               | ``'Interval[<numpy_dtype>]'``,          |                               |
+|                   |                           |                    |                               | ``'Interval[datetime64[ns, <tz>]]'``,   |                               |
+|                   |                           |                    |                               | ``'Interval[timedelta64[<freq>]]'``     |                               |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| nullable integer  + :class:`Int64Dtype`, ...  | (none)             | :class:`arrays.IntegerArray`  | ``'Int8'``, ``'Int16'``, ``'Int32'``,   | :ref:`integer_na`             |
+|                   |                           |                    |                               | ``'Int64'``, ``'UInt8'``, ``'UInt16'``, |                               |
+|                   |                           |                    |                               | ``'UInt32'``, ``'UInt64'``              |                               |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| Strings           | :class:`StringDtype`      | :class:`str`       | :class:`arrays.StringArray`   | ``'string'``                            | :ref:`text`                   |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
+| Boolean (with NA) | :class:`BooleanDtype`     | :class:`bool`      | :class:`arrays.BooleanArray`  | ``'boolean'``                           | :ref:`api.arrays.bool`        |
++-------------------+---------------------------+--------------------+-------------------------------+-----------------------------------------+-------------------------------+
 
 Pandas has two ways to store strings.
 

doc/source/getting_started/comparison/comparison_with_sas.rst

@@ -629,7 +629,7 @@ for more details and examples.
 
 .. ipython:: python
 
-   tips_summed = tips.groupby(['sex', 'smoker'])['total_bill', 'tip'].sum()
+   tips_summed = tips.groupby(['sex', 'smoker'])[['total_bill', 'tip']].sum()
    tips_summed.head()
 
 

doc/source/getting_started/comparison/comparison_with_stata.rst

@@ -617,7 +617,7 @@ for more details and examples.
 
 .. ipython:: python
 
-   tips_summed = tips.groupby(['sex', 'smoker'])['total_bill', 'tip'].sum()
+   tips_summed = tips.groupby(['sex', 'smoker'])[['total_bill', 'tip']].sum()
    tips_summed.head()
 
 

doc/source/index.rst.template

@@ -109,6 +109,7 @@ See the :ref:`overview` for more detail about what's in the library.
 * :doc:`development/index`
 
   * :doc:`development/contributing`
+  * :doc:`development/code_style`
   * :doc:`development/internals`
   * :doc:`development/extending`
   * :doc:`development/developer`

doc/source/reference/arrays.rst

@@ -12,7 +12,8 @@ For most data types, pandas uses NumPy arrays as the concrete
 objects contained with a :class:`Index`, :class:`Series`, or
 :class:`DataFrame`.
 
-For some data types, pandas extends NumPy's type system.
+For some data types, pandas extends NumPy's type system. String aliases for these types
+can be found at :ref:`basics.dtypes`.
 
 =================== ========================= ================== =============================
 Kind of Data        Pandas Data Type          Scalar             Array

doc/source/reference/extensions.rst

@@ -59,3 +59,11 @@ objects.
       api.extensions.ExtensionArray.nbytes
       api.extensions.ExtensionArray.ndim
       api.extensions.ExtensionArray.shape
+
+Additionally, we have some utility methods for ensuring your object
+behaves correctly.
+
+.. autosummary::
+  :toctree: api/
+
+  api.indexers.check_bool_array_indexer

doc/source/user_guide/advanced.rst

@@ -565,19 +565,15 @@ When working with an ``Index`` object directly, rather than via a ``DataFrame``,
    mi2 = mi.rename("new name", level=0)
    mi2
 
-.. warning::
 
-   Prior to pandas 1.0.0, you could also set the names of a ``MultiIndex``
-   by updating the name of a level.
+You cannot set the names of the MultiIndex via a level.
 
-   .. code-block:: none
+.. ipython:: python
+   :okexcept:
 
-      >>> mi.levels[0].name = 'name via level'
-      >>> mi.names[0]  # only works for older pandas
-      'name via level'
+   mi.levels[0].name = "name via level"
 
-   As of pandas 1.0, this will *silently* fail to update the names
-   of the MultiIndex. Use :meth:`Index.set_names` instead.
+Use :meth:`Index.set_names` instead.
 
 Sorting a ``MultiIndex``
 ------------------------

doc/source/user_guide/boolean.rst

@@ -14,6 +14,29 @@ Nullable Boolean Data Type
 
 .. versionadded:: 1.0.0
 
+
+.. _boolean.indexing:
+
+Indexing with NA values
+-----------------------
+
+pandas does not allow indexing with NA values. Attempting to do so
+will raise a ``ValueError``.
+
+.. ipython:: python
+   :okexcept:
+
+   s = pd.Series([1, 2, 3])
+   mask = pd.array([True, False, pd.NA], dtype="boolean")
+   s[mask]
+
+The missing values will need to be explicitly filled with True or False prior
+to using the array as a mask.
+
+.. ipython:: python
+
+   s[mask.fillna(False)]
+
 .. _boolean.kleene:
 
 Kleene Logical Operations