Merge remote-tracking branch 'upstream/master' into docfix-multiindex-set_levels

Author: hweecat

ci/azure/posix.yml

@@ -9,17 +9,16 @@ jobs:
   strategy:
     matrix:
       ${{ if eq(parameters.name, 'macOS') }}:
-        py35_macos:
-          ENV_FILE: ci/deps/azure-macos-35.yaml
-          CONDA_PY: "35"
+        py36_macos:
+          ENV_FILE: ci/deps/azure-macos-36.yaml
+          CONDA_PY: "36"
           PATTERN: "not slow and not network"
 
       ${{ if eq(parameters.name, 'Linux') }}:
-        py35_compat:
-          ENV_FILE: ci/deps/azure-35-compat.yaml
-          CONDA_PY: "35"
+        py36_minimum_versions:
+          ENV_FILE: ci/deps/azure-36-minimum_versions.yaml
+          CONDA_PY: "36"
           PATTERN: "not slow and not network"
-
         py36_locale_slow_old_np:
           ENV_FILE: ci/deps/azure-36-locale.yaml
           CONDA_PY: "36"

ci/deps/azure-35-compat.yaml renamed to ci/deps/azure-36-minimum_versions.yaml

@@ -5,26 +5,23 @@ channels:
 dependencies:
   - beautifulsoup4=4.6.0
   - bottleneck=1.2.1
+  - cython>=0.29.13
   - jinja2=2.8
   - numexpr=2.6.2
   - numpy=1.13.3
   - openpyxl=2.4.8
   - pytables=3.4.2
   - python-dateutil=2.6.1
-  - python=3.5.3
+  - python=3.6.1
   - pytz=2017.2
   - scipy=0.19.0
   - xlrd=1.1.0
   - xlsxwriter=0.9.8
   - xlwt=1.2.0
   # universal
+  - html5lib=1.0.1
   - hypothesis>=3.58.0
+  - pytest=4.5.0
   - pytest-xdist
   - pytest-mock
   - pytest-azurepipelines
-  - pip
-  - pip:
-    # for python 3.5, pytest>=4.0.2, cython>=0.29.13 is not available in conda
-    - cython>=0.29.13
-    - pytest==4.5.0
-    - html5lib==1.0b2

ci/deps/azure-macos-35.yaml renamed to ci/deps/azure-macos-36.yaml

@@ -14,7 +14,7 @@ dependencies:
   - openpyxl
   - pyarrow
   - pytables
-  - python=3.5.*
+  - python=3.6.*
   - python-dateutil==2.6.1
   - pytz
   - xarray

doc/source/development/contributing.rst

@@ -236,7 +236,7 @@ Creating a Python environment (pip)
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you aren't using conda for your development environment, follow these instructions.
-You'll need to have at least python3.5 installed on your system.
+You'll need to have at least Python 3.6.1 installed on your system.
 
 **Unix**/**Mac OS**
 
@@ -847,29 +847,6 @@ The limitation here is that while a human can reasonably understand that ``is_nu
 
 With custom types and inference this is not always possible so exceptions are made, but every effort should be exhausted to avoid ``cast`` before going down such paths.
 
-Syntax Requirements
-~~~~~~~~~~~~~~~~~~~
-
-Because *pandas* still supports Python 3.5, :pep:`526` does not apply and variables **must** be annotated with type comments. Specifically, this is a valid annotation within pandas:
-
-.. code-block:: python
-
-   primes = []  # type: List[int]
-
-Whereas this is **NOT** allowed:
-
-.. code-block:: python
-
-   primes: List[int] = []  # not supported in Python 3.5!
-
-Note that function signatures can always be annotated per :pep:`3107`:
-
-.. code-block:: python
-
-   def sum_of_primes(primes: List[int] = []) -> int:
-       ...
-
-
 Pandas-specific Types
 ~~~~~~~~~~~~~~~~~~~~~
 
@@ -1296,7 +1273,7 @@ environment by::
 
 or, to use a specific Python interpreter,::
 
-    asv run -e -E existing:python3.5
+    asv run -e -E existing:python3.6
 
 This will display stderr from the benchmarks, and use your local
 ``python`` that comes from your ``$PATH``.

doc/source/development/policies.rst

@@ -51,7 +51,7 @@ Pandas may change the behavior of experimental features at any time.
 Python Support
 ~~~~~~~~~~~~~~
 
-Pandas will only drop support for specific Python versions (e.g. 3.5.x, 3.6.x) in
+Pandas will only drop support for specific Python versions (e.g. 3.6.x, 3.7.x) in
 pandas **major** releases.
 
 .. _SemVer: https://semver.org

doc/source/getting_started/dsintro.rst

@@ -564,53 +564,6 @@ to a column created earlier in the same :meth:`~DataFrame.assign`.
 In the second expression, ``x['C']`` will refer to the newly created column,
 that's equal to ``dfa['A'] + dfa['B']``.
 
-To write code compatible with all versions of Python, split the assignment in two.
-
-.. ipython:: python
-
-   dependent = pd.DataFrame({"A": [1, 1, 1]})
-   (dependent.assign(A=lambda x: x['A'] + 1)
-             .assign(B=lambda x: x['A'] + 2))
-
-.. warning::
-
-   Dependent assignment may subtly change the behavior of your code between
-   Python 3.6 and older versions of Python.
-
-   If you wish to write code that supports versions of python before and after 3.6,
-   you'll need to take care when passing ``assign`` expressions that
-
-   * Update an existing column
-   * Refer to the newly updated column in the same ``assign``
-
-   For example, we'll update column "A" and then refer to it when creating "B".
-
-   .. code-block:: python
-
-      >>> dependent = pd.DataFrame({"A": [1, 1, 1]})
-      >>> dependent.assign(A=lambda x: x["A"] + 1, B=lambda x: x["A"] + 2)
-
-   For Python 3.5 and earlier the expression creating ``B`` refers to the
-   "old" value of ``A``, ``[1, 1, 1]``. The output is then
-
-   .. code-block:: console
-
-         A  B
-      0  2  3
-      1  2  3
-      2  2  3
-
-   For Python 3.6 and later, the expression creating ``A`` refers to the
-   "new" value of ``A``, ``[2, 2, 2]``, which results in
-
-   .. code-block:: console
-
-         A  B
-      0  2  4
-      1  2  4
-      2  2  4
-
-
 
 Indexing / selection
 ~~~~~~~~~~~~~~~~~~~~

doc/source/getting_started/install.rst

@@ -18,7 +18,7 @@ Instructions for installing from source,
 Python version support
 ----------------------
 
-Officially Python 3.5.3 and above, 3.6, 3.7, and 3.8.
+Officially Python 3.6.1 and above, 3.7, and 3.8.
 
 Installing pandas
 -----------------
@@ -140,7 +140,7 @@ Installing with ActivePython
 Installation instructions for
 `ActivePython <https://www.activestate.com/activepython>`__ can be found
 `here <https://www.activestate.com/activepython/downloads>`__. Versions
-2.7 and 3.5 include pandas.
+2.7, 3.5 and 3.6 include pandas.
 
 Installing using your Linux distribution's package manager.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

doc/source/reference/window.rst

@@ -34,6 +34,8 @@ Standard moving window functions
    Rolling.quantile
    Window.mean
    Window.sum
+   Window.var
+   Window.std
 
 .. _api.functions_expanding:
 

doc/source/user_guide/categorical.rst

@@ -797,37 +797,52 @@ Assigning a ``Categorical`` to parts of a column of other types will use the val
     df.dtypes
 
 .. _categorical.merge:
+.. _categorical.concat:
 
-Merging
-~~~~~~~
+Merging / Concatenation
+~~~~~~~~~~~~~~~~~~~~~~~
 
-You can concat two ``DataFrames`` containing categorical data together,
-but the categories of these categoricals need to be the same:
+By default, combining ``Series`` or ``DataFrames`` which contain the same
+categories results in ``category`` dtype, otherwise results will depend on the
+dtype of the underlying categories. Merges that result in non-categorical
+dtypes will likely have higher memory usage. Use ``.astype`` or
+``union_categoricals`` to ensure ``category`` results.
 
 .. ipython:: python
 
-    cat = pd.Series(["a", "b"], dtype="category")
-    vals = [1, 2]
-    df = pd.DataFrame({"cats": cat, "vals": vals})
-    res = pd.concat([df, df])
-    res
-    res.dtypes
+   from pandas.api.types import union_categoricals
 
-In this case the categories are not the same, and therefore an error is raised:
+   # same categories
+   s1 = pd.Series(['a', 'b'], dtype='category')
+   s2 = pd.Series(['a', 'b', 'a'], dtype='category')
+   pd.concat([s1, s2])
 
-.. ipython:: python
+   # different categories
+   s3 = pd.Series(['b', 'c'], dtype='category')
+   pd.concat([s1, s3])
 
-    df_different = df.copy()
-    df_different["cats"].cat.categories = ["c", "d"]
-    try:
-        pd.concat([df, df_different])
-    except ValueError as e:
-        print("ValueError:", str(e))
+   # Output dtype is inferred based on categories values
+   int_cats = pd.Series([1, 2], dtype="category")
+   float_cats = pd.Series([3.0, 4.0], dtype="category")
+   pd.concat([int_cats, float_cats])
+
+   pd.concat([s1, s3]).astype('category')
+   union_categoricals([s1.array, s3.array])
 
-The same applies to ``df.append(df_different)``.
+The following table summarizes the results of merging ``Categoricals``:
 
-See also the section on :ref:`merge dtypes<merging.dtypes>` for notes about preserving merge dtypes and performance.
++-------------------+------------------------+----------------------+-----------------------------+
+| arg1              | arg2                   |      identical       | result                      |
++===================+========================+======================+=============================+
+| category          | category               | True                 | category                    |
++-------------------+------------------------+----------------------+-----------------------------+
+| category (object) | category (object)      | False                | object (dtype is inferred)  |
++-------------------+------------------------+----------------------+-----------------------------+
+| category (int)    | category (float)       | False                | float (dtype is inferred)   |
++-------------------+------------------------+----------------------+-----------------------------+
 
+See also the section on :ref:`merge dtypes<merging.dtypes>` for notes about
+preserving merge dtypes and performance.
 
 .. _categorical.union:
 
@@ -918,46 +933,6 @@ the resulting array will always be a plain ``Categorical``:
       # "b" is coded to 0 throughout, same as c1, different from c2
       c.codes
 
-.. _categorical.concat:
-
-Concatenation
-~~~~~~~~~~~~~
-
-This section describes concatenations specific to ``category`` dtype. See :ref:`Concatenating objects<merging.concat>` for general description.
-
-By default, ``Series`` or ``DataFrame`` concatenation which contains the same categories
-results in ``category`` dtype, otherwise results in ``object`` dtype.
-Use ``.astype`` or ``union_categoricals`` to get ``category`` result.
-
-.. ipython:: python
-
-   # same categories
-   s1 = pd.Series(['a', 'b'], dtype='category')
-   s2 = pd.Series(['a', 'b', 'a'], dtype='category')
-   pd.concat([s1, s2])
-
-   # different categories
-   s3 = pd.Series(['b', 'c'], dtype='category')
-   pd.concat([s1, s3])
-
-   pd.concat([s1, s3]).astype('category')
-   union_categoricals([s1.array, s3.array])
-
-
-Following table summarizes the results of ``Categoricals`` related concatenations.
-
-+----------+--------------------------------------------------------+----------------------------+
-| arg1     | arg2                                                   | result                     |
-+==========+========================================================+============================+
-| category | category (identical categories)                        | category                   |
-+----------+--------------------------------------------------------+----------------------------+
-| category | category (different categories, both not ordered)      | object (dtype is inferred) |
-+----------+--------------------------------------------------------+----------------------------+
-| category | category (different categories, either one is ordered) | object (dtype is inferred) |
-+----------+--------------------------------------------------------+----------------------------+
-| category | not category                                           | object (dtype is inferred) |
-+----------+--------------------------------------------------------+----------------------------+
-
 
 Getting data in/out
 -------------------