Merge branch 'master' into cleanup/matplotlib-style

Author: ivanovmg

.pre-commit-config.yaml

@@ -71,3 +71,5 @@ repos:
     hooks:
     -   id: end-of-file-fixer
         exclude: ^LICENSES/|\.(html|csv|txt|svg|py)$
+    -   id: trailing-whitespace
+        exclude: \.(html|svg)$

asv_bench/benchmarks/groupby.py

@@ -358,6 +358,26 @@ def time_category_size(self):
         self.draws.groupby(self.cats).size()
 
 
+class FillNA:
+    def setup(self):
+        N = 100
+        self.df = DataFrame(
+            {"group": [1] * N + [2] * N, "value": [np.nan, 1.0] * N}
+        ).set_index("group")
+
+    def time_df_ffill(self):
+        self.df.groupby("group").fillna(method="ffill")
+
+    def time_df_bfill(self):
+        self.df.groupby("group").fillna(method="bfill")
+
+    def time_srs_ffill(self):
+        self.df.groupby("group")["value"].fillna(method="ffill")
+
+    def time_srs_bfill(self):
+        self.df.groupby("group")["value"].fillna(method="bfill")
+
+
 class GroupByMethods:
 
     param_names = ["dtype", "method", "application"]

doc/source/development/contributing.rst

@@ -206,7 +206,7 @@ You will need `Build Tools for Visual Studio 2017
 	scrolling down to "All downloads" -> "Tools for Visual Studio 2019".
 	In the installer, select the "C++ build tools" workload.
 
-**Mac OS**
+**macOS**
 
 Information about compiler installation can be found here:
 https://devguide.python.org/setup/#macos
@@ -299,7 +299,7 @@ Creating a Python environment (pip)
 If you aren't using conda for your development environment, follow these instructions.
 You'll need to have at least Python 3.6.1 installed on your system.
 
-**Unix**/**Mac OS with virtualenv**
+**Unix**/**macOS with virtualenv**
 
 .. code-block:: bash
 
@@ -318,7 +318,7 @@ You'll need to have at least Python 3.6.1 installed on your system.
    python setup.py build_ext --inplace -j 4
    python -m pip install -e . --no-build-isolation --no-use-pep517
 
-**Unix**/**Mac OS with pyenv**
+**Unix**/**macOS with pyenv**
 
 Consult the docs for setting up pyenv `here <https://github.com/pyenv/pyenv>`__.
 

doc/source/user_guide/io.rst

@@ -23,7 +23,7 @@ The pandas I/O API is a set of top level ``reader`` functions accessed like
     text;`JSON <https://www.json.org/>`__;:ref:`read_json<io.json_reader>`;:ref:`to_json<io.json_writer>`
     text;`HTML <https://en.wikipedia.org/wiki/HTML>`__;:ref:`read_html<io.read_html>`;:ref:`to_html<io.html>`
     text; Local clipboard;:ref:`read_clipboard<io.clipboard>`;:ref:`to_clipboard<io.clipboard>`
-    ;`MS Excel <https://en.wikipedia.org/wiki/Microsoft_Excel>`__;:ref:`read_excel<io.excel_reader>`;:ref:`to_excel<io.excel_writer>`
+    binary;`MS Excel <https://en.wikipedia.org/wiki/Microsoft_Excel>`__;:ref:`read_excel<io.excel_reader>`;:ref:`to_excel<io.excel_writer>`
     binary;`OpenDocument <http://www.opendocumentformat.org>`__;:ref:`read_excel<io.ods>`;
     binary;`HDF5 Format <https://support.hdfgroup.org/HDF5/whatishdf5.html>`__;:ref:`read_hdf<io.hdf5>`;:ref:`to_hdf<io.hdf5>`
     binary;`Feather Format <https://github.com/wesm/feather>`__;:ref:`read_feather<io.feather>`;:ref:`to_feather<io.feather>`

doc/source/user_guide/text.rst

@@ -302,10 +302,10 @@ positional argument (a regex object) and return a string.
        return m.group(0)[::-1]
 
 
-   pd.Series(
-       ["foo 123", "bar baz", np.nan],
-       dtype="string"
-   ).str.replace(pat, repl, regex=True)
+   pd.Series(["foo 123", "bar baz", np.nan], dtype="string").str.replace(
+       pat, repl, regex=True
+   )
+
 
    # Using regex groups
    pat = r"(?P<one>\w+) (?P<two>\w+) (?P<three>\w+)"

doc/source/user_guide/visualization.rst

@@ -64,7 +64,7 @@ On DataFrame, :meth:`~DataFrame.plot` is a convenience to plot all of the column
 
    plt.figure();
    @savefig frame_plot_basic.png
-   df.plot()
+   df.plot();
 
 You can plot one column versus another using the ``x`` and ``y`` keywords in
 :meth:`~DataFrame.plot`:
@@ -119,7 +119,7 @@ For example, a bar plot can be created the following way:
    plt.figure();
 
    @savefig bar_plot_ex.png
-   df.iloc[5].plot(kind="bar")
+   df.iloc[5].plot(kind="bar");
 
 You can also create these other plots using the methods ``DataFrame.plot.<kind>`` instead of providing the ``kind`` keyword argument. This makes it easier to discover plot methods and the specific arguments they use:
 
@@ -180,7 +180,7 @@ bar plot:
    df2 = pd.DataFrame(np.random.rand(10, 4), columns=["a", "b", "c", "d"])
 
    @savefig bar_plot_multi_ex.png
-   df2.plot.bar()
+   df2.plot.bar();
 
 To produce a stacked bar plot, pass ``stacked=True``:
 
@@ -193,7 +193,7 @@ To produce a stacked bar plot, pass ``stacked=True``:
 .. ipython:: python
 
    @savefig bar_plot_stacked_ex.png
-   df2.plot.bar(stacked=True)
+   df2.plot.bar(stacked=True);
 
 To get horizontal bar plots, use the ``barh`` method:
 
@@ -206,7 +206,7 @@ To get horizontal bar plots, use the ``barh`` method:
 .. ipython:: python
 
    @savefig barh_plot_stacked_ex.png
-   df2.plot.barh(stacked=True)
+   df2.plot.barh(stacked=True);
 
 .. _visualization.hist:
 
@@ -414,7 +414,7 @@ groupings.  For instance,
    df = pd.DataFrame(np.random.rand(10, 2), columns=["Col1", "Col2"])
    df["X"] = pd.Series(["A", "A", "A", "A", "A", "B", "B", "B", "B", "B"])
 
-   plt.figure()
+   plt.figure();
 
    @savefig box_plot_ex2.png
    bp = df.boxplot(by="X")
@@ -518,7 +518,7 @@ When input data contains ``NaN``, it will be automatically filled by 0. If you w
    df = pd.DataFrame(np.random.rand(10, 4), columns=["a", "b", "c", "d"])
 
    @savefig area_plot_stacked.png
-   df.plot.area()
+   df.plot.area();
 
 To produce an unstacked plot, pass ``stacked=False``. Alpha value is set to 0.5 unless otherwise specified:
 
@@ -531,7 +531,7 @@ To produce an unstacked plot, pass ``stacked=False``. Alpha value is set to 0.5
 .. ipython:: python
 
    @savefig area_plot_unstacked.png
-   df.plot.area(stacked=False)
+   df.plot.area(stacked=False);
 
 .. _visualization.scatter:
 
@@ -554,7 +554,7 @@ These can be specified by the ``x`` and ``y`` keywords.
    df = pd.DataFrame(np.random.rand(50, 4), columns=["a", "b", "c", "d"])
 
    @savefig scatter_plot.png
-   df.plot.scatter(x="a", y="b")
+   df.plot.scatter(x="a", y="b");
 
 To plot multiple column groups in a single axes, repeat ``plot`` method specifying target ``ax``.
 It is recommended to specify ``color`` and ``label`` keywords to distinguish each groups.
@@ -563,7 +563,7 @@ It is recommended to specify ``color`` and ``label`` keywords to distinguish eac
 
    ax = df.plot.scatter(x="a", y="b", color="DarkBlue", label="Group 1")
    @savefig scatter_plot_repeated.png
-   df.plot.scatter(x="c", y="d", color="DarkGreen", label="Group 2", ax=ax)
+   df.plot.scatter(x="c", y="d", color="DarkGreen", label="Group 2", ax=ax);
 
 .. ipython:: python
    :suppress:
@@ -576,7 +576,7 @@ each point:
 .. ipython:: python
 
    @savefig scatter_plot_colored.png
-   df.plot.scatter(x="a", y="b", c="c", s=50)
+   df.plot.scatter(x="a", y="b", c="c", s=50);
 
 
 .. ipython:: python
@@ -591,7 +591,7 @@ bubble chart using a column of the ``DataFrame`` as the bubble size.
 .. ipython:: python
 
    @savefig scatter_plot_bubble.png
-   df.plot.scatter(x="a", y="b", s=df["c"] * 200)
+   df.plot.scatter(x="a", y="b", s=df["c"] * 200);
 
 .. ipython:: python
    :suppress:
@@ -837,7 +837,7 @@ You can create a scatter plot matrix using the
    df = pd.DataFrame(np.random.randn(1000, 4), columns=["a", "b", "c", "d"])
 
    @savefig scatter_matrix_kde.png
-   scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal="kde")
+   scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal="kde");
 
 .. ipython:: python
    :suppress:
@@ -1086,7 +1086,7 @@ layout and formatting of the returned plot:
 
    plt.figure();
    @savefig series_plot_basic2.png
-   ts.plot(style="k--", label="Series")
+   ts.plot(style="k--", label="Series");
 
 .. ipython:: python
    :suppress:
@@ -1144,7 +1144,7 @@ it empty for ylabel.
    df.plot();
 
    @savefig plot_xlabel_ylabel.png
-   df.plot(xlabel="new x", ylabel="new y")
+   df.plot(xlabel="new x", ylabel="new y");
 
 .. ipython:: python
    :suppress:
@@ -1320,7 +1320,7 @@ with the ``subplots`` keyword:
 .. ipython:: python
 
    @savefig frame_plot_subplots.png
-   df.plot(subplots=True, figsize=(6, 6))
+   df.plot(subplots=True, figsize=(6, 6));
 
 .. ipython:: python
    :suppress:
@@ -1343,7 +1343,7 @@ or columns needed, given the other.
 .. ipython:: python
 
    @savefig frame_plot_subplots_layout.png
-   df.plot(subplots=True, layout=(2, 3), figsize=(6, 6), sharex=False)
+   df.plot(subplots=True, layout=(2, 3), figsize=(6, 6), sharex=False);
 
 .. ipython:: python
    :suppress:
@@ -1354,7 +1354,7 @@ The above example is identical to using:
 
 .. ipython:: python
 
-   df.plot(subplots=True, layout=(2, -1), figsize=(6, 6), sharex=False)
+   df.plot(subplots=True, layout=(2, -1), figsize=(6, 6), sharex=False);
 
 .. ipython:: python
    :suppress:
@@ -1379,9 +1379,9 @@ otherwise you will see a warning.
    target1 = [axes[0][0], axes[1][1], axes[2][2], axes[3][3]]
    target2 = [axes[3][0], axes[2][1], axes[1][2], axes[0][3]]
 
-   df.plot(subplots=True, ax=target1, legend=False, sharex=False, sharey=False)
+   df.plot(subplots=True, ax=target1, legend=False, sharex=False, sharey=False);
    @savefig frame_plot_subplots_multi_ax.png
-   (-df).plot(subplots=True, ax=target2, legend=False, sharex=False, sharey=False)
+   (-df).plot(subplots=True, ax=target2, legend=False, sharex=False, sharey=False);
 
 .. ipython:: python
    :suppress:
@@ -1409,15 +1409,15 @@ Another option is passing an ``ax`` argument to :meth:`Series.plot` to plot on a
 
    fig, axes = plt.subplots(nrows=2, ncols=2)
    plt.subplots_adjust(wspace=0.2, hspace=0.5)
-   df["A"].plot(ax=axes[0, 0])
-   axes[0, 0].set_title("A")
-   df["B"].plot(ax=axes[0, 1])
-   axes[0, 1].set_title("B")
-   df["C"].plot(ax=axes[1, 0])
-   axes[1, 0].set_title("C")
-   df["D"].plot(ax=axes[1, 1])
+   df["A"].plot(ax=axes[0, 0]);
+   axes[0, 0].set_title("A");
+   df["B"].plot(ax=axes[0, 1]);
+   axes[0, 1].set_title("B");
+   df["C"].plot(ax=axes[1, 0]);
+   axes[1, 0].set_title("C");
+   df["D"].plot(ax=axes[1, 1]);
    @savefig series_plot_multi.png
-   axes[1, 1].set_title("D")
+   axes[1, 1].set_title("D");
 
 .. ipython:: python
    :suppress:

doc/source/whatsnew/v1.1.4.rst

@@ -28,6 +28,8 @@ Fixed regressions
 Bug fixes
 ~~~~~~~~~
 - Bug causing ``groupby(...).sum()`` and similar to not preserve metadata (:issue:`29442`)
+- Bug in :meth:`Series.isin` and :meth:`DataFrame.isin` raising a ``ValueError`` when the target was read-only (:issue:`37174`)
+- Bug in :meth:`GroupBy.fillna` that introduced a performance regression after 1.0.5 (:issue:`36757`)
 
 .. ---------------------------------------------------------------------------
 

doc/source/whatsnew/v1.2.0.rst

@@ -96,6 +96,32 @@ For example:
    buffer = io.BytesIO()
    data.to_csv(buffer, mode="w+b", encoding="utf-8", compression="gzip")
 
+Support for short caption and table position in ``to_latex``
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+:meth:`DataFrame.to_latex` now allows one to specify
+a floating table position (:issue:`35281`)
+and a short caption (:issue:`36267`).
+
+New keyword ``position`` is implemented to set the position.
+
+.. ipython:: python
+
+   data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
+   table = data.to_latex(position='ht')
+   print(table)
+
+Usage of keyword ``caption`` is extended.
+Besides taking a single string as an argument,
+one can optionally provide a tuple of ``(full_caption, short_caption)``
+to add a short caption macro.
+
+.. ipython:: python
+
+   data = pd.DataFrame({'a': [1, 2], 'b': [3, 4]})
+   table = data.to_latex(caption=('the full long caption', 'short caption'))
+   print(table)
+
 .. _whatsnew_120.read_csv_table_precision_default:
 
 Change in default floating precision for ``read_csv`` and ``read_table``
@@ -194,6 +220,7 @@ Other enhancements
 - Added :meth:`Rolling.sem()` and :meth:`Expanding.sem()` to compute the standard error of mean (:issue:`26476`).
 - :meth:`Rolling.var()` and :meth:`Rolling.std()` use Kahan summation and Welfords Method to avoid numerical issues (:issue:`37051`)
 - :meth:`DataFrame.plot` now recognizes ``xlabel`` and ``ylabel`` arguments for plots of type ``scatter`` and ``hexbin`` (:issue:`37001`)
+- :class:`DataFrame` now supports ``divmod`` operation (:issue:`37165`)
 
 .. _whatsnew_120.api_breaking.python:
 
@@ -348,8 +375,7 @@ Datetimelike
 Timedelta
 ^^^^^^^^^
 - Bug in :class:`TimedeltaIndex`, :class:`Series`, and :class:`DataFrame` floor-division with ``timedelta64`` dtypes and ``NaT`` in the denominator (:issue:`35529`)
--
--
+- Bug in parsing of ISO 8601 durations in :class:`Timedelta`, :meth:`pd.to_datetime` (:issue:`37159`, fixes :issue:`29773` and :issue:`36204`)
 
 Timezones
 ^^^^^^^^^
@@ -368,9 +394,11 @@ Numeric
 - Bug in :meth:`DataFrame.__rmatmul__` error handling reporting transposed shapes (:issue:`21581`)
 - Bug in :class:`Series` flex arithmetic methods where the result when operating with a ``list``, ``tuple`` or ``np.ndarray`` would have an incorrect name (:issue:`36760`)
 - Bug in :class:`IntegerArray` multiplication with ``timedelta`` and ``np.timedelta64`` objects (:issue:`36870`)
+- Bug in :class:`MultiIndex` comparison with tuple incorrectly treating tuple as array-like (:issue:`21517`)
 - Bug in :meth:`DataFrame.diff` with ``datetime64`` dtypes including ``NaT`` values failing to fill ``NaT`` results correctly (:issue:`32441`)
 - Bug in :class:`DataFrame` arithmetic ops incorrectly accepting keyword arguments (:issue:`36843`)
 - Bug in :class:`IntervalArray` comparisons with :class:`Series` not returning :class:`Series` (:issue:`36908`)
+- Bug in :class:`DataFrame` allowing arithmetic operations with list of array-likes with undefined results. Behavior changed to raising ``ValueError`` (:issue:`36702`)
 
 Conversion
 ^^^^^^^^^^
@@ -400,6 +428,8 @@ Indexing
 - Bug in :meth:`DataFrame.sort_index` where parameter ascending passed as a list on a single level index gives wrong result. (:issue:`32334`)
 - Bug in :meth:`DataFrame.reset_index` was incorrectly raising a ``ValueError`` for input with a :class:`MultiIndex` with missing values in a level with ``Categorical`` dtype (:issue:`24206`)
 - Bug in indexing with boolean masks on datetime-like values sometimes returning a view instead of a copy (:issue:`36210`)
+- Bug in :meth:`DataFrame.__getitem__` and :meth:`DataFrame.loc.__getitem__` with :class:`IntervalIndex` columns and a numeric indexer (:issue:`26490`)
+- Bug in :meth:`Series.loc.__getitem__` with a non-unique :class:`MultiIndex` and an empty-list indexer (:issue:`13691`)
 
 Missing
 ^^^^^^^
@@ -462,6 +492,7 @@ Groupby/resample/rolling
 - Bug in :meth:`RollingGroupby.count` where a ``ValueError`` was raised when specifying the ``closed`` parameter (:issue:`35869`)
 - Bug in :meth:`DataFrame.groupby.rolling` returning wrong values with partial centered window (:issue:`36040`).
 - Bug in :meth:`DataFrameGroupBy.rolling` returned wrong values with timeaware window containing ``NaN``. Raises ``ValueError`` because windows are not monotonic now (:issue:`34617`)
+- Bug in :meth:`Rolling.__iter__` where a ``ValueError`` was not raised when ``min_periods`` was larger than ``window`` (:issue:`37156`)
 
 Reshaping
 ^^^^^^^^^