Merge branch 'master' into single_level

Author: LucasG0

doc/source/development/contributing.rst

@@ -1275,8 +1275,8 @@ Performance matters and it is worth considering whether your code has introduced
 performance regressions. pandas is in the process of migrating to
 `asv benchmarks <https://github.com/spacetelescope/asv>`__
 to enable easy monitoring of the performance of critical pandas operations.
-These benchmarks are all found in the ``pandas/asv_bench`` directory.  asv
-supports both python2 and python3.
+These benchmarks are all found in the ``pandas/asv_bench`` directory, and the
+test results can be found `here <https://pandas.pydata.org/speed/pandas/#/>`__.
 
 To use all features of asv, you will need either ``conda`` or
 ``virtualenv``. For more details please check the `asv installation

doc/source/ecosystem.rst

@@ -153,6 +153,23 @@ A good implementation for Python users is `has2k1/plotnine <https://github.com/h
 Spun off from the main pandas library, the `qtpandas <https://github.com/draperjames/qtpandas>`__
 library enables DataFrame visualization and manipulation in PyQt4 and PySide applications.
 
+`D-Tale <https://github.com/man-group/dtale>`__
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+D-Tale is a lightweight web client for visualizing pandas data structures. It
+provides a rich spreadsheet-style grid which acts as a wrapper for a lot of
+pandas functionality (query, sort, describe, corr...) so users can quickly
+manipulate their data. There is also an interactive chart-builder using Plotly
+Dash allowing users to build nice portable visualizations. D-Tale can be
+invoked with the following command
+
+.. code:: python
+
+    import dtale; dtale.show(df)
+
+D-Tale integrates seamlessly with jupyter notebooks, python terminals, kaggle
+& Google Colab. Here are some demos of the `grid <http://alphatechadmin.pythonanywhere.com/>`__
+and `chart-builder <http://alphatechadmin.pythonanywhere.com/charts/4?chart_type=surface&query=&x=date&z=Col0&agg=raw&cpg=false&y=%5B%22security_id%22%5D>`__.
 
 .. _ecosystem.ide:
 

doc/source/reference/frame.rst

@@ -240,13 +240,14 @@ Reshaping, sorting, transposing
    DataFrame.T
    DataFrame.transpose
 
-Combining / joining / merging
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Combining / comparing / joining / merging
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 .. autosummary::
    :toctree: api/
 
    DataFrame.append
    DataFrame.assign
+   DataFrame.compare
    DataFrame.join
    DataFrame.merge
    DataFrame.update

doc/source/reference/offset_frequency.rst

@@ -59,6 +59,9 @@ Properties
     BusinessDay.normalize
     BusinessDay.rule_code
     BusinessDay.n
+    BusinessDay.weekmask
+    BusinessDay.holidays
+    BusinessDay.calendar
 
 Methods
 ~~~~~~~
@@ -95,6 +98,9 @@ Properties
     BusinessHour.n
     BusinessHour.start
     BusinessHour.end
+    BusinessHour.weekmask
+    BusinessHour.holidays
+    BusinessHour.calendar
 
 Methods
 ~~~~~~~
@@ -128,6 +134,9 @@ Properties
     CustomBusinessDay.normalize
     CustomBusinessDay.rule_code
     CustomBusinessDay.n
+    CustomBusinessDay.weekmask
+    CustomBusinessDay.calendar
+    CustomBusinessDay.holidays
 
 Methods
 ~~~~~~~
@@ -162,6 +171,9 @@ Properties
     CustomBusinessHour.normalize
     CustomBusinessHour.rule_code
     CustomBusinessHour.n
+    CustomBusinessHour.weekmask
+    CustomBusinessHour.calendar
+    CustomBusinessHour.holidays
     CustomBusinessHour.start
     CustomBusinessHour.end
 
@@ -334,6 +346,9 @@ Properties
     CustomBusinessMonthEnd.normalize
     CustomBusinessMonthEnd.rule_code
     CustomBusinessMonthEnd.n
+    CustomBusinessMonthEnd.weekmask
+    CustomBusinessMonthEnd.calendar
+    CustomBusinessMonthEnd.holidays
 
 Methods
 ~~~~~~~
@@ -368,6 +383,9 @@ Properties
     CustomBusinessMonthBegin.normalize
     CustomBusinessMonthBegin.rule_code
     CustomBusinessMonthBegin.n
+    CustomBusinessMonthBegin.weekmask
+    CustomBusinessMonthBegin.calendar
+    CustomBusinessMonthBegin.holidays
 
 Methods
 ~~~~~~~
@@ -1241,6 +1259,9 @@ Properties
     BDay.offset
     BDay.rule_code
     BDay.n
+    BDay.weekmask
+    BDay.holidays
+    BDay.calendar
 
 Methods
 ~~~~~~~
@@ -1356,6 +1377,9 @@ Properties
     CBMonthEnd.offset
     CBMonthEnd.rule_code
     CBMonthEnd.n
+    CBMonthEnd.weekmask
+    CBMonthEnd.holidays
+    CBMonthEnd.calendar
 
 Methods
 ~~~~~~~
@@ -1397,6 +1421,9 @@ Properties
     CBMonthBegin.offset
     CBMonthBegin.rule_code
     CBMonthBegin.n
+    CBMonthBegin.weekmask
+    CBMonthBegin.holidays
+    CBMonthBegin.calendar
 
 Methods
 ~~~~~~~
@@ -1435,6 +1462,9 @@ Properties
     CDay.offset
     CDay.rule_code
     CDay.n
+    CDay.weekmask
+    CDay.calendar
+    CDay.holidays
 
 Methods
 ~~~~~~~
@@ -1452,6 +1482,7 @@ Methods
     CDay.rollforward
     CDay.__call__
 
+
 .. _api.frequencies:
 
 ===========

doc/source/reference/series.rst

@@ -240,12 +240,13 @@ Reshaping, sorting
    Series.squeeze
    Series.view
 
-Combining / joining / merging
------------------------------
+Combining / comparing / joining / merging
+-----------------------------------------
 .. autosummary::
    :toctree: api/
 
    Series.append
+   Series.compare
    Series.replace
    Series.update
 

doc/source/user_guide/merging.rst

@@ -10,15 +10,18 @@
    p = doctools.TablePlotter()
 
 
-****************************
-Merge, join, and concatenate
-****************************
+************************************
+Merge, join, concatenate and compare
+************************************
 
 pandas provides various facilities for easily combining together Series or
 DataFrame with various kinds of set logic for the indexes
 and relational algebra functionality in the case of join / merge-type
 operations.
 
+In addition, pandas also provides utilities to compare two Series or DataFrame
+and summarize their differences.
+
 .. _merging.concat:
 
 Concatenating objects
@@ -1477,3 +1480,61 @@ exclude exact matches on time. Note that though we exclude the exact matches
                  by='ticker',
                  tolerance=pd.Timedelta('10ms'),
                  allow_exact_matches=False)
+
+.. _merging.compare:
+
+Comparing objects
+-----------------
+
+The :meth:`~Series.compare` and :meth:`~DataFrame.compare` methods allow you to
+compare two DataFrame or Series, respectively, and summarize their differences.
+
+This feature was added in :ref:`V1.1.0 <whatsnew_110.dataframe_or_series_comparing>`.
+
+For example, you might want to compare two `DataFrame` and stack their differences
+side by side.
+
+.. ipython:: python
+
+   df = pd.DataFrame(
+       {
+           "col1": ["a", "a", "b", "b", "a"],
+           "col2": [1.0, 2.0, 3.0, np.nan, 5.0],
+           "col3": [1.0, 2.0, 3.0, 4.0, 5.0]
+       },
+       columns=["col1", "col2", "col3"],
+   )
+   df
+
+.. ipython:: python
+
+   df2 = df.copy()
+   df2.loc[0, 'col1'] = 'c'
+   df2.loc[2, 'col3'] = 4.0
+   df2
+
+.. ipython:: python
+
+   df.compare(df2)
+
+By default, if two corresponding values are equal, they will be shown as ``NaN``.
+Furthermore, if all values in an entire row / column, the row / column will be
+omitted from the result. The remaining differences will be aligned on columns.
+
+If you wish, you may choose to stack the differences on rows.
+
+.. ipython:: python
+
+   df.compare(df2, align_axis=0)
+
+If you wish to keep all original rows and columns, set `keep_shape` argument
+to ``True``.
+
+.. ipython:: python
+
+   df.compare(df2, keep_shape=True)
+
+You may also keep all the original values even if they are equal.
+
+.. ipython:: python
+   df.compare(df2, keep_shape=True, keep_equal=True)

doc/source/user_guide/visualization.rst

@@ -821,7 +821,7 @@ You can create a scatter plot matrix using the
    df = pd.DataFrame(np.random.randn(1000, 4), columns=['a', 'b', 'c', 'd'])
 
    @savefig scatter_matrix_kde.png
-   scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal='kde')
+   scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal='kde');
 
 .. ipython:: python
    :suppress:
@@ -1683,4 +1683,4 @@ to generate the plots. Some libraries implementing a backend for pandas are list
 on the ecosystem :ref:`ecosystem.visualization` page.
 
 Developers guide can be found at
-https://dev.pandas.io/docs/development/extending.html#plotting-backends
+https://pandas.pydata.org/docs/dev/development/extending.html#plotting-backends

doc/source/whatsnew/index.rst

@@ -24,6 +24,7 @@ Version 1.0
 .. toctree::
    :maxdepth: 2
 
+   v1.0.4
    v1.0.3
    v1.0.2
    v1.0.1

doc/source/whatsnew/v1.0.3.rst

@@ -26,4 +26,4 @@ Bug fixes
 Contributors
 ~~~~~~~~~~~~
 
-.. contributors:: v1.0.2..v1.0.3|HEAD
+.. contributors:: v1.0.2..v1.0.3

doc/source/whatsnew/v1.0.4.rst

@@ -0,0 +1,48 @@
+
+.. _whatsnew_104:
+
+What's new in 1.0.4 (May 28, 2020)
+------------------------------------
+
+These are the changes in pandas 1.0.4. See :ref:`release` for a full changelog
+including other versions of pandas.
+
+{{ header }}
+
+.. ---------------------------------------------------------------------------
+
+.. _whatsnew_104.regressions:
+
+Fixed regressions
+~~~~~~~~~~~~~~~~~
+- Fix regression where :meth:`Series.isna` and :meth:`DataFrame.isna` would raise for categorical dtype when ``pandas.options.mode.use_inf_as_na`` was set to ``True`` (:issue:`33594`)
+- Fix regression in :meth:`GroupBy.first` and :meth:`GroupBy.last` where None is not preserved in object dtype (:issue:`32800`)
+- Fix regression in DataFrame reductions using ``numeric_only=True`` and ExtensionArrays (:issue:`33256`).
+- Fix performance regression in ``memory_usage(deep=True)`` for object dtype (:issue:`33012`)
+- Fix regression where :meth:`Categorical.replace` would replace with ``NaN`` whenever the new value and replacement value were equal (:issue:`33288`)
+- Fix regression where an ordered :class:`Categorical` containing only ``NaN`` values would raise rather than returning ``NaN`` when taking the minimum or maximum  (:issue:`33450`)
+- Fix regression in :meth:`DataFrameGroupBy.agg` with dictionary input losing ``ExtensionArray`` dtypes (:issue:`32194`)
+- Fix to preserve the ability to index with the "nearest" method with xarray's CFTimeIndex, an :class:`Index` subclass (`pydata/xarray#3751 <https://github.com/pydata/xarray/issues/3751>`_, :issue:`32905`).
+- Fix regression in :meth:`DataFrame.describe` raising ``TypeError: unhashable type: 'dict'`` (:issue:`32409`)
+- Fix regression in :meth:`DataFrame.replace` casts columns to ``object`` dtype if items in ``to_replace`` not in values (:issue:`32988`)
+- Fix regression in :meth:`Series.groupby` would raise ``ValueError`` when grouping by :class:`PeriodIndex` level (:issue:`34010`)
+- Fix regression in :meth:`GroupBy.rolling.apply` ignores args and kwargs parameters (:issue:`33433`)
+- Fix regression in error message with ``np.min`` or ``np.max`` on unordered :class:`Categorical` (:issue:`33115`)
+- Fix regression in :meth:`DataFrame.loc` and :meth:`Series.loc` throwing an error when a ``datetime64[ns, tz]`` value is provided (:issue:`32395`)
+
+.. _whatsnew_104.bug_fixes:
+
+Bug fixes
+~~~~~~~~~
+- Bug in :meth:`SeriesGroupBy.first`, :meth:`SeriesGroupBy.last`, :meth:`SeriesGroupBy.min`, and :meth:`SeriesGroupBy.max` returning floats when applied to nullable Booleans (:issue:`33071`)
+- Bug in :meth:`Rolling.min` and :meth:`Rolling.max`: Growing memory usage after multiple calls when using a fixed window (:issue:`30726`)
+- Bug in :meth:`~DataFrame.to_parquet` was not raising ``PermissionError`` when writing to a private s3 bucket with invalid creds. (:issue:`27679`)
+- Bug in :meth:`~DataFrame.to_csv` was silently failing when writing to an invalid s3 bucket. (:issue:`32486`)
+- Bug in :meth:`read_parquet` was raising a ``FileNotFoundError`` when passed an s3 directory path. (:issue:`26388`)
+- Bug in :meth:`~DataFrame.to_parquet` was throwing an ``AttributeError`` when writing a partitioned parquet file to s3 (:issue:`27596`)
+- Bug in :meth:`GroupBy.quantile` causes the quantiles to be shifted when the ``by`` axis contains ``NaN`` (:issue:`33200`, :issue:`33569`)
+
+Contributors
+~~~~~~~~~~~~
+
+.. contributors:: v1.0.3..v1.0.4|HEAD