Backports round 2

.gitignore

@@ -106,3 +106,4 @@ doc/build/html/index.html
 doc/tmp.sv
 doc/source/styled.xlsx
 doc/source/templates/
+env/

doc/source/computation.rst

@@ -348,7 +348,7 @@ The following methods are available:
 
 The weights used in the window are specified by the ``win_type`` keyword.
 The list of recognized types are the `scipy.signal window functions
-  <https://docs.scipy.org/doc/scipy/reference/signal.html#window-functions>`__:
+<https://docs.scipy.org/doc/scipy/reference/signal.html#window-functions>`__:
 
 - ``boxcar``
 - ``triang``

doc/source/io.rst

@@ -4548,11 +4548,8 @@ dtypes, including extension dtypes such as datetime with tz.
 
 Several caveats.
 
-- The format will NOT write an ``Index``, or ``MultiIndex`` for the
-  ``DataFrame`` and will raise an error if a non-default one is provided. You
-  can ``.reset_index()`` to store the index or ``.reset_index(drop=True)`` to
-  ignore it.
 - Duplicate column names and non-string columns names are not supported
+- Index level names, if specified, must be strings
 - Categorical dtypes can be serialized to parquet, but will de-serialize as ``object`` dtype.
 - Non supported types include ``Period`` and actual python object types. These will raise a helpful error message
   on an attempt at serialization.

doc/source/options.rst

@@ -433,7 +433,7 @@ compute.use_numexpr                     True         Use the numexpr library to
                                                      computation if it is installed.
 plotting.matplotlib.register_converters True         Register custom converters with
                                                      matplotlib. Set to False to de-register.
-======================================= ============ ========================================
+======================================= ============ ==================================
 
 
 .. _basics.console_output:

doc/source/whatsnew/v0.21.1.txt

@@ -3,9 +3,23 @@
 v0.21.1
 -------
 
-This is a minor release from 0.21.1 and includes a number of deprecations, new
-features, enhancements, and performance improvements along with a large number
-of bug fixes. We recommend that all users upgrade to this version.
+This is a minor bug-fix release in the 0.21.x series and includes some small regression fixes,
+bug fixes and performance improvements.
+We recommend that all users upgrade to this version.
+
+Highlights include:
+
+- Temporarily restore matplotlib datetime plotting functionality. This should
+  resolve issues for users who relied implicitly on pandas to plot datetimes
+  with matplotlib. See :ref:`here <whatsnew_0211.special>`.
+- Improvements to the Parquet IO functions introduced in 0.21.0. See
+  :ref:`here <whatsnew_0211.enhancements.parquet>`.
+
+
+.. contents:: What's new in v0.21.1
+    :local:
+    :backlinks: none
+
 
 .. _whatsnew_0211.special:
 
@@ -42,9 +56,16 @@ registering them when they want them.
 New features
 ~~~~~~~~~~~~
 
--
--
--
+.. _whatsnew_0211.enhancements.parquet:
+
+Improvements to the Parquet IO functionality
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- :func:`DataFrame.to_parquet` will now write non-default indexes when the
+  underlying engine supports it. The indexes will be preserved when reading
+  back in with :func:`read_parquet` (:issue:`18581`).
+- :func:`read_parquet` now allows to specify the columns to read from a parquet file (:issue:`18154`)
+- :func:`read_parquet` now allows to specify kwargs which are passed to the respective engine (:issue:`18216`)
 
 .. _whatsnew_0211.enhancements.other:
 
@@ -53,7 +74,6 @@ Other Enhancements
 
 - :meth:`Timestamp.timestamp` is now available in Python 2.7. (:issue:`17329`)
 - :class:`Grouper` and :class:`TimeGrouper` now have a friendly repr output (:issue:`18203`).
--
 
 .. _whatsnew_0211.deprecations:
 
@@ -69,17 +89,6 @@ Performance Improvements
 ~~~~~~~~~~~~~~~~~~~~~~~~
 
 - Improved performance of plotting large series/dataframes (:issue:`18236`).
--
--
-
-.. _whatsnew_0211.docs:
-
-Documentation Changes
-~~~~~~~~~~~~~~~~~~~~~
-
--
--
--
 
 .. _whatsnew_0211.bug_fixes:
 
@@ -94,7 +103,7 @@ Conversion
 - Bug in :meth:`IntervalIndex.copy` when copying and ``IntervalIndex`` with non-default ``closed`` (:issue:`18339`)
 - Bug in :func:`DataFrame.to_dict` where columns of datetime that are tz-aware were not converted to required arrays when used with ``orient='records'``, raising``TypeError` (:issue:`18372`)
 - Bug in :class:`DateTimeIndex` and :meth:`date_range` where mismatching tz-aware ``start`` and ``end`` timezones would not raise an err if ``end.tzinfo`` is None (:issue:`18431`)
--
+- Bug in :meth:`Series.fillna` which raised when passed a long integer on Python 2 (:issue:`18159`).
 
 Indexing
 ^^^^^^^^
@@ -104,7 +113,6 @@ Indexing
 - Bug in :class:`IntervalIndex` constructor when a list of intervals is passed with non-default ``closed`` (:issue:`18334`)
 - Bug in ``Index.putmask`` when an invalid mask passed (:issue:`18368`)
 - Bug in masked assignment of a ``timedelta64[ns]`` dtype ``Series``, incorrectly coerced to float (:issue:`18493`)
--
 
 I/O
 ^^^
@@ -114,21 +122,19 @@ I/O
 - Bug in :func:`read_csv` for handling null values in index columns when specifying ``na_filter=False`` (:issue:`5239`)
 - Bug in :func:`read_csv` when reading numeric category fields with high cardinality (:issue:`18186`)
 - Bug in :meth:`DataFrame.to_csv` when the table had ``MultiIndex`` columns, and a list of strings was passed in for ``header`` (:issue:`5539`)
-- :func:`read_parquet` now allows to specify the columns to read from a parquet file (:issue:`18154`)
-- :func:`read_parquet` now allows to specify kwargs which are passed to the respective engine (:issue:`18216`)
 - Bug in parsing integer datetime-like columns with specified format in ``read_sql`` (:issue:`17855`).
 - Bug in :meth:`DataFrame.to_msgpack` when serializing data of the numpy.bool_ datatype (:issue:`18390`)
 - Bug in :func:`read_json` not decoding when reading line deliminted JSON from S3 (:issue:`17200`)
 - Bug in :func:`pandas.io.json.json_normalize` to avoid modification of ``meta`` (:issue:`18610`)
 - Bug in :func:`to_latex` where repeated multi-index values were not printed even though a higher level index differed from the previous row (:issue:`14484`)
+- Bug when reading NaN-only categorical columns in :class:`HDFStore` (:issue:`18413`)
+- Bug in :meth:`DataFrame.to_latex` with ``longtable=True`` where a latex multicolumn always spanned over three columns (:issue:`17959`)
 
 
 Plotting
 ^^^^^^^^
 
 - Bug in ``DataFrame.plot()`` and ``Series.plot()`` with :class:`DatetimeIndex` where a figure generated by them is not pickleable in Python 3 (:issue:`18439`)
--
--
 
 Groupby/Resample/Rolling
 ^^^^^^^^^^^^^^^^^^^^^^^^
@@ -137,15 +143,6 @@ Groupby/Resample/Rolling
 - Bug in ``DataFrame.resample(...)`` when there is a time change (DST) and resampling frequecy is 12h or higher (:issue:`15549`)
 - Bug in ``pd.DataFrameGroupBy.count()`` when counting over a datetimelike column (:issue:`13393`)
 - Bug in ``rolling.var`` where calculation is inaccurate with a zero-valued array (:issue:`18430`)
--
--
-
-Sparse
-^^^^^^
-
--
--
--
 
 Reshaping
 ^^^^^^^^^
@@ -159,9 +156,8 @@ Numeric
 ^^^^^^^
 
 - Bug in ``pd.Series.rolling.skew()`` and ``rolling.kurt()`` with all equal values has floating issue (:issue:`18044`)
--
--
--
+- Bug in :class:`TimedeltaIndex` subtraction could incorrectly overflow when ``NaT`` is present (:issue:`17791`)
+- Bug in :class:`DatetimeIndex` subtracting datetimelike from DatetimeIndex could fail to overflow (:issue:`18020`)
 
 Categorical
 ^^^^^^^^^^^
@@ -177,9 +173,3 @@ String
 ^^^^^^
 
 - :meth:`Series.str.split()` will now propogate ``NaN`` values across all expanded columns instead of ``None`` (:issue:`18450`)
-
-Other
-^^^^^
-
--
--

pandas/core/indexes/datetimelike.py

@@ -681,7 +681,7 @@ def __sub__(self, other):
                 return self._add_delta(-other)
             elif is_integer(other):
                 return self.shift(-other)
-            elif isinstance(other, datetime):
+            elif isinstance(other, (datetime, np.datetime64)):
                 return self._sub_datelike(other)
             elif isinstance(other, Period):
                 return self._sub_period(other)

pandas/core/indexes/datetimes.py

@@ -31,6 +31,7 @@
 import pandas.core.dtypes.concat as _concat
 from pandas.errors import PerformanceWarning
 from pandas.core.common import _values_from_object, _maybe_box
+from pandas.core.algorithms import checked_add_with_arr
 
 from pandas.core.indexes.base import Index, _index_shared_docs
 from pandas.core.indexes.numeric import Int64Index, Float64Index
@@ -762,7 +763,7 @@ def _sub_datelike(self, other):
                 raise TypeError("DatetimeIndex subtraction must have the same "
                                 "timezones or no timezones")
             result = self._sub_datelike_dti(other)
-        elif isinstance(other, datetime):
+        elif isinstance(other, (datetime, np.datetime64)):
             other = Timestamp(other)
             if other is libts.NaT:
                 result = self._nat_new(box=False)
@@ -772,7 +773,8 @@ def _sub_datelike(self, other):
                                 "timezones or no timezones")
             else:
                 i8 = self.asi8
-                result = i8 - other.value
+                result = checked_add_with_arr(i8, -other.value,
+                                              arr_mask=self._isnan)
                 result = self._maybe_mask_results(result,
                                                   fill_value=libts.iNaT)
         else:

pandas/core/indexes/timedeltas.py

@@ -381,7 +381,8 @@ def _add_datelike(self, other):
         else:
             other = Timestamp(other)
             i8 = self.asi8
-            result = checked_add_with_arr(i8, other.value)
+            result = checked_add_with_arr(i8, other.value,
+                                          arr_mask=self._isnan)
             result = self._maybe_mask_results(result, fill_value=iNaT)
         return DatetimeIndex(result, name=self.name, copy=False)
 

pandas/core/internals.py

@@ -1837,8 +1837,10 @@ def _can_hold_element(self, element):
         if tipo is not None:
             return (issubclass(tipo.type, (np.floating, np.integer)) and
                     not issubclass(tipo.type, (np.datetime64, np.timedelta64)))
-        return (isinstance(element, (float, int, np.floating, np.int_)) and
-                not isinstance(element, (bool, np.bool_, datetime, timedelta,
+        return (
+            isinstance(
+                element, (float, int, np.floating, np.int_, compat.long))
+            and not isinstance(element, (bool, np.bool_, datetime, timedelta,
                                          np.datetime64, np.timedelta64)))
 
     def to_native_types(self, slicer=None, na_rep='', float_format=None,
@@ -1886,9 +1888,11 @@ def _can_hold_element(self, element):
         if tipo is not None:
             return issubclass(tipo.type,
                               (np.floating, np.integer, np.complexfloating))
-        return (isinstance(element,
-                           (float, int, complex, np.float_, np.int_)) and
-                not isinstance(element, (bool, np.bool_)))
+        return (
+            isinstance(
+                element,
+                (float, int, complex, np.float_, np.int_, compat.long))
+            and not isinstance(element, (bool, np.bool_)))
 
     def should_store(self, value):
         return issubclass(value.dtype.type, np.complexfloating)

pandas/io/formats/format.py

@@ -950,8 +950,8 @@ def get_col_type(dtype):
                 if self.longtable:
                     buf.write('\\endhead\n')
                     buf.write('\\midrule\n')
-                    buf.write('\\multicolumn{3}{r}{{Continued on next '
-                              'page}} \\\\\n')
+                    buf.write('\\multicolumn{{{n}}}{{r}}{{{{Continued on next '
+                              'page}}}} \\\\\n'.format(n=len(row)))
                     buf.write('\\midrule\n')
                     buf.write('\\endfoot\n\n')
                     buf.write('\\bottomrule\n')