merge with upstream

Author: arw2019

ci/deps/travis-37-cov.yaml

@@ -1,6 +1,5 @@
 name: pandas-dev
 channels:
-  - defaults
   - conda-forge
 dependencies:
   - python=3.7.*
@@ -15,7 +14,6 @@ dependencies:
   # pandas dependencies
   - beautifulsoup4
   - botocore>=1.11
-  - cython>=0.29.16
   - dask
   - fastparquet>=0.3.2
   - fsspec>=0.7.4
@@ -31,16 +29,18 @@ dependencies:
   - odfpy
   - openpyxl
   - pandas-gbq
+  - google-cloud-bigquery>=1.27.2 # GH 36436
   - psycopg2
   - pyarrow>=0.15.0
-  - pymysql
+  - pymysql=0.7.11
   - pytables
   - python-snappy
+  - python-dateutil
   - pytz
   - s3fs>=0.4.0
   - scikit-learn
   - scipy
-  - sqlalchemy
+  - sqlalchemy=1.3.0
   - statsmodels
   - xarray
   - xlrd
@@ -51,5 +51,4 @@ dependencies:
     - brotlipy
     - coverage
     - pandas-datareader
-    - python-dateutil
     - pyxlsb

ci/deps/travis-37-locale.yaml

@@ -25,10 +25,10 @@ dependencies:
   - numexpr
   - numpy
   - openpyxl
-  - pandas-gbq=0.12.0
+  - pandas-gbq
+  - google-cloud-bigquery>=1.27.2 # GH 36436
   - pyarrow>=0.17
   - psycopg2=2.7
-  - pyarrow>=0.15.0 # GH #35813
   - pymysql=0.7.11
   - pytables
   - python-dateutil

doc/make.py

@@ -291,7 +291,7 @@ def main():
 
     joined = ", ".join(cmds)
     argparser.add_argument(
-        "command", nargs="?", default="html", help=f"command to run: {joined}",
+        "command", nargs="?", default="html", help=f"command to run: {joined}"
     )
     argparser.add_argument(
         "--num-jobs", type=int, default=0, help="number of jobs used by sphinx-build"

doc/source/user_guide/computation.rst

@@ -229,6 +229,15 @@ see the :ref:`groupby docs <groupby.transform.window_resample>`.
 
    The API for window statistics is quite similar to the way one works with ``GroupBy`` objects, see the documentation :ref:`here <groupby>`.
 
+.. warning::
+
+    When using ``rolling()`` and an associated function the results are calculated with rolling sums. As a consequence
+    when having values differing with magnitude :math:`1/np.finfo(np.double).eps` this results in truncation. It must be
+    noted, that large values may have an impact on windows, which do not include these values. `Kahan summation
+    <https://en.wikipedia.org/wiki/Kahan_summation_algorithm>`__ is used
+    to compute the rolling sums to preserve accuracy as much as possible. The same holds true for ``Rolling.var()`` for
+    values differing with magnitude :math:`(1/np.finfo(np.double).eps)^{0.5}`.
+
 We work with ``rolling``, ``expanding`` and ``exponentially weighted`` data through the corresponding
 objects, :class:`~pandas.core.window.Rolling`, :class:`~pandas.core.window.Expanding` and :class:`~pandas.core.window.ExponentialMovingWindow`.
 

doc/source/whatsnew/v1.1.3.rst

@@ -34,7 +34,7 @@ Fixed regressions
 - Fixed regression in :meth:`Series.__getitem__` incorrectly raising when the input was a tuple (:issue:`35534`)
 - Fixed regression in :meth:`Series.__getitem__` incorrectly raising when the input was a frozenset (:issue:`35747`)
 - Fixed regression in :meth:`read_excel` with ``engine="odf"`` caused ``UnboundLocalError`` in some cases where cells had nested child nodes (:issue:`36122`,:issue:`35802`)
--
+- Fixed regression in :class:`DataFrame` and :class:`Series` comparisons between numeric arrays and strings (:issue:`35700`,:issue:`36377`)
 
 .. ---------------------------------------------------------------------------
 
@@ -47,6 +47,7 @@ Bug fixes
 - Bug in :class:`Series` constructor where integer overflow would occur for sufficiently large scalar inputs when an index was provided (:issue:`36291`)
 - Bug in :meth:`DataFrame.sort_values` raising an ``AttributeError`` when sorting on a key that casts column to categorical dtype (:issue:`36383`)
 - Bug in :meth:`DataFrame.stack` raising a ``ValueError`` when stacking :class:`MultiIndex` columns based on position when the levels had duplicate names (:issue:`36353`)
+- Bug in :meth:`Series.isin` and :meth:`DataFrame.isin` when using ``NaN`` and a row length above 1,000,000 (:issue:`22205`)
 
 .. ---------------------------------------------------------------------------
 

doc/source/whatsnew/v1.2.0.rst

@@ -212,6 +212,7 @@ Deprecations
 - Deprecated parameter ``dtype`` in :~meth:`Index.copy` on method all index classes. Use the :meth:`Index.astype` method instead for changing dtype(:issue:`35853`)
 - Date parser functions :func:`~pandas.io.date_converters.parse_date_time`, :func:`~pandas.io.date_converters.parse_date_fields`, :func:`~pandas.io.date_converters.parse_all_fields` and :func:`~pandas.io.date_converters.generic_parser` from ``pandas.io.date_converters`` are deprecated and will be removed in a future version; use :func:`to_datetime` instead (:issue:`35741`)
 - :meth:`DataFrame.lookup` is deprecated and will be removed in a future version, use :meth:`DataFrame.melt` and :meth:`DataFrame.loc` instead (:issue:`18682`)
+- The :meth:`Index.to_native_types` is deprecated. Use ``.astype(str)`` instead (:issue:`28867`)
 
 .. ---------------------------------------------------------------------------
 
@@ -225,6 +226,7 @@ Performance improvements
 - Performance improvement in :meth:`GroupBy.agg` with the ``numba`` engine (:issue:`35759`)
 - Performance improvements when creating :meth:`pd.Series.map` from a huge dictionary (:issue:`34717`)
 - Performance improvement in :meth:`GroupBy.transform` with the ``numba`` engine (:issue:`36240`)
+- ``Styler`` uuid method altered to compress data transmission over web whilst maintaining reasonably low table collision probability (:issue:`36345`)
 - Performance improvement in :meth:`pd.to_datetime` with non-`ns` time unit for `float` `dtype` columns (:issue:`20445`)
 
 .. ---------------------------------------------------------------------------
@@ -368,7 +370,7 @@ Other
 - Bug in :meth:`DataFrame.replace` and :meth:`Series.replace` incorrectly raising ``AssertionError`` instead of ``ValueError`` when invalid parameter combinations are passed (:issue:`36045`)
 - Bug in :meth:`DataFrame.replace` and :meth:`Series.replace` with numeric values and string ``to_replace`` (:issue:`34789`)
 - Bug in :meth:`Series.transform` would give incorrect results or raise when the argument ``func`` was dictionary (:issue:`35811`)
--
+- Bug in :meth:`Index.union` behaving differently depending on whether operand is a :class:`Index` or other list-like (:issue:`36384`)
 
 .. ---------------------------------------------------------------------------
 

pandas/_config/display.py

@@ -22,7 +22,7 @@ def detect_console_encoding() -> str:
     encoding = None
     try:
         encoding = sys.stdout.encoding or sys.stdin.encoding
-    except (AttributeError, IOError):
+    except (AttributeError, OSError):
         pass
 
     # try again for something better

pandas/_libs/reduction.pyx

@@ -16,12 +16,12 @@ from pandas._libs cimport util
 from pandas._libs.lib import is_scalar, maybe_convert_objects
 
 
-cdef _check_result_array(object obj, Py_ssize_t cnt):
+cpdef check_result_array(object obj, Py_ssize_t cnt):
 
     if (util.is_array(obj) or
             (isinstance(obj, list) and len(obj) == cnt) or
             getattr(obj, 'shape', None) == (cnt,)):
-        raise ValueError('Function does not reduce')
+        raise ValueError('Must produce aggregated value')
 
 
 cdef class _BaseGrouper:
@@ -74,12 +74,14 @@ cdef class _BaseGrouper:
         cached_ityp._engine.clear_mapping()
         cached_ityp._cache.clear()  # e.g. inferred_freq must go
         res = self.f(cached_typ)
-        res = _extract_result(res)
+        res = extract_result(res)
         if not initialized:
             # On the first pass, we check the output shape to see
             #  if this looks like a reduction.
             initialized = True
-            _check_result_array(res, len(self.dummy_arr))
+            # In all tests other than test_series_grouper and
+            #  test_series_bin_grouper, we have len(self.dummy_arr) == 0
+            check_result_array(res, len(self.dummy_arr))
 
         return res, initialized
 
@@ -278,9 +280,14 @@ cdef class SeriesGrouper(_BaseGrouper):
         return result, counts
 
 
-cdef inline _extract_result(object res, bint squeeze=True):
+cpdef inline extract_result(object res, bint squeeze=True):
     """ extract the result object, it might be a 0-dim ndarray
         or a len-1 0-dim, or a scalar """
+    if hasattr(res, "_values"):
+        # Preserve EA
+        res = res._values
+        if squeeze and res.ndim == 1 and len(res) == 1:
+            res = res[0]
     if hasattr(res, 'values') and util.is_array(res.values):
         res = res.values
     if util.is_array(res):

pandas/_testing.py

@@ -1960,8 +1960,7 @@ def index_subclass_makers_generator():
         makeCategoricalIndex,
         makeMultiIndex,
     ]
-    for make_index_func in make_index_funcs:
-        yield make_index_func
+    yield from make_index_funcs
 
 
 def all_timeseries_index_generator(k=10):

pandas/_vendored/typing_extensions.py

@@ -409,7 +409,7 @@ def __repr__(self):
 
         def __getitem__(self, parameters):
             item = typing._type_check(
-                parameters, "{} accepts only single type".format(self._name)
+                parameters, f"{self._name} accepts only single type"
             )
             return _GenericAlias(self, (item,))
 
@@ -1671,7 +1671,7 @@ def __class_getitem__(cls, params):
                 params = (params,)
             if not params and cls is not Tuple:
                 raise TypeError(
-                    "Parameter list to {}[...] cannot be empty".format(cls.__qualname__)
+                    f"Parameter list to {cls.__qualname__}[...] cannot be empty"
                 )
             msg = "Parameters to generic types must be types."
             params = tuple(_type_check(p, msg) for p in params)
@@ -2113,7 +2113,7 @@ def __class_getitem__(cls, params):
             return _AnnotatedAlias(origin, metadata)
 
         def __init_subclass__(cls, *args, **kwargs):
-            raise TypeError("Cannot subclass {}.Annotated".format(cls.__module__))
+            raise TypeError(f"Cannot subclass {cls.__module__}.Annotated")
 
     def _strip_annotations(t):
         """Strips the annotations from a given type.
@@ -2195,7 +2195,7 @@ def _tree_repr(self, tree):
             else:
                 tp_repr = origin[0]._tree_repr(origin)
             metadata_reprs = ", ".join(repr(arg) for arg in metadata)
-            return "%s[%s, %s]" % (cls, tp_repr, metadata_reprs)
+            return f"{cls}[{tp_repr}, {metadata_reprs}]"
 
         def _subs_tree(self, tvars=None, args=None):  # noqa
             if self is Annotated:
@@ -2382,7 +2382,7 @@ def TypeAlias(self, parameters):
 
         It's invalid when used anywhere except as in the example above.
         """
-        raise TypeError("{} is not subscriptable".format(self))
+        raise TypeError(f"{self} is not subscriptable")
 
 
 elif sys.version_info[:2] >= (3, 7):

pandas/_version.py

@@ -74,7 +74,7 @@ def run_command(commands, args, cwd=None, verbose=False, hide_stderr=False):
                 stderr=(subprocess.PIPE if hide_stderr else None),
             )
             break
-        except EnvironmentError:
+        except OSError:
             e = sys.exc_info()[1]
             if e.errno == errno.ENOENT:
                 continue
@@ -121,7 +121,7 @@ def git_get_keywords(versionfile_abs):
     # _version.py.
     keywords = {}
     try:
-        f = open(versionfile_abs, "r")
+        f = open(versionfile_abs)
         for line in f.readlines():
             if line.strip().startswith("git_refnames ="):
                 mo = re.search(r'=\s*"(.*)"', line)
@@ -132,7 +132,7 @@ def git_get_keywords(versionfile_abs):
                 if mo:
                     keywords["full"] = mo.group(1)
         f.close()
-    except EnvironmentError:
+    except OSError:
         pass
     return keywords
 

pandas/core/algorithms.py

@@ -440,7 +440,12 @@ def isin(comps: AnyArrayLike, values: AnyArrayLike) -> np.ndarray:
     # GH16012
     # Ensure np.in1d doesn't get object types or it *may* throw an exception
     if len(comps) > 1_000_000 and not is_object_dtype(comps):
-        f = np.in1d
+        # If the the values include nan we need to check for nan explicitly
+        # since np.nan it not equal to np.nan
+        if np.isnan(values).any():
+            f = lambda c, v: np.logical_or(np.in1d(c, v), np.isnan(c))
+        else:
+            f = np.in1d
     elif is_integer_dtype(comps):
         try:
             values = values.astype("int64", copy=False)

pandas/core/arrays/datetimes.py

@@ -570,8 +570,7 @@ def __iter__(self):
             converted = ints_to_pydatetime(
                 data[start_i:end_i], tz=self.tz, freq=self.freq, box="timestamp"
             )
-            for v in converted:
-                yield v
+            yield from converted
 
     def astype(self, dtype, copy=True):
         # We handle

pandas/core/arrays/interval.py

@@ -547,38 +547,7 @@ def __getitem__(self, value):
         return self._shallow_copy(left, right)
 
     def __setitem__(self, key, value):
-        # na value: need special casing to set directly on numpy arrays
-        needs_float_conversion = False
-        if is_scalar(value) and isna(value):
-            if is_integer_dtype(self.dtype.subtype):
-                # can't set NaN on a numpy integer array
-                needs_float_conversion = True
-            elif is_datetime64_any_dtype(self.dtype.subtype):
-                # need proper NaT to set directly on the numpy array
-                value = np.datetime64("NaT")
-            elif is_timedelta64_dtype(self.dtype.subtype):
-                # need proper NaT to set directly on the numpy array
-                value = np.timedelta64("NaT")
-            value_left, value_right = value, value
-
-        # scalar interval
-        elif is_interval_dtype(value) or isinstance(value, Interval):
-            self._check_closed_matches(value, name="value")
-            value_left, value_right = value.left, value.right
-
-        else:
-            # list-like of intervals
-            try:
-                array = IntervalArray(value)
-                value_left, value_right = array.left, array.right
-            except TypeError as err:
-                # wrong type: not interval or NA
-                msg = f"'value' should be an interval type, got {type(value)} instead."
-                raise TypeError(msg) from err
-
-        if needs_float_conversion:
-            raise ValueError("Cannot set float NaN to integer-backed IntervalArray")
-
+        value_left, value_right = self._validate_setitem_value(value)
         key = check_array_indexer(self, key)
 
         # Need to ensure that left and right are updated atomically, so we're
@@ -898,6 +867,41 @@ def _validate_insert_value(self, value):
             )
         return left_insert, right_insert
 
+    def _validate_setitem_value(self, value):
+        needs_float_conversion = False
+
+        if is_scalar(value) and isna(value):
+            # na value: need special casing to set directly on numpy arrays
+            if is_integer_dtype(self.dtype.subtype):
+                # can't set NaN on a numpy integer array
+                needs_float_conversion = True
+            elif is_datetime64_any_dtype(self.dtype.subtype):
+                # need proper NaT to set directly on the numpy array
+                value = np.datetime64("NaT")
+            elif is_timedelta64_dtype(self.dtype.subtype):
+                # need proper NaT to set directly on the numpy array
+                value = np.timedelta64("NaT")
+            value_left, value_right = value, value
+
+        elif is_interval_dtype(value) or isinstance(value, Interval):
+            # scalar interval
+            self._check_closed_matches(value, name="value")
+            value_left, value_right = value.left, value.right
+
+        else:
+            try:
+                # list-like of intervals
+                array = IntervalArray(value)
+                value_left, value_right = array.left, array.right
+            except TypeError as err:
+                # wrong type: not interval or NA
+                msg = f"'value' should be an interval type, got {type(value)} instead."
+                raise TypeError(msg) from err
+
+        if needs_float_conversion:
+            raise ValueError("Cannot set float NaN to integer-backed IntervalArray")
+        return value_left, value_right
+
     def value_counts(self, dropna=True):
         """
         Returns a Series containing counts of each interval.

pandas/core/arrays/sparse/array.py

@@ -1427,7 +1427,7 @@ def sparse_arithmetic_method(self, other):
                     # TODO: look into _wrap_result
                     if len(self) != len(other):
                         raise AssertionError(
-                            (f"length mismatch: {len(self)} vs. {len(other)}")
+                            f"length mismatch: {len(self)} vs. {len(other)}"
                         )
                     if not isinstance(other, SparseArray):
                         dtype = getattr(other, "dtype", None)

pandas/core/common.py

@@ -62,8 +62,7 @@ def flatten(l):
     """
     for el in l:
         if iterable_not_string(el):
-            for s in flatten(el):
-                yield s
+            yield from flatten(el)
         else:
             yield el
 
@@ -434,10 +433,8 @@ def random_state(state=None):
         return np.random
     else:
         raise ValueError(
-            (
-                "random_state must be an integer, array-like, a BitGenerator, "
-                "a numpy RandomState, or None"
-            )
+            "random_state must be an integer, array-like, a BitGenerator, "
+            "a numpy RandomState, or None"
         )